Comprehensive data protection for all workloads
Post Reply
linux4guru
Enthusiast
Posts: 40
Liked: never
Joined: Aug 24, 2011 8:26 pm
Full Name: Thomas Lam
Contact:

Have an interesting IPv6 issue with Veeam B&R

Post by linux4guru »

Just curious - has anyone backup a VM on an IPv4 network, but also house an IPv6 network? My issue is that when we perform a backup on this VM on IPv4 network, the IPv6 somehow dropped off the network - nagios alert say it can't ping the IPv6 network or fetch a page from the site.

If we disabled backup from running - everything works fine on the IPv6 network and when backup turned back on; it gets dropped during a backup.

The ESX host has a 1GbE connection for all VLANs - I am wondering if IPv4 and IPv6 does not play nice together from the same physical interface.
Gostev
Chief Product Officer
Posts: 31457
Liked: 6648 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Have an interesting issue with Veeam B&R

Post by Gostev »

Interesting indeed... I hope some networking gurus chime in with their ideas.
linux4guru
Enthusiast
Posts: 40
Liked: never
Joined: Aug 24, 2011 8:26 pm
Full Name: Thomas Lam
Contact:

Re: Have an interesting issue with Veeam B&R

Post by linux4guru »

Our ESX host server has 2 interfaces.

1. 1GbE - All VLAN traffic for internal and external network use including our IPv6.
2. 1GbE - Dedicated storage connection to the SAN (NetApp)

I hope this provide a little more how the ESX configured.
dellock6
Veeam Software
Posts: 6137
Liked: 1928 times
Joined: Jul 26, 2009 3:39 pm
Full Name: Luca Dell'Oca
Location: Varese, Italy
Contact:

Re: Have an interesting issue with Veeam B&R

Post by dellock6 »

Not a networking guru at all here, but can it simply be the backups activities fills up the network and so ipv6 checks fails? ipv4 and ipv6 uses two separated stacks but share the same network...we also see in our DC nagios checks showing higher latencies during backup operations...
Luca Dell'Oca
Principal EMEA Cloud Architect @ Veeam Software

@dellock6
https://www.virtualtothecore.com/
vExpert 2011 -> 2022
Veeam VMCE #1
linux4guru
Enthusiast
Posts: 40
Liked: never
Joined: Aug 24, 2011 8:26 pm
Full Name: Thomas Lam
Contact:

Re: Have an interesting issue with Veeam B&R

Post by linux4guru »

Ops...! I made a mistake on my posting about 1GbE - CORRECTION: 10GbE to the Nexus 5k. The ESX host lives on Cisco UCS environment. The VM have 2 Nics - one on IPv4 and one on IPv6 (they are virtual interfaces) - that can't be the cause because it is a 10GbE pipe?
dellock6
Veeam Software
Posts: 6137
Liked: 1928 times
Joined: Jul 26, 2009 3:39 pm
Full Name: Luca Dell'Oca
Location: Varese, Italy
Contact:

Re: Have an interesting issue with Veeam B&R

Post by dellock6 »

Can you check how much bandwidth is used by Veeam backup while running? Hard to fill a 10Gb line, but just to exclude bandwidth is the problem.

Luca.
Luca Dell'Oca
Principal EMEA Cloud Architect @ Veeam Software

@dellock6
https://www.virtualtothecore.com/
vExpert 2011 -> 2022
Veeam VMCE #1
linux4guru
Enthusiast
Posts: 40
Liked: never
Joined: Aug 24, 2011 8:26 pm
Full Name: Thomas Lam
Contact:

Re: Have an interesting issue with Veeam B&R

Post by linux4guru »

The full backup ran at 88Mb/s which should be fine. It seems to happen with the full backup only.
dellock6
Veeam Software
Posts: 6137
Liked: 1928 times
Joined: Jul 26, 2009 3:39 pm
Full Name: Luca Dell'Oca
Location: Varese, Italy
Contact:

Re: Have an interesting issue with Veeam B&R

Post by dellock6 »

About your design, I understood correctly Veeam is a VM? So you are using hotadd mode? which type of virtual metwork is configured on the VM? I hope vmxnet3 to take advantage of the underlying 10G network...
Because 88 MBs are near the limit of 1 Gbits network, whose limit is 125 MBs. And 88 MB is the final result of the backup traffic, but maybe the original traffic coming out of the SAN is higher before beeing compressed and deduplicated by Veeam.
Luca Dell'Oca
Principal EMEA Cloud Architect @ Veeam Software

@dellock6
https://www.virtualtothecore.com/
vExpert 2011 -> 2022
Veeam VMCE #1
linux4guru
Enthusiast
Posts: 40
Liked: never
Joined: Aug 24, 2011 8:26 pm
Full Name: Thomas Lam
Contact:

Re: Have an interesting issue with Veeam B&R

Post by linux4guru »

Yes, I am using hotadd mode for our backup. On the vSwitch1 - we have two nics - vmnic2 and vmnic3. Inside vSwitch1, we have multiple Virtual Machine Port Group with multiple VLANs.
tsightler
VP, Product Management
Posts: 6009
Liked: 2843 times
Joined: Jun 05, 2009 12:57 pm
Full Name: Tom Sightler
Contact:

Re: Have an interesting issue with Veeam B&R

Post by tsightler »

How sensitive do you have your Nagios configured? Will it send an alert after on a single failure (i.e. does it consider 1 failed check a "Hard" failure)? Did the problem happen during the entire duration of the backup or only at the end while the snapshot was being removed? I have worked with clients that have IPv6 in their network and they have never reported such issues, nor have I seen any in my lab. It's difficult to imagine that taking a snapshot could possibly be an issue here, but snapshot removal is a different story.
linux4guru
Enthusiast
Posts: 40
Liked: never
Joined: Aug 24, 2011 8:26 pm
Full Name: Thomas Lam
Contact:

Re: Have an interesting issue with Veeam B&R

Post by linux4guru »

Not to sure on Nagios alert setting but I think it is set to every 15 minutes or maybe 30 minutes ping and to also pull up a page via IPv6. I think it happens when it remove the snapshot. I did notice on the share point sql server became very slow when I am actually RDP to it during a snapshot removal. The performance was unbearable.
tsightler
VP, Product Management
Posts: 6009
Liked: 2843 times
Joined: Jun 05, 2009 12:57 pm
Full Name: Tom Sightler
Contact:

Re: Have an interesting IPv6 issue with Veeam B&R

Post by tsightler »

Snapshot removal will always lead to some impact and can cause a short period of no network response due to the required stun to commit the final snapshot data. If Nagios is configured to be very sensitive and consider any single failure as an alert, then you might see an alert if the check occurs concurrently with the snapshot removal.
dellock6
Veeam Software
Posts: 6137
Liked: 1928 times
Joined: Jul 26, 2009 3:39 pm
Full Name: Luca Dell'Oca
Location: Varese, Italy
Contact:

Re: Have an interesting IPv6 issue with Veeam B&R

Post by dellock6 »

Thomas, you can do the usual counter-check in this case: look how much time does your backup lasts. Say it's 30 minutes, then go on vSphere client, take a snapshot of that VM, wait 30 minutes and then delete the snapshot. This is the same activity Veeam does when doing backups.
If you see the same network problems, then the problem is definitely the VM snapshot deletion and its stun operation, and you would need to tune Nagios accordingly, like creating a maintenance window in its daily checks.

Luca.
Luca Dell'Oca
Principal EMEA Cloud Architect @ Veeam Software

@dellock6
https://www.virtualtothecore.com/
vExpert 2011 -> 2022
Veeam VMCE #1
tsightler
VP, Product Management
Posts: 6009
Liked: 2843 times
Joined: Jun 05, 2009 12:57 pm
Full Name: Tom Sightler
Contact:

Re: Have an interesting IPv6 issue with Veeam B&R

Post by tsightler »

Another option to tune Nagios is to simply configure a "soft" and "hard" check. Nagios only sends alerts for "hard" outages. For example, you can configure Nagios to check a server every 15 minutes, if the check fails, then it can check 2 more times at 1 minute intervals, only if it fails all three checks does the error become a "hard" error and send an alert. The "soft" failures will still be in the log, so this allows you to keep your server history, visualize the length of the service issue, but without flooding alerts for every single lost ping/slow response.
linux4guru
Enthusiast
Posts: 40
Liked: never
Joined: Aug 24, 2011 8:26 pm
Full Name: Thomas Lam
Contact:

Re: Have an interesting IPv6 issue with Veeam B&R

Post by linux4guru »

Tuning Nagios is one thing but the IPv6 network is completely off the network - it is not reachable from outside or from an IPv6 network. Normally, we just up the interface when it is down.
Post Reply

Who is online

Users browsing this forum: ante_704, Bing [Bot], Semrush [Bot] and 197 guests