Have an interesting IPv6 issue with Veeam B&R

[linux4guru]

Just curious - has anyone backup a VM on an IPv4 network, but also house an IPv6 network? My issue is that when we perform a backup on this VM on IPv4 network, the IPv6 somehow dropped off the network - nagios alert say it can't ping the IPv6 network or fetch a page from the site.

If we disabled backup from running - everything works fine on the IPv6 network and when backup turned back on; it gets dropped during a backup.

The ESX host has a 1GbE connection for all VLANs - I am wondering if IPv4 and IPv6 does not play nice together from the same physical interface.

[Gostev]

Interesting indeed... I hope some networking gurus chime in with their ideas.

[linux4guru]

Our ESX host server has 2 interfaces.

1. 1GbE - All VLAN traffic for internal and external network use including our IPv6.
2. 1GbE - Dedicated storage connection to the SAN (NetApp)

I hope this provide a little more how the ESX configured.

[dellock6]

Not a networking guru at all here, but can it simply be the backups activities fills up the network and so ipv6 checks fails? ipv4 and ipv6 uses two separated stacks but share the same network...we also see in our DC nagios checks showing higher latencies during backup operations...

[linux4guru]

Ops...! I made a mistake on my posting about 1GbE - CORRECTION: 10GbE to the Nexus 5k. The ESX host lives on Cisco UCS environment. The VM have 2 Nics - one on IPv4 and one on IPv6 (they are virtual interfaces) - that can't be the cause because it is a 10GbE pipe?

[dellock6]

Can you check how much bandwidth is used by Veeam backup while running? Hard to fill a 10Gb line, but just to exclude bandwidth is the problem.

Luca.

[linux4guru]

The full backup ran at 88Mb/s which should be fine. It seems to happen with the full backup only.

[dellock6]

About your design, I understood correctly Veeam is a VM? So you are using hotadd mode? which type of virtual metwork is configured on the VM? I hope vmxnet3 to take advantage of the underlying 10G network...
Because 88 MBs are near the limit of 1 Gbits network, whose limit is 125 MBs. And 88 MB is the final result of the backup traffic, but maybe the original traffic coming out of the SAN is higher before beeing compressed and deduplicated by Veeam.

[linux4guru]

Yes, I am using hotadd mode for our backup. On the vSwitch1 - we have two nics - vmnic2 and vmnic3. Inside vSwitch1, we have multiple Virtual Machine Port Group with multiple VLANs.

[tsightler]

How sensitive do you have your Nagios configured? Will it send an alert after on a single failure (i.e. does it consider 1 failed check a "Hard" failure)? Did the problem happen during the entire duration of the backup or only at the end while the snapshot was being removed? I have worked with clients that have IPv6 in their network and they have never reported such issues, nor have I seen any in my lab. It's difficult to imagine that taking a snapshot could possibly be an issue here, but snapshot removal is a different story.

[linux4guru]

Not to sure on Nagios alert setting but I think it is set to every 15 minutes or maybe 30 minutes ping and to also pull up a page via IPv6. I think it happens when it remove the snapshot. I did notice on the share point sql server became very slow when I am actually RDP to it during a snapshot removal. The performance was unbearable.

[tsightler]

Snapshot removal will always lead to some impact and can cause a short period of no network response due to the required stun to commit the final snapshot data. If Nagios is configured to be very sensitive and consider any single failure as an alert, then you might see an alert if the check occurs concurrently with the snapshot removal.

[dellock6]

Thomas, you can do the usual counter-check in this case: look how much time does your backup lasts. Say it's 30 minutes, then go on vSphere client, take a snapshot of that VM, wait 30 minutes and then delete the snapshot. This is the same activity Veeam does when doing backups.
If you see the same network problems, then the problem is definitely the VM snapshot deletion and its stun operation, and you would need to tune Nagios accordingly, like creating a maintenance window in its daily checks.

Luca.

[tsightler]

Another option to tune Nagios is to simply configure a "soft" and "hard" check. Nagios only sends alerts for "hard" outages. For example, you can configure Nagios to check a server every 15 minutes, if the check fails, then it can check 2 more times at 1 minute intervals, only if it fails all three checks does the error become a "hard" error and send an alert. The "soft" failures will still be in the log, so this allows you to keep your server history, visualize the length of the service issue, but without flooding alerts for every single lost ping/slow response.

[linux4guru]

Tuning Nagios is one thing but the IPv6 network is completely off the network - it is not reachable from outside or from an IPv6 network. Normally, we just up the interface when it is down.