Feature Request: Proxy Prioritization (for NFS Customers)

[jmeyer_cm]

A job has two settings for the proxy: 1. Automatically select a proxy using ProxyDetector, and 2. Manually specify a proxy.

Based on information I've gathered on how ProxyDetector works, a job set to 'Automatic' chooses the appropriate proxy to use based on:

1. Network Location (Based off IP and subnet)
2. Current Load (Within the settings of the GUI you can specify how many concurrent tasks a proxy can do. Default: 1 per 2 cores)
3. Backup Mode Capabilities (In order of priority: SAN, HOTADD, or NBD)

I would like to request a fourth option for ProxyDetector:

4. Prioritize any proxies that exists on the same host as target VM.

This would benefit all customers using NFS storage and Veeam's 'Appliance Transport Mode' which utilizes VMware's VDDK API. This API has a known problem with NFS locking, requring either the backup appliance to run on the same host as the target VM, or to use Network Mode (not recommended by Veeam due to slower performance).

http://kb.vmware.com/selfservice/micros ... Id=2033540

[dellock6]

This is however a partial solution, since you have for sure many VMs and they are not on the same ESXi host. If the job has more than one VM in it, chances are it will be local to a given VM, but remote to many others. I've got the same problem too even if I am using iscsi, and I solved it by creating a new dedicated proxy running in network mode only.

Luca.

[Gostev]

This VDDK bug has just been reported by us to VMware (the article you are referencing has been posted just a few weeks ago), and I am sure VMware will fix this bug soon since this is serious, production-down type issue. There's simply no need to build massive workaround, like this issue is going to be around forever.

What would help if all affected people open support cases with VMware regarding this issue, as the number of affected people directly affects priority of each fix.

[jmeyer_cm]

I had opened a case with VMware for this issue and they explained that the NFS locking behavior (documented in the link above) will exist until [at least] the next full release (v.6) of ESXi which they said would not be released until at lease Q4 2013. They recommened using the workarounds provided until then. Currently I use the network mode workaround, but Appliance/hotadd mode should provide better performance if we can just figure out how to ensure the backup proxy is on the same host as the target VM.

[tsightler]

Has anyone tried this "fix" for the NFS locking problem:

http://kb.vmware.com/selfservice/micros ... Id=2010953

This link/KB references the 2033540 KB article so I've been wondering if it might help.

[jmeyer_cm]

Yes, we did set this parameter on our hosts (it was previously set to 3), but it did not resolve the behavior where the VM is 'stunned' during snapshot removal. This KB also mentions 'Alternately, you can use the NFC (Network File Copy) transport method in your backup solution' which is the solution I've had to settle on up to this point.

[hennys]

I had opened a case with VMware for this issue and they explained that the NFS locking behavior (documented in the link above) will exist until [at least] the next full release (v.6) of ESXi which they said would not be released until at lease Q4 2013. They recommened using the workarounds provided until then. Currently I use the network mode workaround, but Appliance/hotadd mode should provide better performance if we can just figure out how to ensure the backup proxy is on the same host as the target VM.

We also had a case open with vmware. They told us that this would be fixed in 5.0U2, scheduled for Q1 2013.
Strange that they communicate different fix releases, or perhaps the need for the fix has increased, so they moved the fix from esxi6 to esxi5.0U2 ?

Kind Regards,

Hen

[Gostev]

perhaps the need for the fix has increased, so they moved the fix from esxi6 to esxi5.0U2?

Exactly right!

At least, this is how it works in most software companies (Veeam no exception) - the fix priority is solely defined by the number of support cases open. I am sure, if they get 10'000 support cases open for this issue, they will move it even sooner or even make a standalone fix! That's why I always encourage everyone to open support cases for every issue.

[EdQ]

Concerning the VMware 2010953 syndrome (Virtual machines residing on NFS storage become unresponsive during a snapshot removal operation), have any of you seen a case where 100+ VMs on a proxy were fine for many months (5 second snapshot commits) then one day suddenly fell ill (60 second snapshot commits)? Running ESXi 5.1.0 on > 10 nodes, NetApp 8.1.2. Explaining to management how results could have changed so drastically overnight is a challenge. My VMware/NetApp/NFS background is very light, so all leads will be appreciated.

[foggy]

Edward, were there probably some changes performed to infrastructure that could result in longer snapshots commit or large amount of changes were performed inside the VM(s)? Do you see this behavior on all VMs/group of VMs? Is it a one-time occurrence or observed during each job run?

[EdQ]

All VMs and B&R jobs on this VMware cluster were affected (except those sharing the same ESX host as backup proxy) with little or no load dependency. Manually created snapshots kept open for several minutes deleted OK in several seconds. Minute-long snapshot-delete hangs were observed for two consecutive nights before VMware sysadmin ordered all Veeam backups disabled. After substituting network mode for hot-add on one proxy, problem went away so I think I understand the workaround, but concerns remain over how such a severe issue could have developed so suddenly.

[veremin]

Manually created snapshots kept open for several minutes deleted OK in several seconds

Additionally, I’m wondering whether you have kept the snapshot open for long enough time before deleting it. In other words, whether this time was similar to time it takes to back this VM up or not. Thanks.

[EdQ]

Hi Vladimir, yes, the snapshot was left open longer than the backup took. How to duplicate the hang effect by running manual snapshots, however, is not the main issue. The greater concern is how ninety percent of a proxy's hot-adds could develop lengthy VMware NFS snapshot delete freezes on the same day, after many months of trouble-free operation. If there are any such known cases, the next question, of course, would be what particular infrastructure factors were involved. Thanks for reading.