DirectNFS, NBD/Hotadd and Windows

[svenh]

Hi!

I am currently testing Veeam9s DirectNFS mode using ESX5.5u2 and Ontap 8.2.3 7mode and I am going slightly insane here.

I moved some test-VMs over to some new NFS storages in a storage cluster and backing them up using Veeam. This works fine, DirectNFS is used.

But: Not all VMs use DirectNFS, some VMs, notably running Windows VMs will most times (but not always) get backupped using Hotadd or NBD (depending on the Proxy setting). Linux VMs don't show this problem.

But I cannot figure out why. What I already did:

• moved the VM to a different NFS datastore
• switched off the VM (this resulted in the VM getting saved via NFS)
• noticed it was always the same proxy using Hotadd/NBD -> switched off the proxy -> same problem on different proxy
• rebooted all proxies -> nope, still showing
• noticed VMware Tools where out of date -> updated VMware Tools -> nope, problem persists

I don't know why Veeam insists on not using DirectNFS for some VMs while other VMs on the same datastore (!) get backupped via NFS by the same proxy. So this clearly is no connection problem.

It also can't be a contention problem on the storage or the proxy, because even if I do a single QuickBackup of a VM showing this anomaly and no other jobs are running, Hotadd/NBD still gets selected over DirectNFS.

I even read the log files on the proxy but there are no errors or warnings visible.

Does anybody else also see this? What can I try, what have I missed to check?

Since this no serious error and I am still in the testing stage of that setup, I don't want to submit a case right away. Maybe someone here has additional ideas.

[dellock6]

The first check I'd do is if the skipped VMs have some active snapshots, as DirectNFS cannot backup any VM with a snapshot. This is stated clearly in the user guide:

https://helpcenter.veeam.com/backup/vsp ... ccess.html

Luca

[svenh]

No snapshots on any of the VMs.

The really strange part is, there is one VM which was successfully backupped via NFS and then, just 5 minutes later only via NBD since then.

I also re-scanned my Infrastructure and both NetApps several times, but that also did not change the way backups are made for those Windows VMs.

[kjstech]

I think direct NFS access is great. Make sure all your proxies have an interface on the same network as your NFS storage and that the NFS appliance has given access to those ip interfaces in each NFS share.

It halved our backup Windows and now we are seeing the benefit of 10g networking.

I only have one issue. If anyone has ever deployed SolarWinds Lem virtual appliance (they provide an ovf), the default VM folder name is \SolarWinds Log & Event Manager and I get errors in veeam saying it does not have access to this file, but in the veeam logs the ampersand is omitted (it shows SolarWinds Log Event Manager). So I think there MAY be a bug in Veeam NFS client with special characters.

I've since powered that VM down, removed it from job and vSphere, renamed the folder, re added it to vsphere and the job. Now I get a different error... Task failed Error: fault.ManagedObjectNotFound.summary

So again I removed it from the job, did windows updates and rebooted veeam server, and then re-added it to job yet again (I hit refresh 5 times in the VMware browser dialog box).

So I don't think it's without its bugs, which is to be expected for the first release of a new feature. However in my environment I can now backup without having incredibly poor VM STUN times. No more text alerts at night regarding machines going offline (in which we would ignore because it was always the time of the backup). In my situation I'll go without backing up SolarWinds LEM if I have to and just rely on their built in configuration export / import for that. It's just less disruptive and what backup should have been from day 1.

PS. Veeam forums are not working properly in Tapatalk

[svenh]

I think direct NFS access is great. Make sure all your proxies have an interface on the same network as your NFS storage and that the NFS appliance has given access to those ip interfaces in each NFS share.

As I said: DirectNFS works for me (98% if the time).

All proxies have an interface in the SAN and all IPs have the necessary access rights to the NFS shares and I am able to backup systems using DirectNFS from every share. But for some VMs (notably running Windows VMs) the system falls back to Hotadd/NBD and I want to know why.

[svenh]

So I started to experiment a bit more and moved some more VMs over to the NFS datastores, some of them Windows, some of them Linux, some of them with old VMware Tools, some of them even without VMware Tools. I also made sure no snapshots were present.

But guess what: all Linux VMs are processed using DirectNFS, all running Windows VMs get NBD. Only if I switch off a Windows VM, it also is processed using DirectNFS.

Huh? What? What is the difference on the storage-side between a Linux VM and an running Windows VM that is causing that behavior.

[dellock6]

Do you have by any chance VMware tools quiescence enabled in the jobs? I remember some internal discussion about this topic and there's was this exact note in regards to linux vs windows VMs processing. Can you check?

[svenh]

Yes, yes I do have VMware Tools Quiescence enabled. Let me switch that off on my Test VMs and see what happens.

[svenh]

Yes, this was it. Disabling the Quiescing allows the VMs to be processed via NFS.

Now my question is: Why? If the VM is on an iSCSI LUN, Veeam is able to backup it via direct SAN access without problems. Where is the difference storage-wise between iSCSI and NFS when doing quiescing via VMware Tools. That should only be different for the internal state of the VM, shouldn't it? There has to be something different or Veeam wouldn't behave different.

[dellock6]

While I was waiting for your test, I went to the internal forum and found the explanation: Windows VMs are affected because of how VMware Tools quiescence does its thing, in a very weird way. They create an additional temporary virtual disk for each virtual disk in a VM. This is pretty stupid, but the final result is that you end up with a VM with another snapshot, and thus back into the case where DirectNFS cannot be used on a VM with an open snapshot, as the snapshot is created before the transport method selection.

Our suggestion is to use Veeam own application aware processing, and forget vmware tools quiescence, it's more stable and has many more options. We have seen during these years many issues with vmware tools quiescence, this is just another one.

Glad we found the issue of you problem, and you can now fully leverage DirectNFS.

Luca

[svenh]

While I was waiting for your test, I went to the internal forum and found the explanation: Windows VMs are affected because of how VMware Tools quiescence does its thing, in a very weird way. They create an additional temporary virtual disk for each virtual disk in a VM. This is pretty stupid, but the final result is that you end up with a VM with another snapshot, and thus back into the case where DirectNFS cannot be used on a VM with an open snapshot, as the snapshot is created before the transport method selection.

This really should be in a KB entry, since I guess this is something nearly every user of DirectNFS will stumble upon.

Our suggestion is to use Veeam own application aware processing, and forget vmware tools quiescence, it's more stable and has many more options. We have seen during these years many issues with vmware tools quiescence, this is just another one.

That won't be an option for me, as I don't have (and can't have) Guest OS credentials for many, if not all, of the VMs I am processing with Veeam.

But I guess I will just disable VMware Tools VSS, as I ran a different backup solution for years before that which didn't have this option in the first place and everything was fine and only enabled it in Veeam because it was available.

But nonetheless, many thanks for your help on that topic.

[dellock6]

It was in a discussion just few days ago, pretty sure a KB will be created now that we have additional feedback from users like you.
Thanks Sven.

Luca