DirectNFS, NBD/Hotadd and Windows

VMware specific discussions

DirectNFS, NBD/Hotadd and Windows

Veeam Logoby svenh » Fri Feb 12, 2016 5:08 pm

Hi!

I am currently testing Veeam9s DirectNFS mode using ESX5.5u2 and Ontap 8.2.3 7mode and I am going slightly insane here.

I moved some test-VMs over to some new NFS storages in a storage cluster and backing them up using Veeam. This works fine, DirectNFS is used.

But: Not all VMs use DirectNFS, some VMs, notably running Windows VMs will most times (but not always) get backupped using Hotadd or NBD (depending on the Proxy setting). Linux VMs don't show this problem.

But I cannot figure out why. What I already did:

  • moved the VM to a different NFS datastore
  • switched off the VM (this resulted in the VM getting saved via NFS)
  • noticed it was always the same proxy using Hotadd/NBD -> switched off the proxy -> same problem on different proxy
  • rebooted all proxies -> nope, still showing
  • noticed VMware Tools where out of date -> updated VMware Tools -> nope, problem persists

I don't know why Veeam insists on not using DirectNFS for some VMs while other VMs on the same datastore (!) get backupped via NFS by the same proxy. So this clearly is no connection problem.

It also can't be a contention problem on the storage or the proxy, because even if I do a single QuickBackup of a VM showing this anomaly and no other jobs are running, Hotadd/NBD still gets selected over DirectNFS.

I even read the log files on the proxy but there are no errors or warnings visible.

Does anybody else also see this? What can I try, what have I missed to check?

Since this no serious error and I am still in the testing stage of that setup, I don't want to submit a case right away. Maybe someone here has additional ideas.
svenh
Enthusiast
 
Posts: 26
Liked: 5 times
Joined: Thu Jan 21, 2016 12:56 pm
Location: Germany
Full Name: Sven Hartge

Re: DirectNFS, NBD/Hotadd and Windows

Veeam Logoby dellock6 » Sat Feb 13, 2016 1:37 pm

The first check I'd do is if the skipped VMs have some active snapshots, as DirectNFS cannot backup any VM with a snapshot. This is stated clearly in the user guide:

https://helpcenter.veeam.com/backup/vsp ... ccess.html

Luca
Luca Dell'Oca
EMEA Cloud Architect @ Veeam Software

@dellock6
http://www.virtualtothecore.com
vExpert 2011-2012-2013-2014-2015-2016
Veeam VMCE #1
dellock6
Veeam Software
 
Posts: 5047
Liked: 1330 times
Joined: Sun Jul 26, 2009 3:39 pm
Location: Varese, Italy
Full Name: Luca Dell'Oca

Re: DirectNFS, NBD/Hotadd and Windows

Veeam Logoby svenh » Sat Feb 13, 2016 2:48 pm

No snapshots on any of the VMs.

The really strange part is, there is one VM which was successfully backupped via NFS and then, just 5 minutes later only via NBD since then.

I also re-scanned my Infrastructure and both NetApps several times, but that also did not change the way backups are made for those Windows VMs.
svenh
Enthusiast
 
Posts: 26
Liked: 5 times
Joined: Thu Jan 21, 2016 12:56 pm
Location: Germany
Full Name: Sven Hartge

Re: DirectNFS, NBD/Hotadd and Windows

Veeam Logoby kjstech » Sat Feb 13, 2016 3:02 pm

I think direct NFS access is great. Make sure all your proxies have an interface on the same network as your NFS storage and that the NFS appliance has given access to those ip interfaces in each NFS share.

It halved our backup Windows and now we are seeing the benefit of 10g networking.

I only have one issue. If anyone has ever deployed SolarWinds Lem virtual appliance (they provide an ovf), the default VM folder name is \SolarWinds Log & Event Manager and I get errors in veeam saying it does not have access to this file, but in the veeam logs the ampersand is omitted (it shows SolarWinds Log Event Manager). So I think there MAY be a bug in Veeam NFS client with special characters.

I've since powered that VM down, removed it from job and vSphere, renamed the folder, re added it to vsphere and the job. Now I get a different error... Task failed Error: fault.ManagedObjectNotFound.summary

So again I removed it from the job, did windows updates and rebooted veeam server, and then re-added it to job yet again (I hit refresh 5 times in the VMware browser dialog box).

So I don't think it's without its bugs, which is to be expected for the first release of a new feature. However in my environment I can now backup without having incredibly poor VM STUN times. No more text alerts at night regarding machines going offline (in which we would ignore because it was always the time of the backup). In my situation I'll go without backing up SolarWinds LEM if I have to and just rely on their built in configuration export / import for that. It's just less disruptive and what backup should have been from day 1.

PS. Veeam forums are not working properly in Tapatalk
kjstech
Expert
 
Posts: 142
Liked: 14 times
Joined: Fri Jan 17, 2014 4:12 pm
Full Name: Keith S

Re: DirectNFS, NBD/Hotadd and Windows

Veeam Logoby svenh » Sat Feb 13, 2016 4:22 pm

kjstech wrote:I think direct NFS access is great. Make sure all your proxies have an interface on the same network as your NFS storage and that the NFS appliance has given access to those ip interfaces in each NFS share.


As I said: DirectNFS works for me (98% if the time).

All proxies have an interface in the SAN and all IPs have the necessary access rights to the NFS shares and I am able to backup systems using DirectNFS from every share. But for some VMs (notably running Windows VMs) the system falls back to Hotadd/NBD and I want to know why.
svenh
Enthusiast
 
Posts: 26
Liked: 5 times
Joined: Thu Jan 21, 2016 12:56 pm
Location: Germany
Full Name: Sven Hartge

Re: DirectNFS, NBD/Hotadd and Windows

Veeam Logoby svenh » Sat Feb 13, 2016 11:02 pm

So I started to experiment a bit more and moved some more VMs over to the NFS datastores, some of them Windows, some of them Linux, some of them with old VMware Tools, some of them even without VMware Tools. I also made sure no snapshots were present.

But guess what: all Linux VMs are processed using DirectNFS, all running Windows VMs get NBD. Only if I switch off a Windows VM, it also is processed using DirectNFS.

Huh? What? What is the difference on the storage-side between a Linux VM and an running Windows VM that is causing that behavior.
svenh
Enthusiast
 
Posts: 26
Liked: 5 times
Joined: Thu Jan 21, 2016 12:56 pm
Location: Germany
Full Name: Sven Hartge

Re: DirectNFS, NBD/Hotadd and Windows

Veeam Logoby dellock6 » Sun Feb 14, 2016 12:28 am 1 person likes this post

Do you have by any chance VMware tools quiescence enabled in the jobs? I remember some internal discussion about this topic and there's was this exact note in regards to linux vs windows VMs processing. Can you check?
Luca Dell'Oca
EMEA Cloud Architect @ Veeam Software

@dellock6
http://www.virtualtothecore.com
vExpert 2011-2012-2013-2014-2015-2016
Veeam VMCE #1
dellock6
Veeam Software
 
Posts: 5047
Liked: 1330 times
Joined: Sun Jul 26, 2009 3:39 pm
Location: Varese, Italy
Full Name: Luca Dell'Oca

Re: DirectNFS, NBD/Hotadd and Windows

Veeam Logoby svenh » Sun Feb 14, 2016 12:31 am

Yes, yes I do have VMware Tools Quiescence enabled. Let me switch that off on my Test VMs and see what happens.
svenh
Enthusiast
 
Posts: 26
Liked: 5 times
Joined: Thu Jan 21, 2016 12:56 pm
Location: Germany
Full Name: Sven Hartge

Re: DirectNFS, NBD/Hotadd and Windows

Veeam Logoby svenh » Sun Feb 14, 2016 12:44 am

Yes, this was it. Disabling the Quiescing allows the VMs to be processed via NFS.

Now my question is: Why? If the VM is on an iSCSI LUN, Veeam is able to backup it via direct SAN access without problems. Where is the difference storage-wise between iSCSI and NFS when doing quiescing via VMware Tools. That should only be different for the internal state of the VM, shouldn't it? There has to be something different or Veeam wouldn't behave different.
svenh
Enthusiast
 
Posts: 26
Liked: 5 times
Joined: Thu Jan 21, 2016 12:56 pm
Location: Germany
Full Name: Sven Hartge

Re: DirectNFS, NBD/Hotadd and Windows

Veeam Logoby dellock6 » Sun Feb 14, 2016 12:54 am 1 person likes this post

While I was waiting for your test, I went to the internal forum and found the explanation: Windows VMs are affected because of how VMware Tools quiescence does its thing, in a very weird way. They create an additional temporary virtual disk for each virtual disk in a VM. This is pretty stupid, but the final result is that you end up with a VM with another snapshot, and thus back into the case where DirectNFS cannot be used on a VM with an open snapshot, as the snapshot is created before the transport method selection.

Our suggestion is to use Veeam own application aware processing, and forget vmware tools quiescence, it's more stable and has many more options. We have seen during these years many issues with vmware tools quiescence, this is just another one.

Glad we found the issue of you problem, and you can now fully leverage DirectNFS.

Luca
Luca Dell'Oca
EMEA Cloud Architect @ Veeam Software

@dellock6
http://www.virtualtothecore.com
vExpert 2011-2012-2013-2014-2015-2016
Veeam VMCE #1
dellock6
Veeam Software
 
Posts: 5047
Liked: 1330 times
Joined: Sun Jul 26, 2009 3:39 pm
Location: Varese, Italy
Full Name: Luca Dell'Oca

Re: DirectNFS, NBD/Hotadd and Windows

Veeam Logoby svenh » Sun Feb 14, 2016 1:00 am

dellock6 wrote:While I was waiting for your test, I went to the internal forum and found the explanation: Windows VMs are affected because of how VMware Tools quiescence does its thing, in a very weird way. They create an additional temporary virtual disk for each virtual disk in a VM. This is pretty stupid, but the final result is that you end up with a VM with another snapshot, and thus back into the case where DirectNFS cannot be used on a VM with an open snapshot, as the snapshot is created before the transport method selection.


This really should be in a KB entry, since I guess this is something nearly every user of DirectNFS will stumble upon.

Our suggestion is to use Veeam own application aware processing, and forget vmware tools quiescence, it's more stable and has many more options. We have seen during these years many issues with vmware tools quiescence, this is just another one.


That won't be an option for me, as I don't have (and can't have) Guest OS credentials for many, if not all, of the VMs I am processing with Veeam.

But I guess I will just disable VMware Tools VSS, as I ran a different backup solution for years before that which didn't have this option in the first place and everything was fine and only enabled it in Veeam because it was available.

But nonetheless, many thanks for your help on that topic.
svenh
Enthusiast
 
Posts: 26
Liked: 5 times
Joined: Thu Jan 21, 2016 12:56 pm
Location: Germany
Full Name: Sven Hartge

Re: DirectNFS, NBD/Hotadd and Windows

Veeam Logoby dellock6 » Sun Feb 14, 2016 1:02 am 1 person likes this post

It was in a discussion just few days ago, pretty sure a KB will be created now that we have additional feedback from users like you.
Thanks Sven.

Luca
Luca Dell'Oca
EMEA Cloud Architect @ Veeam Software

@dellock6
http://www.virtualtothecore.com
vExpert 2011-2012-2013-2014-2015-2016
Veeam VMCE #1
dellock6
Veeam Software
 
Posts: 5047
Liked: 1330 times
Joined: Sun Jul 26, 2009 3:39 pm
Location: Varese, Italy
Full Name: Luca Dell'Oca


Return to VMware vSphere



Who is online

Users browsing this forum: ERecker and 22 guests