Host-based backup of VMware vSphere VMs.
ChuckS42
Expert
Posts: 189
Liked: 27 times
Joined: Apr 24, 2013 8:53 pm
Full Name: Chuck Stevens
Location: Seattle, WA
Contact:

NBD backups when they should be NFS

Post by ChuckS42 »

We have Nutanix clusters running vSphere ESXi 6.5. We are backing servers up with Veeam B&R v10. All our proxy servers are configured for Direct Storage Access with Network failover, built as per Nutanix/Veeam documentation. I've recently noticed that a fair number of servers are using Network mode (NBD) when they shouldn't be; the only reason they should is if there's an existing snapshot on the server when the backup runs, and that is almost never the case. Anyone seen this before? (Note: every time a server backup incorrectly uses NBD it starts with a message about there being no proxies on the ESXi management subnet; ones that use NFS don't give this message. Nothing in our backup infrastructure is on the ESXi management subnet.)

It's not all servers in a backup job, maybe a third of them.

I don't know if this behavior started with v10; I just noticed it last week.

Case 04195709
Veeaming since 2013
ronnmartin61
Veeam Software
Posts: 428
Liked: 123 times
Joined: Mar 07, 2016 3:55 pm
Full Name: Ronn Martin
Contact:

Re: NBD backups when they should be NFS

Post by ronnmartin61 »

Hey Chuck, it sounds as if you have everything set up correctly but just to double-check you do have a Veeam Windows proxy deployed per ESXi server / Nutanix node and each proxy has \\HKLM\Software\Veeam\Veeam Backup and Replication\EnableSameHostDirectNFSMode = 2 set?
ChuckS42
Expert
Posts: 189
Liked: 27 times
Joined: Apr 24, 2013 8:53 pm
Full Name: Chuck Stevens
Location: Seattle, WA
Contact:

Re: NBD backups when they should be NFS

Post by ChuckS42 »

Yep, both of those.
Veeaming since 2013
ronnmartin61
Veeam Software
Posts: 428
Liked: 123 times
Joined: Mar 07, 2016 3:55 pm
Full Name: Ronn Martin
Contact:

Re: NBD backups when they should be NFS

Post by ronnmartin61 »

Thinking about this a bit more and I had recently heard of a case where this was failing over to NBD unexpectedly as you're seeing. Though the best practices guide advises "Configure a Veeam Backup & Replication global traffic rule to use the 192.168.5.0 network for backups" (which should be correct), in this particular case removing the traffic rule allowed everything to function correctly.
ChuckS42
Expert
Posts: 189
Liked: 27 times
Joined: Apr 24, 2013 8:53 pm
Full Name: Chuck Stevens
Location: Seattle, WA
Contact:

Re: NBD backups when they should be NFS

Post by ChuckS42 »

Hmm, interesting. I do have that traffic rule enabled. I'll disable it and see what happens.

EDIT: Nope, didn't make a difference.
Veeaming since 2013
nitramd
Veteran
Posts: 297
Liked: 85 times
Joined: Feb 16, 2017 8:05 pm
Contact:

Re: NBD backups when they should be NFS

Post by nitramd » 1 person likes this post

Hello Chuck.

Take a look at this article: https://www.virtualtothecore.com/protec ... l-proxies/
ChuckS42
Expert
Posts: 189
Liked: 27 times
Joined: Apr 24, 2013 8:53 pm
Full Name: Chuck Stevens
Location: Seattle, WA
Contact:

Re: NBD backups when they should be NFS

Post by ChuckS42 »

Interesting article, but nothing new there for me. We aren't using hotadd at all any more.
Veeaming since 2013
ChuckS42
Expert
Posts: 189
Liked: 27 times
Joined: Apr 24, 2013 8:53 pm
Full Name: Chuck Stevens
Location: Seattle, WA
Contact:

Re: NBD backups when they should be NFS

Post by ChuckS42 »

I've gone back over the Veeam/Nutanix best practice guide (https://www.veeam.com/wp-nutanix-veeam- ... tices.html), and the only place where we diverge is that the ESXi Management VLAN is different from the VLAN used by the public Proxy interfaces, Veeam B&R Server, and ExaGrid repositories. (Different VLANs/Subnets but no firewalls between them.)
Veeaming since 2013
ChuckS42
Expert
Posts: 189
Liked: 27 times
Joined: Apr 24, 2013 8:53 pm
Full Name: Chuck Stevens
Location: Seattle, WA
Contact:

Re: NBD backups when they should be NFS

Post by ChuckS42 »

Doing some more focused testing, I see that the VM servers that go NBD are always the same... and are on three of my hosts. The ones that run NFS as they should are on the other five hosts. I'm going to swap a couple of my proxies around and see if the problem follows the proxy.
Veeaming since 2013
ChuckS42
Expert
Posts: 189
Liked: 27 times
Joined: Apr 24, 2013 8:53 pm
Full Name: Chuck Stevens
Location: Seattle, WA
Contact:

Re: NBD backups when they should be NFS

Post by ChuckS42 »

... and the problem followed the proxy. Looks like I need to deploy new proxies.
Veeaming since 2013
nitramd
Veteran
Posts: 297
Liked: 85 times
Joined: Feb 16, 2017 8:05 pm
Contact:

Re: NBD backups when they should be NFS

Post by nitramd »

Do you have proxy affinity set?
ChuckS42
Expert
Posts: 189
Liked: 27 times
Joined: Apr 24, 2013 8:53 pm
Full Name: Chuck Stevens
Location: Seattle, WA
Contact:

Re: NBD backups when they should be NFS

Post by ChuckS42 »

Yes.
Veeaming since 2013
PetrM
Veeam Software
Posts: 3229
Liked: 520 times
Joined: Aug 28, 2013 8:23 am
Full Name: Petr Makarov
Location: Prague, Czech Republic
Contact:

Re: NBD backups when they should be NFS

Post by PetrM »

Hi Chuck,

As far as I see you have opened the support case for this issue. Could you please share the results of support case?

The proper inspection of debug logs should reveal the root cause of failover from NFS to NBD.
If you have no clear explanation so far I would recommend to request an escalation of the support case.

Thanks!
ChuckS42
Expert
Posts: 189
Liked: 27 times
Joined: Apr 24, 2013 8:53 pm
Full Name: Chuck Stevens
Location: Seattle, WA
Contact:

Re: NBD backups when they should be NFS

Post by ChuckS42 »

There are no results yet; the last thing I got was that case is being escalated to an engineer. No word back yet. In the mean time, I'm going to re-deploy the proxies that seem to have the issue and see if that fixes the problem.
Veeaming since 2013
ChuckS42
Expert
Posts: 189
Liked: 27 times
Joined: Apr 24, 2013 8:53 pm
Full Name: Chuck Stevens
Location: Seattle, WA
Contact:

Re: NBD backups when they should be NFS

Post by ChuckS42 » 2 people like this post

I have replaced the suspect proxy servers and that solved the problem. No idea what was wrong with them; they were all configured properly. Anyway, all-NFS now.
Veeaming since 2013
ChuckS42
Expert
Posts: 189
Liked: 27 times
Joined: Apr 24, 2013 8:53 pm
Full Name: Chuck Stevens
Location: Seattle, WA
Contact:

Re: NBD backups when they should be NFS

Post by ChuckS42 »

Spoke too soon; my tests were good but with regular backups last night the exact problem came back... hopefully the engineer assigned to the case can find something.
Veeaming since 2013
mfc
Enthusiast
Posts: 26
Liked: 3 times
Joined: Sep 17, 2013 12:48 pm
Full Name: Mike Czajkowski
Contact:

Re: NBD backups when they should be NFS

Post by mfc »

I'm having the same issue (using Veeam and Nuntanix just as OP), support case #04207553. One thing I noticed, only Windows VMs will use NBD. Linux VMs are still backed up properly using DirectNFS.

I had this all configured and working properly before upgrading to v10.
mfc
Enthusiast
Posts: 26
Liked: 3 times
Joined: Sep 17, 2013 12:48 pm
Full Name: Mike Czajkowski
Contact:

Re: NBD backups when they should be NFS

Post by mfc » 1 person likes this post

FYI, it seems that the V10 or CP2 update wipes out the following registry value on the backup proxies:
vmware-vsphere-f24/veeam-on-nutanix-and ... ml#p233862

Once I put the value back on all proxies they started using NFS again for all VMs.
PetrM
Veeam Software
Posts: 3229
Liked: 520 times
Joined: Aug 28, 2013 8:23 am
Full Name: Petr Makarov
Location: Prague, Czech Republic
Contact:

Re: NBD backups when they should be NFS

Post by PetrM »

Hello,

@mfc Thanks for sharing the information with us!

@ChuckS42 I've checked the support case 04195709 on our ticket tracking system and I see that the case has been closed.
Did you have a chance to open the new one? Please don't forget to provide us with ID of the new case.

Thanks!
ChuckS42
Expert
Posts: 189
Liked: 27 times
Joined: Apr 24, 2013 8:53 pm
Full Name: Chuck Stevens
Location: Seattle, WA
Contact:

Re: NBD backups when they should be NFS

Post by ChuckS42 »

The new case is 04207220.

Also, I have verified all my proxies do have the appropriate registry key (I redeployed them all after the v10 upgrade.)

What seems to be happening is that those three proxies (01, 05, and 06 in Seattle; four different ones in our remote site) cannot access the local Nutanix storage, throwing them into NBD. It doesn’t seem to be the host, because if I move the proxies around, the problem follows the proxy.

The proxies are configured with two network interfaces, the primary facing our backup network (where the Veeam servers, repositories, and storage live), and the second on the local iSCSI interface for Nutanix (192.168.5.x). Each proxy has a unique address on that local network.

I’m tempted to fiddle with the iSCSI IP addresses to see if that makes a difference.
Veeaming since 2013
nitramd
Veteran
Posts: 297
Liked: 85 times
Joined: Feb 16, 2017 8:05 pm
Contact:

Re: NBD backups when they should be NFS

Post by nitramd »

Chuck,

Any idea what version of NFS you're running connecting to the Nutanix cluster?

Also, what's the speed of you network?
ChuckS42
Expert
Posts: 189
Liked: 27 times
Joined: Apr 24, 2013 8:53 pm
Full Name: Chuck Stevens
Location: Seattle, WA
Contact:

Re: NBD backups when they should be NFS

Post by ChuckS42 »

I think it supports NFS v3 and v4. Not sure exactly.
Our hosts have dual 10GbE uplinks.
Veeaming since 2013
nitramd
Veteran
Posts: 297
Liked: 85 times
Joined: Feb 16, 2017 8:05 pm
Contact:

Re: NBD backups when they should be NFS

Post by nitramd »

The reason I ask is that sometimes the NFS version is 3 which a history of issues, especially with NFS lock stun.
PetrM
Veeam Software
Posts: 3229
Liked: 520 times
Joined: Aug 28, 2013 8:23 am
Full Name: Petr Makarov
Location: Prague, Czech Republic
Contact:

Re: NBD backups when they should be NFS

Post by PetrM »

Hello,
ChuckS42 wrote:those three proxies (01, 05, and 06 in Seattle; four different ones in our remote site) cannot access the local Nutanix storage
It explains the failover to NBD. Looks like a deeper investigation at the level of infrastructure is required in order to find an origin of the issue with an access to the local Nutanix storage.

Thanks!
ChuckS42
Expert
Posts: 189
Liked: 27 times
Joined: Apr 24, 2013 8:53 pm
Full Name: Chuck Stevens
Location: Seattle, WA
Contact:

Re: NBD backups when they should be NFS

Post by ChuckS42 »

Clarification: The proxies in question CAN access the local Nutanix storage fine. However, the logs suggest that somehow Veeam cannot correlate the proxy and the VM being on the same host, so it goes NBD.

Code: Select all

[ProxyDetector] DifferentHostDirectNFS proxy [proxy01.domain.net] becomes NBD because of EDirectNFSMode.SameHostOnly
The proxy and the VM being backed up ARE on the same host. Odd.

There's also this, which I get for all three of the troublesome proxies:

Code: Select all

[ViProxyVmSearcher] Can not find VM for proxy host proxy01.domain.net:17df7b3c-d003-4226-a5f8-fd04501f86da by bios uuid 05520242-618E-33C3-FAA1-96B6061FCF29
Thoughts?
Veeaming since 2013
PetrM
Veeam Software
Posts: 3229
Liked: 520 times
Joined: Aug 28, 2013 8:23 am
Full Name: Petr Makarov
Location: Prague, Czech Republic
Contact:

Re: NBD backups when they should be NFS

Post by PetrM »

Hi Chuck,

I suppose that the reason is represented by the first pasted log snippet. I recommend to wait for the reply from our support team, I'm sure that our engineers will be able to explain the behavior.

Thanks!
ChuckS42
Expert
Posts: 189
Liked: 27 times
Joined: Apr 24, 2013 8:53 pm
Full Name: Chuck Stevens
Location: Seattle, WA
Contact:

Re: NBD backups when they should be NFS

Post by ChuckS42 »

Interestingly, if I disable network failover for all the proxies, it all works as its supposed to - all backups use NFS. However, this means if there's a legitimate reason to not use NFS (like a pre-existing snapshot), the backup will fail. The engineer I'm working with suspects some sort of proxy selection logic problem.
Veeaming since 2013
PetrM
Veeam Software
Posts: 3229
Liked: 520 times
Joined: Aug 28, 2013 8:23 am
Full Name: Petr Makarov
Location: Prague, Czech Republic
Contact:

Re: NBD backups when they should be NFS

Post by PetrM »

Thanks for sharing this information, let's wait for the final conclusion.

Thanks!
ChuckS42
Expert
Posts: 189
Liked: 27 times
Joined: Apr 24, 2013 8:53 pm
Full Name: Chuck Stevens
Location: Seattle, WA
Contact:

Re: NBD backups when they should be NFS

Post by ChuckS42 » 1 person likes this post

Finally fixed it. We found that certain proxy servers in our local and remote datacenters were duplicating their bios_uuid numbers in the database. After raising the log level to Ludicrous Logging (6) we found they were seeing getting IP addresses mixed up (the primary backup network addresses). Suspected some sort of DNS problem, ruled that out.

I got to thinking about IP addresses, and realized that for the second interface, the one for local NFS connectivity (192.168.5.x), I was duplicating addresses for proxies in both datacenters (e.g., Local Proxy 1 and Remote Proxy 1 = 192.168.5.11, Local Proxy 2 and Remote Proxy 2 = 192.168.5.12, etc.). Nothing wrong with this, these addresses are completely local to the host. However, it seems that Veeam B&R has been grabbing those addresses to uniquely identify proxy servers, getting confused, over-writing bios_uuids, and breaking Direct Access.

So I changed the addressing used for the remote datacenter proxies to make them unique, removed and re-added the proxies with incorrect bios_uuid entries in the database, and BAM!, problem solved. No more dupe bios_uuids, and everything's running NFS as it should.
Veeaming since 2013
PetrM
Veeam Software
Posts: 3229
Liked: 520 times
Joined: Aug 28, 2013 8:23 am
Full Name: Petr Makarov
Location: Prague, Czech Republic
Contact:

Re: NBD backups when they should be NFS

Post by PetrM »

Hi Chuck,

I'm glad to hear that the issue is resolved and many thanks for sharing this information with us!

Thanks!
Post Reply

Who is online

Users browsing this forum: RobTurk and 91 guests