NBD backups when they should be NFS

[ChuckS42]

We have Nutanix clusters running vSphere ESXi 6.5. We are backing servers up with Veeam B&R v10. All our proxy servers are configured for Direct Storage Access with Network failover, built as per Nutanix/Veeam documentation. I've recently noticed that a fair number of servers are using Network mode (NBD) when they shouldn't be; the only reason they should is if there's an existing snapshot on the server when the backup runs, and that is almost never the case. Anyone seen this before? (Note: every time a server backup incorrectly uses NBD it starts with a message about there being no proxies on the ESXi management subnet; ones that use NFS don't give this message. Nothing in our backup infrastructure is on the ESXi management subnet.)

It's not all servers in a backup job, maybe a third of them.

I don't know if this behavior started with v10; I just noticed it last week.

Case 04195709

[ronnmartin61]

Hey Chuck, it sounds as if you have everything set up correctly but just to double-check you do have a Veeam Windows proxy deployed per ESXi server / Nutanix node and each proxy has \\HKLM\Software\Veeam\Veeam Backup and Replication\EnableSameHostDirectNFSMode = 2 set?

[ChuckS42]

Yep, both of those.

[ronnmartin61]

Thinking about this a bit more and I had recently heard of a case where this was failing over to NBD unexpectedly as you're seeing. Though the best practices guide advises "Configure a Veeam Backup & Replication global traffic rule to use the 192.168.5.0 network for backups" (which should be correct), in this particular case removing the traffic rule allowed everything to function correctly.

[ChuckS42]

Hmm, interesting. I do have that traffic rule enabled. I'll disable it and see what happens.

EDIT: Nope, didn't make a difference.

[nitramd]

Hello Chuck.

Take a look at this article: https://www.virtualtothecore.com/protec ... l-proxies/

[ChuckS42]

Interesting article, but nothing new there for me. We aren't using hotadd at all any more.

[ChuckS42]

I've gone back over the Veeam/Nutanix best practice guide (https://www.veeam.com/wp-nutanix-veeam- ... tices.html), and the only place where we diverge is that the ESXi Management VLAN is different from the VLAN used by the public Proxy interfaces, Veeam B&R Server, and ExaGrid repositories. (Different VLANs/Subnets but no firewalls between them.)

[ChuckS42]

Doing some more focused testing, I see that the VM servers that go NBD are always the same... and are on three of my hosts. The ones that run NFS as they should are on the other five hosts. I'm going to swap a couple of my proxies around and see if the problem follows the proxy.

[ChuckS42]

... and the problem followed the proxy. Looks like I need to deploy new proxies.

[nitramd]

Do you have proxy affinity set?

[ChuckS42]

Yes.

[PetrM]

Hi Chuck,

As far as I see you have opened the support case for this issue. Could you please share the results of support case?

The proper inspection of debug logs should reveal the root cause of failover from NFS to NBD.
If you have no clear explanation so far I would recommend to request an escalation of the support case.

Thanks!

[ChuckS42]

There are no results yet; the last thing I got was that case is being escalated to an engineer. No word back yet. In the mean time, I'm going to re-deploy the proxies that seem to have the issue and see if that fixes the problem.

[ChuckS42]

I have replaced the suspect proxy servers and that solved the problem. No idea what was wrong with them; they were all configured properly. Anyway, all-NFS now.

[ChuckS42]

Spoke too soon; my tests were good but with regular backups last night the exact problem came back... hopefully the engineer assigned to the case can find something.

[mfc]

I'm having the same issue (using Veeam and Nuntanix just as OP), support case #04207553. One thing I noticed, only Windows VMs will use NBD. Linux VMs are still backed up properly using DirectNFS.

I had this all configured and working properly before upgrading to v10.

[mfc]

FYI, it seems that the V10 or CP2 update wipes out the following registry value on the backup proxies:
vmware-vsphere-f24/veeam-on-nutanix-and ... ml#p233862

Once I put the value back on all proxies they started using NFS again for all VMs.

[PetrM]

Hello,

@mfc Thanks for sharing the information with us!

@ChuckS42 I've checked the support case 04195709 on our ticket tracking system and I see that the case has been closed.
Did you have a chance to open the new one? Please don't forget to provide us with ID of the new case.

Thanks!

[ChuckS42]

The new case is 04207220.

Also, I have verified all my proxies do have the appropriate registry key (I redeployed them all after the v10 upgrade.)

What seems to be happening is that those three proxies (01, 05, and 06 in Seattle; four different ones in our remote site) cannot access the local Nutanix storage, throwing them into NBD. It doesn’t seem to be the host, because if I move the proxies around, the problem follows the proxy.

The proxies are configured with two network interfaces, the primary facing our backup network (where the Veeam servers, repositories, and storage live), and the second on the local iSCSI interface for Nutanix (192.168.5.x). Each proxy has a unique address on that local network.

I’m tempted to fiddle with the iSCSI IP addresses to see if that makes a difference.

[nitramd]

Chuck,

Any idea what version of NFS you're running connecting to the Nutanix cluster?

Also, what's the speed of you network?

[ChuckS42]

I think it supports NFS v3 and v4. Not sure exactly.
Our hosts have dual 10GbE uplinks.

[nitramd]

The reason I ask is that sometimes the NFS version is 3 which a history of issues, especially with NFS lock stun.

[PetrM]

Hello,

those three proxies (01, 05, and 06 in Seattle; four different ones in our remote site) cannot access the local Nutanix storage

It explains the failover to NBD. Looks like a deeper investigation at the level of infrastructure is required in order to find an origin of the issue with an access to the local Nutanix storage.

Thanks!

[ChuckS42]

Clarification: The proxies in question CAN access the local Nutanix storage fine. However, the logs suggest that somehow Veeam cannot correlate the proxy and the VM being on the same host, so it goes NBD.

Code: Select all

[ProxyDetector] DifferentHostDirectNFS proxy [proxy01.domain.net] becomes NBD because of EDirectNFSMode.SameHostOnly

The proxy and the VM being backed up ARE on the same host. Odd.

There's also this, which I get for all three of the troublesome proxies:

Code: Select all

[ViProxyVmSearcher] Can not find VM for proxy host proxy01.domain.net:17df7b3c-d003-4226-a5f8-fd04501f86da by bios uuid 05520242-618E-33C3-FAA1-96B6061FCF29

Thoughts?

[PetrM]

Hi Chuck,

I suppose that the reason is represented by the first pasted log snippet. I recommend to wait for the reply from our support team, I'm sure that our engineers will be able to explain the behavior.

Thanks!

[ChuckS42]

Interestingly, if I disable network failover for all the proxies, it all works as its supposed to - all backups use NFS. However, this means if there's a legitimate reason to not use NFS (like a pre-existing snapshot), the backup will fail. The engineer I'm working with suspects some sort of proxy selection logic problem.

[PetrM]

Thanks for sharing this information, let's wait for the final conclusion.

Thanks!

[ChuckS42]

Finally fixed it. We found that certain proxy servers in our local and remote datacenters were duplicating their bios_uuid numbers in the database. After raising the log level to Ludicrous Logging (6) we found they were seeing getting IP addresses mixed up (the primary backup network addresses). Suspected some sort of DNS problem, ruled that out.

I got to thinking about IP addresses, and realized that for the second interface, the one for local NFS connectivity (192.168.5.x), I was duplicating addresses for proxies in both datacenters (e.g., Local Proxy 1 and Remote Proxy 1 = 192.168.5.11, Local Proxy 2 and Remote Proxy 2 = 192.168.5.12, etc.). Nothing wrong with this, these addresses are completely local to the host. However, it seems that Veeam B&R has been grabbing those addresses to uniquely identify proxy servers, getting confused, over-writing bios_uuids, and breaking Direct Access.

So I changed the addressing used for the remote datacenter proxies to make them unique, removed and re-added the proxies with incorrect bios_uuid entries in the database, and BAM!, problem solved. No more dupe bios_uuids, and everything's running NFS as it should.

[PetrM]

Hi Chuck,

I'm glad to hear that the issue is resolved and many thanks for sharing this information with us!

Thanks!