v11 appears to be ignoring preferred network for vmdk files but not others

[brooksbridges]

Case #04869368

I just recently upgraded to v11 and have run into a very strange issue that I cannot resolve. My server running Veeam is on my LAN only, and does not have direct access to my storage network.

After upgrading, a job that worked just fine in v10 with a preferred network configured for my LAN network has suddenly stopped working with the error that it cannot download the vmdk file. Upon digging into the logs, I see successful downloads of all the other files for the VM (nvram, etc) over the LAN network, however when it gets to the vmdk for the snapshot, it all of a sudden changes to trying to download that file over the storage network, which it can't reach. Predictably, this causes a timeout and eventually the job fails.

I have done a complete removal and reinstallation of Veeam, a complete strip and manual reconfiguration, disabled ipv6 on all interfaces (including the registry tweak), and nothing changes.

Thoughts? Is this possibly a bug?

[soncscy]

I don't think Preferred Network applies for the Proxy => Host connection:

https://helpcenter.veeam.com/docs/backu ... ml?ver=110

Preferred network rule applies only to traffic between the following backup infrastructure components:

Backup server
WAN accelerator1
Gateway server2
Backup proxy
Backup repository
Log shipping server
Tape server
Storage system (backup from storage snapshots scenario)3
Veeam Agent

So it shouldn't try at all, and this comes down to DNS and routing in your environment, not to the preferred networks.

If you nslookup the host from the proxy (and vice versa), which IP is returned first in both cases? I believe that's going to be the answer here, and probably either DNS editing or /etc/hosts edits are in order.

[brooksbridges]

The only value in DNS for those hosts is the LAN IP. As I said, it finds those servers with that IP just fine for the nvram and other files, it's only when it goes to get the vmdk that it suddenly decides to use the storage network IP. The only way it could possibly even know about that network is if the esx server is telling it about it, but as I said, I've changed nothing on the VMware side between old and new versions.

[soncscy]

Well, again, what does nslookup show?

Looking at just symptoms isn't the right way to do it -- when we understand it must be getting the network connection info from somewhere, you have to check what it's getting.

Clearly, something has changed even if you're not aware of it So start with the basic troubleshooting.

[brooksbridges]

Yes something has changed. I upgraded to v11.

The storage network does not exist in my DNS server. Nothing in my network knows about the storage network except the 2 esx servers and my nas. The DNS server has no knowledge of it, and couldn't return those IPs if it wanted to.

I will say again for the 3rd time, please listen this time... Veeam is using the LAN IP, that is in DNS, to download the nvram file of the snapshot, but then suddenly switches and tries to use the storage network IP, that does not exist anywhere in DNS, to download the vmdk file.

I've already been through all the basic troubleshooting for network issues, yes including verifying DNS, so please... The horse is dead.

[tsightler]

It's not clear to me, but are you using NBD mode? If so, is there any chance that in the VMkernel port properties that the vSphere NFC Backup is selected on the storage network or perhaps not selected on any network?

I've recently seen this on a few v11 installs and it seems to be related to this relatively new vSphere feature, perhaps combined with the newer VDDK used in v11. In one of the cases that I was involved with neither VMkernel interface had the vSphere NFC Backup service selected as available, so it seemed like the VDDK just picked the first interface that was returned in the API, which is different from the old behavior which selected the NFC interface based on hostname resolution. The simple solution was simply to select the checkbox for "vSphere NFC Backup" on the interface the customer wanted the backup traffic to go across and this resolved the issue.

[brooksbridges]

Now we're getting somewhere! I am using whatever mode it defaults to, when I last looked at the config it was set to automatic, so it very well could be. Your description of that issue sounds like it fits the bill pretty dang closely, so let me make some tweaks today and I'll report back. Thank you!

[brooksbridges]

So it appears that the storage adapter was the only one selected for backup NFC. I've changed it to the LAN adapter and now the backup job appears to be working just fine. Nice catch! Thank you for the speedy and useful reply!