Replication failing to copy .nvram file with EFI boot option

[Amarokada]

Hi All

I have an open case currently (#03582747) which has had me and a few VMware engineers scratching our heads.

Environment:

2x vSphere clusters (3 hosts in each), with each cluster in a separate datacentre and connected via VPN/FWs.
Latest vSphere versions (6.7 U2 build 13644319), and latest vCenter (6.7 build 13643870).
Clusters have their own vCenters locally, but they are linked in enhanced linked mode with a single SSO domain.
Veeam is latest 9.5 4a on all components.
Proxy VMs exist on both clusters and are set for hot-add mode.

The issue pccurs when some VMs are replicated between clusters using Veeam. Some VMs replicate fine, but on others the process fails as it tries to read the .nvram file from the source. The destination is left with the .vmx file in the _replica folder on the datastore, proving that at least 1 file transferred fine before the failure.

After trying many things in relation to networking, vcenter permissions, Veeam proxy mode changes, I finally discovered the difference between the working and non-working replication jobs.

All the VMs that failed had been created as v14 or v15 VMs in vCenter. For a Windows 2016 VM this defaults to setting the boot option to EFI with secure boot, and creates a 265KB .nvram file. Prior to v14, the VM uses BIOS boot mode which creates a 9KB .nvram file. I tested creating a v13 VM which replicated fine, then upgraded it to v15 and it continued to replicate fine, however as soon as I switched the VM to EFI boot mode the .nvram file grows to 265KB and it no longer replicates. In all my tests I've been doing replication with the VM powered off to avoid any potential file lock issues.

When reversing the setting from EFI to BIOS, you need to delete the .nvram file while the VM is powered off and then power cycle it in order for the default 9KB .nvram file to be created, after which the replication works fine again.

The issue is there with and without the secure boot option ticked while in EFI mode.

I have at least a workaround to the issue for now, but I wonder if this can be recreated in order to verify it as a bug of some sort.

Regards
Rob

[HannesK]

Hello,
did you test that on the same VCenter? I had no issues changing from v13 to v15 and from bios to EFI on the same VCenter. Ah well, and I used NBD instead of Hot-Add.

I could also see the bigger .nvram file, but no issues.

Best regards,
Hannes

[Amarokada]

Yes, the tests have been done with the same vCenter connection (source site). I've also tried adding a vSphere host directly to Veeam infrastructure and retesting and the same thing happens. Also forcing NBD mode on the proxies still has the same result.

I'm going to try replication in the opposite direction now, but both of these sites have been built with the exact same hardware and software versions so I'm expecting to see the same thing. Perhaps it needs something in the environment to trigger this issue with the .nvram file (particular hardware/drivers etc).

[Amarokada]

Ok after some more testing and guidance from Veeam, I have re-tried with some new proxies I built on the same network segment as the vcenter/hosts and everything replicated fine. Our original proxies are on a 10Gb network segment and have to route through a firewall to talk to the hosts/vcenter, which is great for backups (as our repo is on the 10Gb segment), but for some reason the NFC port 902 connection between the proxies and the hosts is breaking once the file is bigger than a certain size (.vmx and vmxf files transfer fine, as do 9K .nvram files, but 73k .nvram files won't copy).

So it looks like it's something firewall related or other issue as the packets traverse the 10Gb to 1Gb subnets. I've made wireshark captures to attempt to track it down.