Hotadd fails after ESXi host reboot.

[crond]

Hi All,

An ESXi 7.0.3 host is running VCSA and and latest VBR11 in a Windows 2019 VM.

Host went through reboot to install additional network card, nothing else has changed. But now all VM backup and replications jobs fail with

Code: Select all

Processing Error: Failed to open VDDK disk [[XYZ.vmdk] ( is read-only mode - [true] )
Failed to open virtual disk [YXZ.vmdk (flags: 4)
Logon attempt with parameters [VC/ESX: [Login];Port: 443;Login: [];VMX Spec: [moref=vm-];Snapshot mor: [snapshot-443];Transports: [nbd];Read Only: [true]] failed because of the following errors:
Failed to open disk for read.
Failed to upload disk.

Searching KB and forum leads to severa discussion and 2 KBs like https://www.veeam.com/kb1054
I verified UUIDs but other than that I don't think anything is applicable to my case.

Adding a new backup jobs with new source VM results in the same failure.
As a test veeam user was added to "Administrators" role on VCSA with the same results.
No errors in the VCSA or host logs. Snapshots are successfully created / removed.

Any recommendation how I can further localize the issue ?

[Andreas Neufert]

OK I would say the factor that you use "NBD" which uses the VMKernel interface and that you have installed a new network card are an indication that this is somehow related.

Can you share what you have done with the network card added to the ESXi? How did you configured it?

Check on the vcenter and klick on the ESXi host. What FQDN name is listed there (even when you have added the ESXi by IP, the backup APIs will use the FQDN. Check if you can PING this FQDN from Veeam Backup server and from the Veeam Proxy.

Restart the Veeam Service and the Veeam vcenter Cache Service (don´t know the exact name for now, but if in doubt reboot all Veeam* Services on the Backup server). This will force Veeam to build a new VMware topology in it´s cache.

[PetrM]

Hello,

Also, according to the report snippet, there is an attempt to open a disk for read in Network mode, but not in HotAdd:
"Snapshot mor: [snapshot-443];Transports: [nbd]"
So far, it's unclear how the issue can be related to HotAdd. I would check what Andreas has suggested and open a support case so that our engineers can analyze debug logs. Please don't forget to share a support case ID over here for our reference.

If you want to try HotAdd mode, make sure that all the requirements are met and you can HotAdd disks manually in vSphere client.

Thanks!

[crond]

Apologies for misleading snapshot of error logs.

To avoid any doubts

* All jobs are configured to use "HotAdd" and were working correctly for some time. This includes ESXi VM backup and replication jobs.
* VCSA, VBR VM and source VM are on the same esxi host and are using local (directly attached) storage
* All requirements (https://www.veeam.com/kb1054) are satisfied.

After reboot/HW changes "hot add source disk" no longer works but from Job perspective it is not considered to be a failure. I can see an information message that Job is "Unable to hot add source disk" and it falls back to network mode which in turn also fails.

I see successful login of veeam user account on VCSA side and no error messages.
Snapshot is taken successfully and there are several NFC messages related to operation performed by Veeam on a source VM/disks all market "Successful"

At this point I'm trying to understand how to debug "Unable to hot add source disk" part.

[crond]

Hmm looks like VBR sends set of commands to default default IP address (assigned to VMK with default TCP/IP stack) for ESXi host. At some point veeam starts to send commands to a different IP assigned to new VMK adapter (default stack with only vMotion service enabled) which VBR has no access to. All subsequent VDDK request never reach host.

So I guess it's borderline VMware/Veeam question specifically for VDDK. I.E how to force VBR to use a specific IP address to communicate with esxi host

[crond]

Issue is resolved.
Removing backup NFC service from new VMK solves the problem.

Still not clear why Veeam switches form one IP address to a different one mid flight, but I suspect it can be more VMware/VDDK type of question

[Andreas Neufert]

Yes, this is a correct assumption. The VDDK (and so Veeam) will follow this setting. It was a PowerCLI configurable option since a longer time.