vSphere 7 hotadd issue

[ac0lada]

I've added 2 vSphere VMs as VMware backup proxy in Veeam, to use them in hotadd mode.
In all the VMware backup jobs we have the same error message, regardless of the guest OS, virtual disk controller, or datastore conectivity(FC scsi or nvme-Of).

"Unable to hot add source disk, failing over to network mode..." (in the console)
VDDK error: 3 (One of the parameters was invalid). Value: 0x0000000000000003

Other relevant log entries:
in c++: Failed to connect to the virtual disk library
in c++: Failed to execute VDDK command 'ConnectEx'
in c++: Failed to create VM connection.
in c++: Failed to get physical drives for vddk disks

[DiskLinkProvider] Failed to attach disk 32000 to source proxy <proxy name>

Unable to hot add source disk, failing over to network mode...

We already have a support ticket #05499972 and we have checked all the requirements from https://www.veeam.com/kb1218 and we have added all the necessary AV exclusions.

We have a vpshere cluster and we get the above error message with the VM hotadd proxy and VM to be backed up on the same host.
There is only one datacenter and vcenter configuration and the hosts have acces to the datastores.
Also the VMs were added to the backup job through VC and the entire VMware infrastructure was added to Veeam B&R server also through VC connection.

No issues with replication jobs with a VMware virtual proxy used as a proxy for the target(the target ESXI is standalone, not in a cluster, but added to the same VC).

There is a rather funny mention in the above KB article:

"In the other situations, however, Hotadd/Virtual Appliance mode is not applicable or possible given the current VMWare architecture."

If that is the case, I would still like to know what is the cause that prevents usage of hotadd conectivity.

[HannesK]

Hello,
if you really want to know it, then I can only recommend to continue working with support. Because support has the log files and can give definite answers on such topics.

For that kind if technical issues we can do some guesses on a forum, but "knowing", can only come from support

Best regards,
Hannes

[ac0lada]

I am not looking for guess answers. Only hoping that someone else had this issue or something similar, that could help. Or some ideas on things that were possibly overlooked.
Knowing doesn't always come from support. It's a difficult process: upload logs, update to latest version, logs again, anything modified - upload again, than comes the off topic errors hunting and so on.

But we can only hope that some one up there "knows".
Those "other situations" that are not mentioned in the KB.

[ac0lada]

I'm glad to declare the issue resolved.
It was kb 2159 https://www.veeam.com/kb2159 and kb1054 https://www.veeam.com/kb1054, option 9 and not KB1218(the actual error) that provided the actual solution.

We had the VMware proxy VM added in a replica job and the target host was managed by the same vcenter.
As soon as we removed the VM from the replica job and deleted the replicated VM and the files from the target datastore the hotadd mode started working and no more "failing over to network mode".
The issue is still not resolved in v11a, at least with an indication in the error logs that hotadd cannot work, if the proxy VM is being replicated in the same vcenter managed infrastructure.
The logs indicate correctly that the replicated VM has the same UUID and it's powered off and will be skipped(as expected), but no indication that this could make hotadd not functional.

Another possible solution could be an additional verification when installing VMware backup proxy on a VM being replicated, in order to verify if the VM has a replica on a target host, managed by the same vcenter.
Or a note in kb1218 that a possible solution could be found in kb2159.
Also in kb1218 it would be better to write "Veeam Backup Proxy VM" instead of "Veeam VM" to avoid possible confusions(eg. it could be confused with Veeam B&R server and not the actual proxy vm).

Best regards.