Unreliable Hotadd with vSphere 6.7 and 9.5 U3a (Plus Hotfix)

[solveit]

Hi guys,

We have a case open (03268440) which has been troubling us for some weeks since moving from vSphere 5.5 to 6.7.

Our customer has a tight Replicas & Backups window and everything was perfect with 9.5 on vSphere 5.5 and even NBD speeds were good over the 40Gb Infiniband network infrastructure.
Since moving all the VM's to a new 6.7 we get sporadic HotAdd issues where the same Replica job one night will use HotADD and the next night tries to use HotADD but eventually fails to NBD.
with 'Unable to hot add target disk, failing over to network mode'
There is no consistency in it. We installed the hotfix from https://www.veeam.com/kb2711 yesterday but this still hasn't fixed it. It seems to have improved things slightly, but we had two jobs just stick part way through and not continue past the next disk. This has never happened before.
It's usually The target disks that proxies failover on as Source disks use NFS. I've checked and can always mount the Target disks on the proxy the next day.
Could it be timing out too early possibly?

We have 5 hosts with a proxy on each. They read NFS from the shared storage and write with Hotadd to local storage on the hosts. NBD has slowed to 35-45MB/s and I've checked to ensure there are no bottlenecks here. We mostly read from proxy on e.g. Host01 (NFS) and write through proxy on e.g. HOST02 (Local VMFS Storage). This reduction in speed seems it might be down to NBDSSL from other forum postings. If we could improve NBD speed that would at least help in the short term.

The advice we got from support was to force the target proxies to use virtual appliance mode. We're not keen on this solution because it means some replicas will fail as they can't use hotadd and to overcome this we would therefore need to create a dedicated NBD Proxy and point those jobs to that manually. We'd much prefer it to be like it was before with 5.5 where hotadd works reliably and automatically.

I look forward to your help.

Steve

[foggy]

Please continue investigating the issues with support, they will be able to identify the reasons of transport mode failover.