Hotadd fails backup when more than 1 scsi controller

[JaySt]

i'm running into an issue with a VMware hotadd proxy, Windows 2012 R2 fully patched, backing up a vSphere 6.5 environment running latest patches. Case id is 02189771
i never seen this one before in previous installs.

the backup fails when more than one SCSI (LSI, SAS) controller is presented to the proxy VM. But only in that case. When the backup is started with only the first SCSI controller active, everything runs fine.
I can see a VM disk being mounted (in disk management) during the backup in the case of 2 SCSI controllers on the proxy, but no read activity is performed at all on the mounted disk. it gets dismounted eventually, and the job fails (or fails over to nbd). I run the exact same job again, only this time with one SCSI controller on the proxy, job runs ok.

Any ideas? anyone seen this before?

More details:
VM hardware version 13
no AV active in proxy
automount is disabled
Veeam version 9.5 update 2

[skrause]

Have you tried VMWare Paravirtual for the second controller?

[JaySt]

no not yet. Is a Paravirtual adapter supported by VDDK/Hotadd? i think this may not be case case when reading https://www.veeam.com/kb1054

Update:
Hmm seems like at some point VDDK best practice reccomendations switched from LSI tot PVSCSI in vSphere 6.5
“Best Practices for HotAdd Transport” formerly recommended the LSI controller over the paravirtual SCSI controller (PVSCSI). Documentation has been corrected; PVSCSI is preferred for HotAdd in VDDK 6.5. The current recommendation is to use PVSCSI only and upgrade to 6.5 or later. If problems occur, HotAdd a dummy disk in case the first HotAdd found an unexpected target/unit on the guest OS"
http://pubs.vmware.com/Release_Notes/en ... notes.html

Not sure if things work when the boot controller is LSI and the second controller is PVSCSI. It might require PVSCSI for the bootcontroller as well.

[skrause]

I have been using the PVSCSI controllers exclusively on my 5.5 environment for 3 years now and have had no issues using hot-add.

[tmtech]

I'm running vbr 9.5 u2 on server 2016 datacenter with one proxy, VMware esxi 6. After lots of troubleshooting final verdict was to remove cdrom and sata controller from vbr vm's.

[doslager]

<bump>

Did anyone come up with a definitive set of hardware to try?

I'm having the same issue...failing over to network mode. New 2016 Proxies, fully patched.

I've tried the following proxy configs:

• Install 2012R2 from template (2xLSI logic SCSI adapters), add VBR proxy and it works fine with HotAdd.
  Change win2016 2xLSI logic to 2xVMware Paravirtual and it fails to network mode.
  Change win2016 to remove 2nd SCSI adapter and it fails over to network mode.

Im starting to think i should just replace my proxies with 2012R2 vms and be done with it. Might be easier than trying to figure out the 2016 mystery. This is my first attempt to using 2016 as a proxy server. All other proxies in my environment are 2012r2 or 2008r2.

And, yes, i did the diskpart automount steps.

Thanks!

[doslager]

So, it seems that changing to the Paravirtual *did* indeed work.

But...it seems that it is only using the 1st PV adapter. It is failing on a HDD over to network mode while the first PV adapter is full. There are 2 other unused PV adapters to hotadd disk to.

To change from existing LSI to Paravirtual i had to do the following:

1-Add Paravirtual SCSI adapter and let 2016 find it and install drivers. (you can hotadd it)

2-Shut down Proxy VM.

3-Change existing 2 LSI logic to paravirtual (must click "confirm change" button in vSphere web client when prompted)--Do NOT remove third paravirtual adapter

4-Power on w2016 and let it sort itself out and you should have 3 paravirtual adapters.

5-You can now power down and remove the 3rd PV adapter. It doesnt appear that the PV are hot removable.

FYI, i'm at the latest version/patch as of this message for both vSphere (6.5up1+) and Veeam (9.5up3). The W2016 template is fully patched as well, with latest version of VMware Tools.

[Didi7]

Ok, let me try to explain, why removing the 2nd LSI SCSI controller from the proxy VM worked here ...

Environment:
1x Windows Server 2016 (Cumulative update 2020-07) VBR-server (v10 P2) with 8GB RAM and 4 vCPUs running on vSphere 6.7 Build 15160138
1x Windows Server 2016 (Cumulative update 2020-07) VBR-proxy (v10 P2) with 4GB RAM and 4 vCPUs running on vSphere 6.0 Build 5572656

I was used to Windows Server 2012 R2 proxies running with 4GB RAM, 4 vCPUs and 2x LSI SCSI controller, therefore I thought, doing the same in Windows Server 2016 works as well.

I got problems using HOTADD-mode failing over to NBD getting errors like 'VDDK error: 13' and 'read-only mode [true]'.

This ...

https://www.veeam.com/kb2008

... didn't help.

Then, I found this ...

https://www.virtual-allan.com/veeam-pro ... -error-13/

... but it wasn't the SATA optical device, it was the 2nd LSI SCSI controller. As soon, as I added a 2nd LSI SCSI controller, HOTADD failed. If I removed the 2nd LSI SCCI controller, it worked.

Strange!

I never used PARAVIRTUAL controllers, as this once wasn't supported, iirc.

Regards,
Didi7