Windows Server 2019 Backup Server and Backup Proxy

[jossmi]

I recently installed Veeam Backup & Replication 9.5 Update 4a on a Windows Server 2019 VM running on VMware ESXi 6.7 with the latest updates. The VM is a Veeam Backup Server and Backup Proxy. I noticed that every backup job generates at least 100 events with Event ID 225 in the System event log. I'm not too concerned about this because I observed the same behavior on another Veeam Backup Server and Backup Proxy that is running Windows Server 2016. However, I also noticed that I am notified that I need to restart the VM to finish setting up the LSI Adapter, SAS 3000 series, 8-port with 1068 device every time I login. I confirm this notification is caused by Veeam by rebooting the backup server and running a backup. As soon as I logout and login after running a backup, I receive the notification again. It seems that Veeam's interaction with the LSI Adapter on Windows Server 2019 triggers something that cause the device manager to think a change has been made.

I have a case open with Veeam support, 03571111. They suggested switching from hot-add to network proxy OR switching from the LSI Adapter to the VMware Paravirtual SCSI driver. I am reluctant to switch from hot-add to network proxy because I don't think the client's 1 Gb network will perform as well as its 16 Gb FC SAN. I would consider switching from LSI to the VMware Paravirtual SCSI driver, IF it is fully supported with Hot-Add on a backup server that is also a proxy server.

Has anyone else seen this problem with Windows Server 2019.

[Vitaliy S.]

I was able to find a similar case in this topic, but nothing useful. Since hot-add mode triggers these events, do you see the same behavior when you perform "hot-add actions" manually? For example, add a virtual disk, read some data from it and then remove it?

[tsightler]

I would consider switching from LSI to the VMware Paravirtual SCSI driver, IF it is fully supported with Hot-Add on a backup server that is also a proxy server.

Is there some reason that you believe it is not? VMware has recommended the PVSCSI controller for hotadd operations since VDDK 6.5 and I'm not aware of any issues with it. It's pretty low risk to try it as the steps to convert work in both directions (LSI->PVSCSI and PVSCSI->LSI) so if something goes wrong you can just go back.

[jossmi]

I tried the PVSCSI controller yesterday and it did not resolve either problem. I still see over 100 225 Events for each backup in the System event log, which I can live with, and now I am prompted to reboot the server to complete the installation of the PVSCSI driver repeatedly, which is quite annoying.

[skrause]

Have you added additional SCSI adapters (LSI and PV) to your Hot Add Proxy and reinstalled VMware Tools?

As far as PVSCSI being supported, I have been using them as the SCSI controller on every VM in our infrastructure for 4+ years (5.5 through 6.5 ESXi) and had no issues from a Veeam or VMware perspective.

[jossmi]

I have previously reinstalled VMware tools. I have not created additional SCSI adapters, LSI or PV. This is a small customer and additional adapters are not really needed. I rebuilt the server from the ground up yesterday and I still have the problem. I'm pretty sure the problem is with Windows Server 2019. Veeam support is still working with me on the case.

[skrause]

The reason I asked is that VMware Tools may not install the proper drivers for the SCSI adapters if they are not present in the system at the time the installer runs.

[benny_rt2]

@jossmi

I've been seeing this same situation with the "Restart required - LSI Adapter, SAS 3000 series, 8-port with 1608".

ESXi 6.7 update 1 (build 11675023)

Veeam Backup Recovery 9.5 u4 (9.5.42753) Community Edition - installed in VM running Windows 10 1903.

Restarting doesn't take care of it as the "Restart required" always resurfaces again.

I'm fairly new to Veeam, having just begun using vSphere and Veeam just this year. Being new I just figured it was something I would live with and I've basically been ignoring it. It still works fine, just annoying getting that message.

Today I finally decided to research this and I found your post. Have you or Veeam found the cause or solution?

The VM from which I installed Veeam originated from a Windows 10 VM that I originally had as a guest on my laptop inside Workstation Pro 15. I uploaded that Workstation VM to my ESXi host. In doing that the vmconverter automatically chose the "LSI Logic SAS" as the SCSI controller.

I've not tried changing from "LSI Logic SAS' to the "Paravirtual" but it sounds like that didn't solve it for you. I may try it anyway just for kicks. I may also try to update my VMtools in that VM. I'm pretty sure it is already up to date (vmtools version 10.3.2, build-9925305) .

Anyway, just checking to see if you found anything since your last posting.

Thanks,

Benny

[jossmi]

The issue has not been resolved. I had a joint conference call with Veeam and VMware on 7/11. Issue is currently with VMware escalation engineers, SR 19006654606. Veeam ticket is 03571111.

[benny_rt2]

Thanks for your reply @jossmi.
I'll monitor your thread to see if your escalated ticket will find resolution.

I tried a couple things without success over the weekend:

I uninstalled VMtools in the veeam guest
I shutdown the guest and changed setup SCSI controller from LSI Logic SAS to be Paravirtual.
Booting up the VM I had bsod. tried booting it up a few more times and always bsod.
I changed back to LSI Logic SAS in the settings.
Booted again and no more bsod.
Downloaded latest vmtools from vmware website. Installed vmtools with "complete" option.
And today I still get the "Restart Required" because of the LSI adapter.

[skrause]

Did you add an additional PVSCSI adapter while the drive is still using the LSI and made sure windows had the drivers installed before you changed the type of adapter?

[benny_rt2]

@skrause
I see. Thank you for the detailed instructions. I'll do as you recommend and create "additional" paravirtual SCSI adapter (without applying the HDD to it), and then perhaps re-install vmtools again to be sure the PVSCSI drivers are installed before switching solely to the PVCSI adapter for the HDD.

I'll report back how that goes.

Thanks,

Benny

[benny_rt2]

-I've added "additional" PVSCSI controller (the veeam guest single HDD is still using LSI controller)
-Booted without getting a bsod. Windows device manager at this point sees the LSI adapter driver but not the PVSCSI adapter driver.
-Uninstalled vmtools. Reinstalled vmtools. Now Windows sees both the LSI and PVSCSI adapter drivers.
-Shutdown guest. Changed setup so the veeam guest HDD using the PVSCSI controller instead of LSI controller.
-Booted without getting a bsod.
-Started up a test backup in veeam and then signed out while it continues in background. NOTE: it has always been the "requires restart" appears after signing back in after a backup has performed.
-Waited to allow Veeam to attach the disks from the guest system that is being backed up. Then I logged back in. I still see the "Requires Restart - LSI Sas" warning. That is kindof expected though since I never removed the LSI controller . I will now shutdown veeam and remove the LSI controller. Then I will try another test backup with ONLY the PVSCSI controller installed.

*I'll report back after I test with using only the PVSCSI controller installed while the LSI controller completely removed.

[benny_rt2]

After doing the steps in my previous post and then also removing the LSI Logic SAS controller (SCSI Controller 0), leaving only the PVSCSI controller (Controller 1), I retested to see if I would get the "Restart Required" messages after a backup job completed.

I started up a backup job, then I let that run in background while I signed out of that VM.

In vCenter I watched the Veeam VM attach the disks of the guest it was backing up. I waited for that backup job to complete. Then I logged into the Veeam VM. It's at this point when I'd log in to the Veeam VM Windows OS that I'd see the "Requires Restart". I'm no longer getting that message (not for the LSI SAS or for the PVCSCI controller). That's good!

I'll continue to monitor this as my scheduled backup jobs run nightly. I'll see if this resurfaces, but I'm trusting my problem is solved. Thank you @skrause for my solution and especially to @jossmi for bringing attention to this issue.

Please note however that I am not the OP of this thread. The OP @jossmi still has this issue and with an open ticket. I do not want to detract from Jossmi's issue.

I'll post back tomorrow to confirm if my issue has re-surfaced by then or not.

Thank you,

Benny

[benny_rt2]

Confirmed again this morning, this problem no longer plagues me.

In short, what I did ...
- added additional PVSCSI controller (controller1) to the veeam guest but did not initially assign any HDD to it.
- booted into the guest. In Device Manager when viewed by "Device Container" there were two SAS controllers. One is the "LSI SAS" controller and the other was unknown and missing it's driver (this was the newly added PVSCSI controller).
- uninstalled vmtools using windows "Apps & Features".
- Downloaded latest vmtools from vmware website (10.3.10 build 12406962).
- Installed vmtools and I chose install with "complete" option.
- restarted the veeam guestt to confirm driver is seen for PVSCSI in device manager
- shutdown the guest and removed the LSI controller (controller0) from the VM.
- Changed the HDD of the VM to use the controller1 (PVSCSI controller)
- started up the veeam guest and no longer getting "Restart required - LSI Adapter, SAS 3000 series, 8-port with 1608" after running several backup jobs. Also not seeing Restart Required for the PVSCSI controller.

[jossmi]

Veeam has published KB2989 to address this issue, https://www.veeam.com/kb2989