HyperV Server 2019 - Domain Controller backup fails with AAP

[JeWe]

Hi,

we bought new HyperV-Servers and migrated our VMs from a Server 2012R2-Host to Server 2019. Backup with Veeam worked fine before (latest version 9.5.4.2866), now backup of our DC-VMs (two DCs on two different HyperV-Hosts) fails if AAP is avtivated. OS of the DCs is Server 2012R2.

I get this error in Veeam:

Code: Select all

27.01.2020 11:03:53 :: Failed to create VM recovery checkpoint (mode: Veeam application-aware processing) Details: Job failed (''). Error code: '32768'.
Failed to create VM recovery snapshot, VM ID '21e4da00-ea9c-47bf-be62-4b94a307db65'.  

Windows eventlog tells me this (sorry, we got a German version):
Event ID 489, source ESENT

Code: Select all

lsass (616) Versuch, Datei "\\?\Volume{96c221e7-9f29-4670-b0b8-b9b268ff9421}\Windows\NTDS\ntds.dit" für den Lesezugriff zu öffnen, ist mit Systemfehler 32 (0x00000020): "Der Prozess kann nicht auf die Datei zugreifen, da sie von einem anderen Prozess verwendet wird. " fehlgeschlagen. Fehler -1032 (0xfffffbf8) beim Öffnen von Dateien.

Event ID 8229, source VSS

Code: Select all

Von einem VSS Writer wurde ein Ereignis mit dem Fehler "0x800423f4, Für den Generator wurde ein nicht vorübergehender Fehler festgestellt. Wenn der Sicherungsvorgang wiederholt wird,
tritt der Fehler höchstwahrscheinlich erneut auf.
" zurückgewiesen. Änderungen, die während der Verarbeitung des Ereignisses an den Writerkomponenten vorgenommen wurden, sind für den Anforderer nicht verfügbar. Zugehörige Ereignisse der Hostanwendung des VSS Writer finden Sie im Ereignisprotokoll. 

Vorgang:
   PostSnapshot-Ereignis

Kontext:
   Ausführungskontext: Writer
   Generatorklassen-ID: {b2014c9e-8711-4c5c-a5a9-3cf384484757}
   Generatorname: NTDS
   Generatorinstanz-ID: {f912dc7e-a386-4740-996c-abf60c3d4e5a}
   Befehlszeile: C:\Windows\system32\lsass.exe
   Prozess-ID: 616

Event ID 2, source vmicvss

Code: Select all

Fehler bei VSS Writer NTDS mit dem Status 11 und dem Writer-spezifischen Fehlercode 0x800423F4.

Workaround:
I have to restart the Windows services "COM+ Eventsystem" and "Volume Shadow Copy" (hope, the names are correct, as I said, we have an German version). Even if I rebooted the DC before backing it up, I have to restart the services before.

Is this a known bug? Windows Server version is 1809 (Build 17763.973). Windows related or Veeam related?

Regards,
Jens

[Egor Yakovlev]

Hi JeWe,
VSS errors are quite common in a Microsoft Hyper-V backup world, because most(if not all?) backup solutions rely on a working VSS subsystem. Errors mentioned in the Veeam and Windows logs clearly state that there is a problem with VSS(specifically NTDS Writer), and the restart of services mentioned above usually fixes it(as confirmed by you).
- I would start with a quick check on a Hyper-V Integration Services and VM Versions after migration. Those shall be upgraded properly upon VM migration from older to newer Hyper-V version. In Hyper-V server 2019, correct Hyper-V Integration services shall arrive as normal Windows Updates, if those are enabled, however VM Version upgrade is a manual task.
- I would also check with "vssadmin list providers" to make sure official Microsoft VSS Provider is the only one being used on said DCs - in many cases there are leftover "presents" from previous backup solutions.
- please also check this KB which might shed some light on the NTDS writer problem roots
- you can also schedule a service restart script, so you won't have to do it manually after server reboot
- last but not least, check with Microsoft up on a fix for the problem.
Hope that helps!

[JeWe]

Hi Egor,

thanks for your help.

- I would start with a quick check on a Hyper-V Integration Services and VM Versions after migration. Those shall be upgraded properly upon VM migration from older to newer Hyper-V version. In Hyper-V server 2019, correct Hyper-V Integration services shall arrive as normal Windows Updates, if those are enabled, however VM Version upgrade is a manual task.
>> Already done that, HyperV-Manager shows version 9.0 (before upgrading it was 5.0)

- I would also check with "vssadmin list providers" to make sure official Microsoft VSS Provider is the only one being used on said DCs - in many cases there are leftover "presents" from previous backup solutions.
>> Never backed up this VM with any other solution than Veeam. Listing shows no error, if I don't restart the services after reboot, the error shows after I tried the backup with Veeam.

- please also check this KB which might shed some light on the NTDS writer problem roots
>> Already done that, no luck with that.

- you can also schedule a service restart script, so you won't have to do it manually after server reboot
>> Sure

- last but not least, check with Microsoft up on a fix for the problem.
>> Will post this also in Microsoft forum.

[seckinhacioglu]

Hello JeWe;
We had the same problem after the Physical Servers we used were upgraded to 2012 R2 in 2019. There are actually 2 different updates / changes that need to be done with the subject.
The first one is to update "Configuration Version" on "Hyper-V" (Detailed article on the subject is available on the web page below.) "Https://robertsmit.wordpress.com/2019/0 ... rver-2019- upgrade-virtual-machine-version-in-Hyper-V-Hyper-V-WinServer-hybrid / "
For the second update / upgrade, for "Object Item Backup", "vmbus.sys" update is required if it is in the virtual server to be backed up. The detailed article about this is available at (https://msitpros.com/?p=3983). When you have successfully done all the steps, you will see that the backup is completed without any problems. Also, I recommend you to make a "Normal Backup" to avoid possible problems before the second operation.

[JeWe]

Hi seckinhacioglu,

thanks for your response. Not sure, what is wrong here...

Configuration version is already updated.
Your link (https://msitpros.com/?p=3983) refers to Server 2016, I should check vmbus.sys within the VM, right?

I used this to check the version of Integration Services:
REG QUERY "HKLM\Software\Microsoft\Virtual Machine\Auto" /v IntegrationServicesVersion
--> Gives me 6.3.9600.18692 on DC.
--> OK, I checked another server VM with 2012R2, this one is backed up without any issue: 6.3.9600.18692.
--> And a third one, also backed up without any problems: 6.3.9600.18907. Seems that the first and the second one are not up to date?!?

Check vmbus.sys via file properties --> Details
DC: 6.3.9600.18080
Second one: 6.3.9600.18692 (not sure if this version should be identical to Integration Services Version)
Third one: 6.3.9600.18692

So I downloaded this one:
https://www.microsoft.com/de-DE/downloa ... x?id=48204

Can't install, I'm getting "Error: 775 Die Implementierung kann die Anforderung nicht ausführen". KB2919355 as a prerequisite is installed.

Seems to be another one of "Microsoft Finest's"... Also read this and also stumbled with 2012R2 earlier, as with some VMs, Hyper V Manager says, Integration Services aren't up to date. Nevertheless I found no way to do this. On the other hand I didn't encounter any problems, so I ignored it.
https://community.spiceworks.com/topic/ ... ft-support

Maybe you have another idea, else I'll live with this for the next few weeks and then install two new 2019-DCs...

[JeWe]

Hi,

got an update, problem solved.

We have another 2012R2-VM, freshly installed. When I try to backup with Veeam, I got this:
Unable to allocate processing resources. Error: Failed to take in-guest VSS snapshot COM error: Code: 0x80042308

Although this error message differs from the one above, the cause is the same: vmbus.sys isn't up to date. So I installed all the updates I could get via Microsoft update and now have, so it seems, the latest one: 6.3.9600.18907. Backup now works fine.
Whatever I do, version from our two DCs remain at 6.3.9600.18080. Solution: I simply copy the file vmbus.sys from the working machine to both DCs. Also works now, I don't have to restart any services after reboot now. Assuming you got a working backup, just in case...

Thanks to @seckinhacioglu for pointing the right direction!

P.S.: I also made opened a threat in Microsoft forum, unfortunately I didn't get a helpful response there. Instead of that they posted the link from here and marked it as solved though it wasn't solved at this time. Confession of failure IMHO. So what, I learned that I got answers in Veeam forum, not in Microsoft forum. Thanks!

[Egor Yakovlev]

Thanks for update and sharing solution!
Glad it worked out after all.
/Cheers!

[seckinhacioglu]

In the scenario of upgrading existing Windows 2012 R2 Edition hosts to 2016 or 2019, I am sure this error will happen to all system experts. So it is very important to solve the problem and share it with us. Thank you

[nfma]

Just a quick success story: We had the same issue (Error 32768) with a 2012 (non-R2) DC on a new 2019 Hyper-V but managed to correct it by installing the guest integrations from a vmguest.iso of an up2Date 2012R2 Hyper-V

Windows Update (Online) couldn't find any newer integrations for that 2012, so we had to "steal" that vmguest.iso from somewhere else since 2019 doesn't have vmguest.iso anymore & Microsoft - for whatever reason - likes to hide this stuff.