9.5 Hyper-V 2016 Known Issues

[TT-DL]

Also FYI, Windows 2k16 updates can be tracked here.

https://support.microsoft.com/en-au/help/4000825

I've just had a quick read of the August updates and I don't see anything relevant that would solve the backup/prod checkpoint issue.

[rkovhaev]

From my conversations with 2nd/3rd level support with both Veeam and Microsoft I don't think any 2k16 updates will resolve this issue.

MS has a least 2 ways to address the problem:
1) fix old(pre-2016) NTDS/ADAM VSS writers - so that these writers become aware that they should not perform autorecovery steps
or
2) fix 2016 HV server and guest integration components - so that VSS requester in guest creates shadow copy with autorecovery attribute - and this way MS doesn't need to fix VSS writers because writers will perform autorecovery steps just fine

[frankive]

and why doesn't Microsoft do this? what do they answer when Veeam states this?

[hoFFy]

We are experiencing this error on a 2016 Guest OS running a WSUS, not a DC.
Has anybody else those problems with a server not running a DC?

[mkaec]

I experienced similar problems on a 2016 guest running DFS. Turning off AAP seems to have fixed it for me.

[ashfwa]

Veeam support (Case 02296778) has acknowledged that they are unable to confirm that the following issue has been resolved:

Resilient Change Tracking (RCT) returns incorrect changes for VMs which had their virtual disks moved with Storage Live Migration, resulting in corrupt backups.
This is a Hyper-V 2016 issue that Microsoft is planning to address in the future update. Until then, disable the use of changed block tracking for any jobs processing such VMs.

I apologize if I missed it in these 15 pages of posts, but does anyone know if this issue has been addressed or how to track it from the Microsoft side? Presumably Veeam needs to update its Release Notes at each iteration and has this issue earmarked so they know when it can be removed as a "Known Issue", but I am only being forwarded to Microsoft. I cannot find anything that even specifically references this bug...let alone it's fix.

Can anyone offer any further insight?

[markus.s99]

I have the same problem with Hyper-V 2016 host and windows server 2016 VM with SQL Server 2016. (could not initiate a checkpoint operation: %%2147754767 (0x8004230F). Error code: '32768'.)
But 20% of the incremental backups are working...80% fails.

Is that also a "known" bug from microsoft???

After turning off AAP backup is working fine - but not a solution.

Thanks for the update

[Mike Resseler]

Markus,

First: Welcome to the forums
Second: I doubt this is a known bug. Please create a support call and post the case ID here and the follow-up after investigation

Thanks
Mike

[TT-DL]

Some answers.

Production Checkpoints on 2k8/2k8R2 Domain Controllers - Not supported.
See this kb https://technet.microsoft.com/en-us/lib ... s.11).aspx

Exchange 2010
Production Checkpoint of a machine running Edge Transport role (ADAM VSS writer). Not supported.

This is directly from MSoft Premier support.
“Microsoft does not support the use of checkpoints on virtual machines running the Active Directory Domain Services role (also known as “domain controllers”) or the Active Directory Lightweight Directory Services role in Windows Server 2008 or Windows Server 2008 R2. For more information, see Operational Considerations for Virtualized Domain Controllers (http://go.microsoft.com/fwlink/?LinkId=159631).”
*ADAM is former name of AD LDS.

Exchange 2010 backups on hyper-v 2016 with AAP enabled. Works. (As long at edge transport role is not installed) This above would also explain why virtualised SBS etc doesn't work on a 2016 host.

Not sure about production checkpoints on AD 2012 or Exchange 2013 (I imagine exchange 2013 is same as above). I understand Veeam is still investigating/clarifying with Microsoft exactly what is/isnt supported in relation to AD checkpoints.

Many thanks to Rustam and Veeam support team for their assistance in getting some answers on this.

[RGijsen]

In the past I've worked around the Edge-issues by having Veeam trigger a script that stops all Exchanges services on that VM for a while (although if I remember correctly, stopping ADAM is enough) and having it restarted after the backup. Depending on your setup (redundant, balancers and such) there should be no disruption in doing so.

[marieelieg]

Hello dears,

I have the same issue as AlexLeadingEdge. Backup job with App Aware enabled not able to create snapshot on DC VM.
"Job failed (''). Error code: '32768'. Failed to create VM recovery snapshot"

Was there a solution for this?

Thanks!

[Nail.Mukhametshin]

Hi guys!

Question to moderators: If you really have some known issues, can you create a new topic (may be readonly) where in first post you will track all issues and fixes? It will be very helpful not to read 15 pages of this topic to understand about witch issues you speaking.

[pesos]

+1

We have jobs failing at least 2-3 times a month and have to restart all veeam services on our 2016 host to resolve (freshly built server). Curious to know if there are known issues potentially causing this but this thread is out of control :p

[sbaltic]

Same problem here ... host -> w2k16, VM -> 2k16 (not DC, regular file server).

11.10.2017 20:01:26 :: Failed to create VM recovery checkpoint (mode: Veeam application-aware processing) Details: Job failed (''). Error code: '32772'.
Failed to create VM recovery snapshot, VM ID '6604883e-fc6e-4403-b72d-454d3741377b'.

[mkaec]

For my 2012 R2 DCs, I turned off Application Aware Processing and I have the VM settings configured to use Standard checkpoints. If I don't also turn off Guest Quiescence, Veeam reports that the backup was successful, but the server logs an error "could not initiate a checkpoint operation".

I know its been mentioned that AAP is not needed for Veeam AD Explorer. But, that doesn't seem to be entirely true. If I create a new job and put my DCs in it, that job will not show up in the Microsoft Active Directory Object Restore wizard. So, it seems that AAP must be active at least once for Veeam to know the job has DCs in it. If you deploy Veeam fresh into an environment comprised of 2016 hosts with 2012 R2 DCs, it doesn't look like you can get Veeam AD explorer working.

It's been a year since this problem was first documented. I think the note that "this is a Hyper-V 2016 issue that Microsoft is planning to address in the future update" was incorrect. It's really disappointing that this was never resolved.

[nmdange]

It is possible to open AD Explorer without AAP. In the Veeam console, go to Backups->Disks in the left navigation pane. Then search for the DC VM you want to restore from. Select the VM and then on the top menu in the Backup tab, select "Applications Items" and then "Microsoft Active Directory".

However, you still need to have Hyper-V Native Quiescence enabled with production checkpoints to get a consistent backup. I have not had any issues backing up 2012 R2 DCs on a 2016 host with this setup.

[mkaec]

...go to Backups->Disks in the left navigation pane. Then search for the DC VM you want to restore from. Select the VM and then on the top menu in the Backup tab, select "Applications Items" and then "Microsoft Active Directory".

Thanks. I had tried this earlier, but I was in the wrong job at the time. It worked when I tried it again from the correct job.

However, you still need to have Hyper-V Native Quiescence enabled with production checkpoints to get a consistent backup. I have not had any issues backing up 2012 R2 DCs on a 2016 host with this setup.

First, if you have production checkpoints enabled, try manually making a checkpoint in Hyper-V Manager. It should fail.

Then check the Microsoft-Windows-Hyper-V-VMMS-Admin event log around the time of your backups. You should see some events in there being logged about failing checkpoints.

And in your backup job, you might want to try leaving guest quiescence enabled, but turning off the "crash consistent backup" fallback option. This should cause the backup to fail.

From what I can tell, the quiescence checkpoint is failing and the backup is falling back to a crash consistent backup. Veeam just isn't letting us know in the job log that there were problems.

[nmdange]

Well you're definitely right about production checkpoints failing. Looks like I might have to open a ticket with Microsoft on that one

[nmdange]

Well according to Microsoft this specific issue (2012 R2 DC on 2016 host) is a known bug. Waiting on some more details but it sounds like the fix is scheduled to be included in the January Monthly update.

[mkaec]

That would be excellent. Did the source confirm the year of the expected fix? Given that the issue was known in Nov 2016, I would have thought a fix would have been delivered by Jan 2017.

[nmdange]

I meant next January

[gotsch]

Hi everyone, I would like to confirm this great MS Bug, show our situation and maybe someone has a workaround until MS fix this.

Situation:
- 2016 HyperV Cluster, 2 nodes, 105 VMs, most of them 2012R2
- VMs are different customers, so they are all seperated with VLANs. The Cluster and Backup Server itself have no network connection within the VMs.
- Backup Server is a dedicated Server 2016 Machine with Veeam 9.5, fully patched
- Storage is a HBA SAN attached to the cluster nodes

What happens:
- Since our Backup and Cluster has no Network Connection in the VMs, we use HyperV Quiescence for years now, without any problem until 2016 HypeV Hosts. HyperV Quiescence is set in the Backupjob with the fallback option enabled.
- Some VMs Backup may failling, VMs couldn't be accessed anymore, VMs are stucked in Cluster/HyperV with the status "Creating Snapshot..." or "Merging Snapshot...."
- Only Solution ist to move all uneffected VMs to the other Host and then hard reset the failed Host with the failed VM. It is not possible to kill the vmms.exe from the failing VM
- Last two situations where this happens: today in the early hours and 2 days a go. Affected VMs: 2012R2
- It doesn't makes a different if Production Snapshot or Standard Snapshot is set for the VM

MS Support:
MS has confirmed this Bug in our case, the technican has no ETA of a bugfix, so we are all a little bit f***ed up by Microsoft.

Veeam Suport:
Escelated, but they are far behind MS (which means a lot ), the last thing the told us was that it is not the known bug... sure... so no help here.

What I tried:
- I disabled the CSV Cache. I found a forum where some did this with success, but in our case, It didn't help.

What I maybe will do:
- Put the DCs (2012R2) in a seperate Backupjob and disable the fallback option in HyperV Quiescence. <-- anyone tried this? Will the disabled fallback help?

Someone... has any other ideas? Application Aware Backup is no option an as I read it, it will fail anyway.

Thx for your help in advance!

Patrick

[nmdange]

It sounds like your issue is different. Microsoft did provide me with a temporary workaround to get production checkpoints working on 2012 R2 DCs. The issue will come back but it does fix it temporarily. Not sure if it will help in your case or not. I have observed the hung vmms.exe a few times but I hadn't connected it with the 2012 R2 DC issue.

Issue: Production Checkpoints failing for 2012 R2 DC on 2016 host

This is a known issue and a bug is filed on the same. Microsoft is currently working on this and would be releasing fix in upcoming months.

Bug number: 13748809

As a workaround, kindly restart the below services in a sequence in VM and then try the checkpoint:

COM+ System Application Service
Com+ Event System
Cryptographic Services
Volume Shadow Copy Service
Microsoft Software Shadow copy provider

[mkaec]

- Put the DCs (2012R2) in a seperate Backupjob and disable the fallback option in HyperV Quiescence. <-- anyone tried this? Will the disabled fallback help?

I had tried this with my backup job and this caused the backup to fail outright. My conclusion was that quiescence was failing and falling back every time, Veeam just wasn't telling me about this in its job log. I ended up turning off quiescence altogether, but hope that Microsoft will come through with a patch in January and I'll be able to turn it back on.

[gotsch]

I had tried this with my backup job and this caused the backup to fail outright. My conclusion was that quiescence was failing and falling back every time, Veeam just wasn't telling me about this in its job log. I ended up turning off quiescence altogether, but hope that Microsoft will come through with a patch in January and I'll be able to turn it back on.

Ty for this info, I will disable it for my seperate DC Job also.

It sounds like your issue is different. Microsoft did provide me with a temporary workaround to get production checkpoints working on 2012 R2 DCs. The issue will come back but it does fix it temporarily. Not sure if it will help in your case or not. I have observed the hung vmms.exe a few times but I hadn't connected it with the 2012 R2 DC issue.

Issue: Production Checkpoints failing for 2012 R2 DC on 2016 host

This is a known issue and a bug is filed on the same. Microsoft is currently working on this and would be releasing fix in upcoming months.

Bug number: 13748809

As a workaround, kindly restart the below services in a sequence in VM and then try the checkpoint:

COM+ System Application Service
Com+ Event System
Cryptographic Services
Volume Shadow Copy Service
Microsoft Software Shadow copy provider

We got the same "workaround" from Microsoft:

Mögliche Lösungsansätze /zur Zeit ist noch kein Fix von MS erlassen/

1.Host auf die letzte Version aktualisieren /bereits gemacht?/:
https://support.microsoft.com/en-us/help/4038782
2.Auf Gast folgenden KB installieren oder aber Gäste durch Windows Updates mit neuesten Updates versehen: https://support.microsoft.com/en-us/hel ... view-hosts
3.Hyper-V Rolle auf Host deinstallieren und erneut installieren /optional, ich würde dies nicht empfehlen/

Der Ansatz, der erfahrungsgemäß am Besten funktioniert hat– wie im Veeam Forum von einem Kunden beschrieben – ist es, die folgenden Kommandos in Batch zusammen zu fassen, bzw. vor jedem Backup-Job /scheduled task?/ auf allen DC-Gästen ausführen:
net stop BITS
net stop SENS
net stop NtFrs
net stop DFSR
net stop EventSystem
net start EventSystem
net start DFSR
net start NtFrs
net start SENS
net start BITS
net stop VSS
net start VSS

[ricardo.simoes]

Hi everyone. I have the same problem.

Host -> w2k16 (VBR 9.5)
VM -> w2k16 (VM with System Center 2016)

When I run the backup it gives the following error:
Failed to create VM recovery checkpoint (mode: Veeam application-aware processing) Details: Job failed (''). Error code: '32772'. Failed to create VM recovery snapshot, VM ID 'fd18471c-6ff3-4178-952f-da111b51b246'.

Both OS are updated and I have multiple VM on the same host and I have no problem backing up the remaining.
I have already moved the VM to other Hosts and the problem remains.
At the moment I have 17 VM and I only have problems with 1.

Help...

[Afterburned]

I have the same issue (2012r2 vm on 2016 host)

My workaround for this is:
Check this is the issue by taking a snapshot in Hyper-V manager - should fail.
Edit the VM settings in cluster manager - take off the backup integration service.
Take a checkpoint from Hyper-V manager (should work) - then delete the checkpoint.
Edit the VM settings in cluster manager - turn the backup integration service back on.
Take a checkpoint - should work. Delete it again just to be neat.

Next backup works. Please note I am not using Veeam but have the same issue with another backup software and this thread has helped focus my investigations (MS bug) so am sharing.

[Mike Resseler]

Aaron,

First: Welcome to the forums!
Second: Thanks for sharing your tip! Really appreciate it and I am rooting for you to get further with MSFT. This bug exists for too long imho

[Mike Resseler]

Ricardo,

Am I right in seeing that you have the issue with a 2k16 VM and not with a 2k12 VM? In that case something else is wrong and I would advice you to create a support call. Post the support case ID here and the follow-up after investigation of our engineers
Thanks
Mike

[ricardo.simoes]

Hi @Mike Resseler. Yes, you're right. My Host and VM have installed Win2k16.

However. I've been testing the following:

I disabled the "Enable application-aware processing" option and the backup job ran successfully.
When I re-enable the "Enable application-aware processing" option, it continues to not back up. Does it fail because of the SQL instance created by System Center ?!

It also had the same problem with creating VM checkpoints in Hyper-V (problem identified by @Afterburned). After the workaround has been performing Checkpoints but the backup job continues to fail.

After these, Veeam returns the following error:

"Unable to subscribe to guest processing components: Can not do HvPrepareSnapshot Error: Failed to call RPC function 'Vss.HvPrepareSnapshot': Error code: 0x80004005 Failed to invoke func [HvPrepareSnapshot]: Unspecified error Failed to inject VSS plug-in. PID is NULL.
Failed to create VM recovery checkpoint (mode: Veeam application-aware processing) Details: Failed to prepare guests for volume snapshot.
Error: Failed to prepare guests for volume snapshot. "

I apologize for my English...
Best regards