Host-based backup of Microsoft Hyper-V VMs.
Post Reply
TT-DL
Influencer
Posts: 21
Liked: 2 times
Joined: Feb 14, 2017 2:50 am
Full Name: Daniel L
Contact:

Re: 9.5 Hyper-V 2016 Known Issues

Post by TT-DL »

Also FYI, Windows 2k16 updates can be tracked here.

https://support.microsoft.com/en-au/help/4000825

I've just had a quick read of the August updates and I don't see anything relevant that would solve the backup/prod checkpoint issue.
rkovhaev
Veeam Software
Posts: 39
Liked: 21 times
Joined: May 17, 2010 6:49 pm
Full Name: Rustam
Location: hockey night in canada
Contact:

Re: 9.5 Hyper-V 2016 Known Issues

Post by rkovhaev »

TT-DL wrote: From my conversations with 2nd/3rd level support with both Veeam and Microsoft I don't think any 2k16 updates will resolve this issue.
MS has a least 2 ways to address the problem:
1) fix old(pre-2016) NTDS/ADAM VSS writers - so that these writers become aware that they should not perform autorecovery steps
or
2) fix 2016 HV server and guest integration components - so that VSS requester in guest creates shadow copy with autorecovery attribute - and this way MS doesn't need to fix VSS writers because writers will perform autorecovery steps just fine
frankive
Service Provider
Posts: 1092
Liked: 134 times
Joined: May 14, 2013 8:35 pm
Full Name: Frank Iversen
Location: Norway
Contact:

Re: 9.5 Hyper-V 2016 Known Issues

Post by frankive »

and why doesn't Microsoft do this? what do they answer when Veeam states this?
hoFFy
Service Provider
Posts: 183
Liked: 40 times
Joined: Apr 27, 2012 1:10 pm
Full Name: Sebastian Hoffmann
Location: Germany / Lohne
Contact:

Re: 9.5 Hyper-V 2016 Known Issues

Post by hoFFy »

We are experiencing this error on a 2016 Guest OS running a WSUS, not a DC.
Has anybody else those problems with a server not running a DC?
VMCE 7 / 8 / 9, VCP-DC 5 / 5.5 / 6, MCITP:SA
Blog: machinewithoutbrain.de
mkaec
Veteran
Posts: 462
Liked: 133 times
Joined: Jul 16, 2015 1:31 pm
Full Name: Marc K
Contact:

Re: 9.5 Hyper-V 2016 Known Issues

Post by mkaec »

I experienced similar problems on a 2016 guest running DFS. Turning off AAP seems to have fixed it for me.
ashfwa
Novice
Posts: 4
Liked: never
Joined: Mar 09, 2012 2:50 pm
Full Name: Aaron Schneider
Contact:

Re: 9.5 Hyper-V 2016 Known Issues

Post by ashfwa »

Veeam support (Case 02296778) has acknowledged that they are unable to confirm that the following issue has been resolved:

Resilient Change Tracking (RCT) returns incorrect changes for VMs which had their virtual disks moved with Storage Live Migration, resulting in corrupt backups.
This is a Hyper-V 2016 issue that Microsoft is planning to address in the future update. Until then, disable the use of changed block tracking for any jobs processing such VMs.

I apologize if I missed it in these 15 pages of posts, but does anyone know if this issue has been addressed or how to track it from the Microsoft side? Presumably Veeam needs to update its Release Notes at each iteration and has this issue earmarked so they know when it can be removed as a "Known Issue", but I am only being forwarded to Microsoft. I cannot find anything that even specifically references this bug...let alone it's fix.

Can anyone offer any further insight?
markus.s99
Lurker
Posts: 1
Liked: never
Joined: Sep 11, 2017 12:31 pm
Contact:

Re: 9.5 Hyper-V 2016 Known Issues

Post by markus.s99 »

I have the same problem with Hyper-V 2016 host and windows server 2016 VM with SQL Server 2016. (could not initiate a checkpoint operation: %%2147754767 (0x8004230F). Error code: '32768'.)
But 20% of the incremental backups are working...80% fails.

Is that also a "known" bug from microsoft???

After turning off AAP backup is working fine - but not a solution.

Thanks for the update
Mike Resseler
Product Manager
Posts: 8044
Liked: 1263 times
Joined: Feb 08, 2013 3:08 pm
Full Name: Mike Resseler
Location: Belgium
Contact:

Re: 9.5 Hyper-V 2016 Known Issues

Post by Mike Resseler »

Markus,

First: Welcome to the forums
Second: I doubt this is a known bug. Please create a support call and post the case ID here and the follow-up after investigation

Thanks
Mike
TT-DL
Influencer
Posts: 21
Liked: 2 times
Joined: Feb 14, 2017 2:50 am
Full Name: Daniel L
Contact:

Re: 9.5 Hyper-V 2016 Known Issues

Post by TT-DL »

Some answers.

Production Checkpoints on 2k8/2k8R2 Domain Controllers - Not supported.
See this kb https://technet.microsoft.com/en-us/lib ... s.11).aspx

Exchange 2010
Production Checkpoint of a machine running Edge Transport role (ADAM VSS writer). Not supported.

This is directly from MSoft Premier support.
“Microsoft does not support the use of checkpoints on virtual machines running the Active Directory Domain Services role (also known as “domain controllers”) or the Active Directory Lightweight Directory Services role in Windows Server 2008 or Windows Server 2008 R2. For more information, see Operational Considerations for Virtualized Domain Controllers (http://go.microsoft.com/fwlink/?LinkId=159631).”
*ADAM is former name of AD LDS.


Exchange 2010 backups on hyper-v 2016 with AAP enabled. Works. (As long at edge transport role is not installed) This above would also explain why virtualised SBS etc doesn't work on a 2016 host.

Not sure about production checkpoints on AD 2012 or Exchange 2013 (I imagine exchange 2013 is same as above). I understand Veeam is still investigating/clarifying with Microsoft exactly what is/isnt supported in relation to AD checkpoints.

Many thanks to Rustam and Veeam support team for their assistance in getting some answers on this.
RGijsen
Expert
Posts: 124
Liked: 25 times
Joined: Oct 10, 2014 2:06 pm
Contact:

Re: 9.5 Hyper-V 2016 Known Issues

Post by RGijsen »

In the past I've worked around the Edge-issues by having Veeam trigger a script that stops all Exchanges services on that VM for a while (although if I remember correctly, stopping ADAM is enough) and having it restarted after the backup. Depending on your setup (redundant, balancers and such) there should be no disruption in doing so.
marieelieg
Expert
Posts: 147
Liked: 8 times
Joined: Oct 20, 2016 7:15 am
Contact:

Re: 9.5 Hyper-V 2016 Known Issues

Post by marieelieg »

Hello dears,

I have the same issue as AlexLeadingEdge. Backup job with App Aware enabled not able to create snapshot on DC VM.
"Job failed (''). Error code: '32768'. Failed to create VM recovery snapshot"

Was there a solution for this?

Thanks!
Nail.Mukhametshin
Enthusiast
Posts: 51
Liked: 3 times
Joined: May 05, 2016 1:15 pm
Full Name: Наиль Мухаметшин
Location: Russia, Moscow
Contact:

Re: 9.5 Hyper-V 2016 Known Issues

Post by Nail.Mukhametshin »

Hi guys!

Question to moderators: If you really have some known issues, can you create a new topic (may be readonly) where in first post you will track all issues and fixes? It will be very helpful not to read 15 pages of this topic to understand about witch issues you speaking.
Nail Mukhametshin | System Engineer | R&D Services Infrastructure Development | Kaspersky Lab
pesos
Expert
Posts: 205
Liked: 17 times
Joined: Nov 12, 2014 9:40 am
Full Name: John Johnson
Contact:

Re: 9.5 Hyper-V 2016 Known Issues

Post by pesos »

+1

We have jobs failing at least 2-3 times a month and have to restart all veeam services on our 2016 host to resolve (freshly built server). Curious to know if there are known issues potentially causing this but this thread is out of control :p
sbaltic
Enthusiast
Posts: 31
Liked: 1 time
Joined: Nov 18, 2013 6:10 pm
Full Name: Sebastijan
Contact:

Re: 9.5 Hyper-V 2016 Known Issues

Post by sbaltic »

Same problem here ... host -> w2k16, VM -> 2k16 (not DC, regular file server).

11.10.2017 20:01:26 :: Failed to create VM recovery checkpoint (mode: Veeam application-aware processing) Details: Job failed (''). Error code: '32772'.
Failed to create VM recovery snapshot, VM ID '6604883e-fc6e-4403-b72d-454d3741377b'.
mkaec
Veteran
Posts: 462
Liked: 133 times
Joined: Jul 16, 2015 1:31 pm
Full Name: Marc K
Contact:

Re: 9.5 Hyper-V 2016 Known Issues

Post by mkaec »

For my 2012 R2 DCs, I turned off Application Aware Processing and I have the VM settings configured to use Standard checkpoints. If I don't also turn off Guest Quiescence, Veeam reports that the backup was successful, but the server logs an error "could not initiate a checkpoint operation".

I know its been mentioned that AAP is not needed for Veeam AD Explorer. But, that doesn't seem to be entirely true. If I create a new job and put my DCs in it, that job will not show up in the Microsoft Active Directory Object Restore wizard. So, it seems that AAP must be active at least once for Veeam to know the job has DCs in it. If you deploy Veeam fresh into an environment comprised of 2016 hosts with 2012 R2 DCs, it doesn't look like you can get Veeam AD explorer working.

It's been a year since this problem was first documented. I think the note that "this is a Hyper-V 2016 issue that Microsoft is planning to address in the future update" was incorrect. It's really disappointing that this was never resolved.
nmdange
Veteran
Posts: 527
Liked: 142 times
Joined: Aug 20, 2015 9:30 pm
Contact:

Re: 9.5 Hyper-V 2016 Known Issues

Post by nmdange »

It is possible to open AD Explorer without AAP. In the Veeam console, go to Backups->Disks in the left navigation pane. Then search for the DC VM you want to restore from. Select the VM and then on the top menu in the Backup tab, select "Applications Items" and then "Microsoft Active Directory".

However, you still need to have Hyper-V Native Quiescence enabled with production checkpoints to get a consistent backup. I have not had any issues backing up 2012 R2 DCs on a 2016 host with this setup.
mkaec
Veteran
Posts: 462
Liked: 133 times
Joined: Jul 16, 2015 1:31 pm
Full Name: Marc K
Contact:

Re: 9.5 Hyper-V 2016 Known Issues

Post by mkaec »

nmdange wrote:...go to Backups->Disks in the left navigation pane. Then search for the DC VM you want to restore from. Select the VM and then on the top menu in the Backup tab, select "Applications Items" and then "Microsoft Active Directory".
Thanks. I had tried this earlier, but I was in the wrong job at the time. It worked when I tried it again from the correct job.
However, you still need to have Hyper-V Native Quiescence enabled with production checkpoints to get a consistent backup. I have not had any issues backing up 2012 R2 DCs on a 2016 host with this setup.
First, if you have production checkpoints enabled, try manually making a checkpoint in Hyper-V Manager. It should fail.

Then check the Microsoft-Windows-Hyper-V-VMMS-Admin event log around the time of your backups. You should see some events in there being logged about failing checkpoints.

And in your backup job, you might want to try leaving guest quiescence enabled, but turning off the "crash consistent backup" fallback option. This should cause the backup to fail.

From what I can tell, the quiescence checkpoint is failing and the backup is falling back to a crash consistent backup. Veeam just isn't letting us know in the job log that there were problems.
nmdange
Veteran
Posts: 527
Liked: 142 times
Joined: Aug 20, 2015 9:30 pm
Contact:

Re: 9.5 Hyper-V 2016 Known Issues

Post by nmdange »

Well you're definitely right about production checkpoints failing. Looks like I might have to open a ticket with Microsoft on that one :)
nmdange
Veteran
Posts: 527
Liked: 142 times
Joined: Aug 20, 2015 9:30 pm
Contact:

Re: 9.5 Hyper-V 2016 Known Issues

Post by nmdange »

Well according to Microsoft this specific issue (2012 R2 DC on 2016 host) is a known bug. Waiting on some more details but it sounds like the fix is scheduled to be included in the January Monthly update.
mkaec
Veteran
Posts: 462
Liked: 133 times
Joined: Jul 16, 2015 1:31 pm
Full Name: Marc K
Contact:

Re: 9.5 Hyper-V 2016 Known Issues

Post by mkaec »

That would be excellent. Did the source confirm the year of the expected fix? Given that the issue was known in Nov 2016, I would have thought a fix would have been delivered by Jan 2017.
nmdange
Veteran
Posts: 527
Liked: 142 times
Joined: Aug 20, 2015 9:30 pm
Contact:

Re: 9.5 Hyper-V 2016 Known Issues

Post by nmdange »

I meant next January :)
gotsch
Influencer
Posts: 12
Liked: never
Joined: Nov 09, 2017 5:42 am
Full Name: Patrick Gotsch
Contact:

Re: 9.5 Hyper-V 2016 Known Issues

Post by gotsch »

Hi everyone, I would like to confirm this great MS Bug, show our situation and maybe someone has a workaround until MS fix this.

Situation:
- 2016 HyperV Cluster, 2 nodes, 105 VMs, most of them 2012R2
- VMs are different customers, so they are all seperated with VLANs. The Cluster and Backup Server itself have no network connection within the VMs.
- Backup Server is a dedicated Server 2016 Machine with Veeam 9.5, fully patched
- Storage is a HBA SAN attached to the cluster nodes

What happens:
- Since our Backup and Cluster has no Network Connection in the VMs, we use HyperV Quiescence for years now, without any problem until 2016 HypeV Hosts. HyperV Quiescence is set in the Backupjob with the fallback option enabled.
- Some VMs Backup may failling, VMs couldn't be accessed anymore, VMs are stucked in Cluster/HyperV with the status "Creating Snapshot..." or "Merging Snapshot...."
- Only Solution ist to move all uneffected VMs to the other Host and then hard reset the failed Host with the failed VM. It is not possible to kill the vmms.exe from the failing VM
- Last two situations where this happens: today in the early hours and 2 days a go. Affected VMs: 2012R2
- It doesn't makes a different if Production Snapshot or Standard Snapshot is set for the VM

MS Support:
MS has confirmed this Bug in our case, the technican has no ETA of a bugfix, so we are all a little bit f***ed up by Microsoft.

Veeam Suport:
Escelated, but they are far behind MS (which means a lot :-D ), the last thing the told us was that it is not the known bug... sure... so no help here.

What I tried:
- I disabled the CSV Cache. I found a forum where some did this with success, but in our case, It didn't help.

What I maybe will do:
- Put the DCs (2012R2) in a seperate Backupjob and disable the fallback option in HyperV Quiescence. <-- anyone tried this? Will the disabled fallback help?

Someone... has any other ideas? Application Aware Backup is no option an as I read it, it will fail anyway.

Thx for your help in advance!

Patrick
nmdange
Veteran
Posts: 527
Liked: 142 times
Joined: Aug 20, 2015 9:30 pm
Contact:

Re: 9.5 Hyper-V 2016 Known Issues

Post by nmdange »

It sounds like your issue is different. Microsoft did provide me with a temporary workaround to get production checkpoints working on 2012 R2 DCs. The issue will come back but it does fix it temporarily. Not sure if it will help in your case or not. I have observed the hung vmms.exe a few times but I hadn't connected it with the 2012 R2 DC issue.
Issue: Production Checkpoints failing for 2012 R2 DC on 2016 host

This is a known issue and a bug is filed on the same. Microsoft is currently working on this and would be releasing fix in upcoming months.

Bug number: 13748809

As a workaround, kindly restart the below services in a sequence in VM and then try the checkpoint:

COM+ System Application Service
Com+ Event System
Cryptographic Services
Volume Shadow Copy Service
Microsoft Software Shadow copy provider
mkaec
Veteran
Posts: 462
Liked: 133 times
Joined: Jul 16, 2015 1:31 pm
Full Name: Marc K
Contact:

Re: 9.5 Hyper-V 2016 Known Issues

Post by mkaec »

gotsch wrote:- Put the DCs (2012R2) in a seperate Backupjob and disable the fallback option in HyperV Quiescence. <-- anyone tried this? Will the disabled fallback help?
I had tried this with my backup job and this caused the backup to fail outright. My conclusion was that quiescence was failing and falling back every time, Veeam just wasn't telling me about this in its job log. I ended up turning off quiescence altogether, but hope that Microsoft will come through with a patch in January and I'll be able to turn it back on.
gotsch
Influencer
Posts: 12
Liked: never
Joined: Nov 09, 2017 5:42 am
Full Name: Patrick Gotsch
Contact:

Re: 9.5 Hyper-V 2016 Known Issues

Post by gotsch »

mkaec wrote:I had tried this with my backup job and this caused the backup to fail outright. My conclusion was that quiescence was failing and falling back every time, Veeam just wasn't telling me about this in its job log. I ended up turning off quiescence altogether, but hope that Microsoft will come through with a patch in January and I'll be able to turn it back on.

Ty for this info, I will disable it for my seperate DC Job also.

nmdange wrote:It sounds like your issue is different. Microsoft did provide me with a temporary workaround to get production checkpoints working on 2012 R2 DCs. The issue will come back but it does fix it temporarily. Not sure if it will help in your case or not. I have observed the hung vmms.exe a few times but I hadn't connected it with the 2012 R2 DC issue.

Issue: Production Checkpoints failing for 2012 R2 DC on 2016 host

This is a known issue and a bug is filed on the same. Microsoft is currently working on this and would be releasing fix in upcoming months.

Bug number: 13748809

As a workaround, kindly restart the below services in a sequence in VM and then try the checkpoint:

COM+ System Application Service
Com+ Event System
Cryptographic Services
Volume Shadow Copy Service
Microsoft Software Shadow copy provider
We got the same "workaround" from Microsoft:
Microsoft wrote: Mögliche Lösungsansätze /zur Zeit ist noch kein Fix von MS erlassen/

1.Host auf die letzte Version aktualisieren /bereits gemacht?/:
https://support.microsoft.com/en-us/help/4038782
2.Auf Gast folgenden KB installieren oder aber Gäste durch Windows Updates mit neuesten Updates versehen: https://support.microsoft.com/en-us/hel ... view-hosts
3.Hyper-V Rolle auf Host deinstallieren und erneut installieren /optional, ich würde dies nicht empfehlen/

Der Ansatz, der erfahrungsgemäß am Besten funktioniert hat– wie im Veeam Forum von einem Kunden beschrieben – ist es, die folgenden Kommandos in Batch zusammen zu fassen, bzw. vor jedem Backup-Job /scheduled task?/ auf allen DC-Gästen ausführen:
net stop BITS
net stop SENS
net stop NtFrs
net stop DFSR
net stop EventSystem
net start EventSystem
net start DFSR
net start NtFrs
net start SENS
net start BITS
net stop VSS
net start VSS
ricardo.simoes
Influencer
Posts: 13
Liked: 3 times
Joined: Jun 29, 2017 4:05 pm
Contact:

Re: 9.5 Hyper-V 2016 Known Issues

Post by ricardo.simoes »

Hi everyone. I have the same problem.

Host -> w2k16 (VBR 9.5)
VM -> w2k16 (VM with System Center 2016)

When I run the backup it gives the following error:
Failed to create VM recovery checkpoint (mode: Veeam application-aware processing) Details: Job failed (''). Error code: '32772'. Failed to create VM recovery snapshot, VM ID 'fd18471c-6ff3-4178-952f-da111b51b246'.

Both OS are updated and I have multiple VM on the same host and I have no problem backing up the remaining.
I have already moved the VM to other Hosts and the problem remains.
At the moment I have 17 VM and I only have problems with 1. :x

Help...
Afterburned
Lurker
Posts: 1
Liked: never
Joined: Nov 29, 2017 10:07 am
Full Name: Aaron Davies
Contact:

Re: 9.5 Hyper-V 2016 Known Issues

Post by Afterburned »

I have the same issue (2012r2 vm on 2016 host)

My workaround for this is:
Check this is the issue by taking a snapshot in Hyper-V manager - should fail.
Edit the VM settings in cluster manager - take off the backup integration service.
Take a checkpoint from Hyper-V manager (should work) - then delete the checkpoint.
Edit the VM settings in cluster manager - turn the backup integration service back on.
Take a checkpoint - should work. Delete it again just to be neat.

Next backup works. Please note I am not using Veeam but have the same issue with another backup software and this thread has helped focus my investigations (MS bug) so am sharing.
Mike Resseler
Product Manager
Posts: 8044
Liked: 1263 times
Joined: Feb 08, 2013 3:08 pm
Full Name: Mike Resseler
Location: Belgium
Contact:

Re: 9.5 Hyper-V 2016 Known Issues

Post by Mike Resseler »

Aaron,

First: Welcome to the forums!
Second: Thanks for sharing your tip! Really appreciate it and I am rooting for you to get further with MSFT. This bug exists for too long imho ;-)
Mike Resseler
Product Manager
Posts: 8044
Liked: 1263 times
Joined: Feb 08, 2013 3:08 pm
Full Name: Mike Resseler
Location: Belgium
Contact:

Re: 9.5 Hyper-V 2016 Known Issues

Post by Mike Resseler »

Ricardo,

Am I right in seeing that you have the issue with a 2k16 VM and not with a 2k12 VM? In that case something else is wrong and I would advice you to create a support call. Post the support case ID here and the follow-up after investigation of our engineers
Thanks
Mike
ricardo.simoes
Influencer
Posts: 13
Liked: 3 times
Joined: Jun 29, 2017 4:05 pm
Contact:

Re: 9.5 Hyper-V 2016 Known Issues

Post by ricardo.simoes »

Hi @Mike Resseler. Yes, you're right. My Host and VM have installed Win2k16.

However. I've been testing the following:

I disabled the "Enable application-aware processing" option and the backup job ran successfully.
When I re-enable the "Enable application-aware processing" option, it continues to not back up. Does it fail because of the SQL instance created by System Center ?!

It also had the same problem with creating VM checkpoints in Hyper-V (problem identified by @Afterburned). After the workaround has been performing Checkpoints but the backup job continues to fail.

After these, Veeam returns the following error:

"Unable to subscribe to guest processing components: Can not do HvPrepareSnapshot Error: Failed to call RPC function 'Vss.HvPrepareSnapshot': Error code: 0x80004005 Failed to invoke func [HvPrepareSnapshot]: Unspecified error Failed to inject VSS plug-in. PID is NULL.
Failed to create VM recovery checkpoint (mode: Veeam application-aware processing) Details: Failed to prepare guests for volume snapshot.
Error: Failed to prepare guests for volume snapshot. "

I apologize for my English...
Best regards
Post Reply

Who is online

Users browsing this forum: No registered users and 20 guests