Discussions specific to the Microsoft Hyper-V hypervisor
Post Reply
nmdange
Expert
Posts: 341
Liked: 82 times
Joined: Aug 20, 2015 9:30 pm
Contact:

Re: 9.5 Hyper-V 2016 Known Issues

Post by nmdange » Oct 30, 2017 5:41 pm

It is possible to open AD Explorer without AAP. In the Veeam console, go to Backups->Disks in the left navigation pane. Then search for the DC VM you want to restore from. Select the VM and then on the top menu in the Backup tab, select "Applications Items" and then "Microsoft Active Directory".

However, you still need to have Hyper-V Native Quiescence enabled with production checkpoints to get a consistent backup. I have not had any issues backing up 2012 R2 DCs on a 2016 host with this setup.

mkaec
Expert
Posts: 259
Liked: 55 times
Joined: Jul 16, 2015 1:31 pm
Full Name: Marc K
Contact:

Re: 9.5 Hyper-V 2016 Known Issues

Post by mkaec » Oct 30, 2017 8:35 pm

nmdange wrote:...go to Backups->Disks in the left navigation pane. Then search for the DC VM you want to restore from. Select the VM and then on the top menu in the Backup tab, select "Applications Items" and then "Microsoft Active Directory".
Thanks. I had tried this earlier, but I was in the wrong job at the time. It worked when I tried it again from the correct job.
However, you still need to have Hyper-V Native Quiescence enabled with production checkpoints to get a consistent backup. I have not had any issues backing up 2012 R2 DCs on a 2016 host with this setup.
First, if you have production checkpoints enabled, try manually making a checkpoint in Hyper-V Manager. It should fail.

Then check the Microsoft-Windows-Hyper-V-VMMS-Admin event log around the time of your backups. You should see some events in there being logged about failing checkpoints.

And in your backup job, you might want to try leaving guest quiescence enabled, but turning off the "crash consistent backup" fallback option. This should cause the backup to fail.

From what I can tell, the quiescence checkpoint is failing and the backup is falling back to a crash consistent backup. Veeam just isn't letting us know in the job log that there were problems.

nmdange
Expert
Posts: 341
Liked: 82 times
Joined: Aug 20, 2015 9:30 pm
Contact:

Re: 9.5 Hyper-V 2016 Known Issues

Post by nmdange » Oct 30, 2017 9:12 pm

Well you're definitely right about production checkpoints failing. Looks like I might have to open a ticket with Microsoft on that one :)

nmdange
Expert
Posts: 341
Liked: 82 times
Joined: Aug 20, 2015 9:30 pm
Contact:

Re: 9.5 Hyper-V 2016 Known Issues

Post by nmdange » Nov 06, 2017 3:01 pm

Well according to Microsoft this specific issue (2012 R2 DC on 2016 host) is a known bug. Waiting on some more details but it sounds like the fix is scheduled to be included in the January Monthly update.

mkaec
Expert
Posts: 259
Liked: 55 times
Joined: Jul 16, 2015 1:31 pm
Full Name: Marc K
Contact:

Re: 9.5 Hyper-V 2016 Known Issues

Post by mkaec » Nov 06, 2017 3:10 pm

That would be excellent. Did the source confirm the year of the expected fix? Given that the issue was known in Nov 2016, I would have thought a fix would have been delivered by Jan 2017.

nmdange
Expert
Posts: 341
Liked: 82 times
Joined: Aug 20, 2015 9:30 pm
Contact:

Re: 9.5 Hyper-V 2016 Known Issues

Post by nmdange » Nov 06, 2017 3:52 pm

I meant next January :)

gotsch
Novice
Posts: 7
Liked: never
Joined: Nov 09, 2017 5:42 am
Full Name: Patrick Gotsch
Contact:

Re: 9.5 Hyper-V 2016 Known Issues

Post by gotsch » Nov 09, 2017 6:05 am

Hi everyone, I would like to confirm this great MS Bug, show our situation and maybe someone has a workaround until MS fix this.

Situation:
- 2016 HyperV Cluster, 2 nodes, 105 VMs, most of them 2012R2
- VMs are different customers, so they are all seperated with VLANs. The Cluster and Backup Server itself have no network connection within the VMs.
- Backup Server is a dedicated Server 2016 Machine with Veeam 9.5, fully patched
- Storage is a HBA SAN attached to the cluster nodes

What happens:
- Since our Backup and Cluster has no Network Connection in the VMs, we use HyperV Quiescence for years now, without any problem until 2016 HypeV Hosts. HyperV Quiescence is set in the Backupjob with the fallback option enabled.
- Some VMs Backup may failling, VMs couldn't be accessed anymore, VMs are stucked in Cluster/HyperV with the status "Creating Snapshot..." or "Merging Snapshot...."
- Only Solution ist to move all uneffected VMs to the other Host and then hard reset the failed Host with the failed VM. It is not possible to kill the vmms.exe from the failing VM
- Last two situations where this happens: today in the early hours and 2 days a go. Affected VMs: 2012R2
- It doesn't makes a different if Production Snapshot or Standard Snapshot is set for the VM

MS Support:
MS has confirmed this Bug in our case, the technican has no ETA of a bugfix, so we are all a little bit f***ed up by Microsoft.

Veeam Suport:
Escelated, but they are far behind MS (which means a lot :-D ), the last thing the told us was that it is not the known bug... sure... so no help here.

What I tried:
- I disabled the CSV Cache. I found a forum where some did this with success, but in our case, It didn't help.

What I maybe will do:
- Put the DCs (2012R2) in a seperate Backupjob and disable the fallback option in HyperV Quiescence. <-- anyone tried this? Will the disabled fallback help?

Someone... has any other ideas? Application Aware Backup is no option an as I read it, it will fail anyway.

Thx for your help in advance!

Patrick

nmdange
Expert
Posts: 341
Liked: 82 times
Joined: Aug 20, 2015 9:30 pm
Contact:

Re: 9.5 Hyper-V 2016 Known Issues

Post by nmdange » Nov 09, 2017 3:21 pm

It sounds like your issue is different. Microsoft did provide me with a temporary workaround to get production checkpoints working on 2012 R2 DCs. The issue will come back but it does fix it temporarily. Not sure if it will help in your case or not. I have observed the hung vmms.exe a few times but I hadn't connected it with the 2012 R2 DC issue.
Issue: Production Checkpoints failing for 2012 R2 DC on 2016 host

This is a known issue and a bug is filed on the same. Microsoft is currently working on this and would be releasing fix in upcoming months.

Bug number: 13748809

As a workaround, kindly restart the below services in a sequence in VM and then try the checkpoint:

COM+ System Application Service
Com+ Event System
Cryptographic Services
Volume Shadow Copy Service
Microsoft Software Shadow copy provider

mkaec
Expert
Posts: 259
Liked: 55 times
Joined: Jul 16, 2015 1:31 pm
Full Name: Marc K
Contact:

Re: 9.5 Hyper-V 2016 Known Issues

Post by mkaec » Nov 09, 2017 5:36 pm

gotsch wrote:- Put the DCs (2012R2) in a seperate Backupjob and disable the fallback option in HyperV Quiescence. <-- anyone tried this? Will the disabled fallback help?
I had tried this with my backup job and this caused the backup to fail outright. My conclusion was that quiescence was failing and falling back every time, Veeam just wasn't telling me about this in its job log. I ended up turning off quiescence altogether, but hope that Microsoft will come through with a patch in January and I'll be able to turn it back on.

gotsch
Novice
Posts: 7
Liked: never
Joined: Nov 09, 2017 5:42 am
Full Name: Patrick Gotsch
Contact:

Re: 9.5 Hyper-V 2016 Known Issues

Post by gotsch » Nov 09, 2017 6:53 pm

mkaec wrote:I had tried this with my backup job and this caused the backup to fail outright. My conclusion was that quiescence was failing and falling back every time, Veeam just wasn't telling me about this in its job log. I ended up turning off quiescence altogether, but hope that Microsoft will come through with a patch in January and I'll be able to turn it back on.

Ty for this info, I will disable it for my seperate DC Job also.

nmdange wrote:It sounds like your issue is different. Microsoft did provide me with a temporary workaround to get production checkpoints working on 2012 R2 DCs. The issue will come back but it does fix it temporarily. Not sure if it will help in your case or not. I have observed the hung vmms.exe a few times but I hadn't connected it with the 2012 R2 DC issue.

Issue: Production Checkpoints failing for 2012 R2 DC on 2016 host

This is a known issue and a bug is filed on the same. Microsoft is currently working on this and would be releasing fix in upcoming months.

Bug number: 13748809

As a workaround, kindly restart the below services in a sequence in VM and then try the checkpoint:

COM+ System Application Service
Com+ Event System
Cryptographic Services
Volume Shadow Copy Service
Microsoft Software Shadow copy provider
We got the same "workaround" from Microsoft:
Microsoft wrote: Mögliche Lösungsansätze /zur Zeit ist noch kein Fix von MS erlassen/

1.Host auf die letzte Version aktualisieren /bereits gemacht?/:
https://support.microsoft.com/en-us/help/4038782
2.Auf Gast folgenden KB installieren oder aber Gäste durch Windows Updates mit neuesten Updates versehen: https://support.microsoft.com/en-us/hel ... view-hosts
3.Hyper-V Rolle auf Host deinstallieren und erneut installieren /optional, ich würde dies nicht empfehlen/

Der Ansatz, der erfahrungsgemäß am Besten funktioniert hat– wie im Veeam Forum von einem Kunden beschrieben – ist es, die folgenden Kommandos in Batch zusammen zu fassen, bzw. vor jedem Backup-Job /scheduled task?/ auf allen DC-Gästen ausführen:
net stop BITS
net stop SENS
net stop NtFrs
net stop DFSR
net stop EventSystem
net start EventSystem
net start DFSR
net start NtFrs
net start SENS
net start BITS
net stop VSS
net start VSS

ricardo.simoes
Novice
Posts: 4
Liked: never
Joined: Jun 29, 2017 4:05 pm
Contact:

Re: 9.5 Hyper-V 2016 Known Issues

Post by ricardo.simoes » Nov 21, 2017 3:24 pm

Hi everyone. I have the same problem.

Host -> w2k16 (VBR 9.5)
VM -> w2k16 (VM with System Center 2016)

When I run the backup it gives the following error:
Failed to create VM recovery checkpoint (mode: Veeam application-aware processing) Details: Job failed (''). Error code: '32772'. Failed to create VM recovery snapshot, VM ID 'fd18471c-6ff3-4178-952f-da111b51b246'.

Both OS are updated and I have multiple VM on the same host and I have no problem backing up the remaining.
I have already moved the VM to other Hosts and the problem remains.
At the moment I have 17 VM and I only have problems with 1. :x

Help...

Afterburned
Lurker
Posts: 1
Liked: never
Joined: Nov 29, 2017 10:07 am
Full Name: Aaron Davies
Contact:

Re: 9.5 Hyper-V 2016 Known Issues

Post by Afterburned » Nov 29, 2017 10:12 am

I have the same issue (2012r2 vm on 2016 host)

My workaround for this is:
Check this is the issue by taking a snapshot in Hyper-V manager - should fail.
Edit the VM settings in cluster manager - take off the backup integration service.
Take a checkpoint from Hyper-V manager (should work) - then delete the checkpoint.
Edit the VM settings in cluster manager - turn the backup integration service back on.
Take a checkpoint - should work. Delete it again just to be neat.

Next backup works. Please note I am not using Veeam but have the same issue with another backup software and this thread has helped focus my investigations (MS bug) so am sharing.

Mike Resseler
Veeam Software
Posts: 4825
Liked: 509 times
Joined: Feb 08, 2013 3:08 pm
Full Name: Mike Resseler
Location: Belgium
Contact:

Re: 9.5 Hyper-V 2016 Known Issues

Post by Mike Resseler » Nov 30, 2017 8:30 am

Aaron,

First: Welcome to the forums!
Second: Thanks for sharing your tip! Really appreciate it and I am rooting for you to get further with MSFT. This bug exists for too long imho ;-)

Mike Resseler
Veeam Software
Posts: 4825
Liked: 509 times
Joined: Feb 08, 2013 3:08 pm
Full Name: Mike Resseler
Location: Belgium
Contact:

Re: 9.5 Hyper-V 2016 Known Issues

Post by Mike Resseler » Nov 30, 2017 8:31 am

Ricardo,

Am I right in seeing that you have the issue with a 2k16 VM and not with a 2k12 VM? In that case something else is wrong and I would advice you to create a support call. Post the support case ID here and the follow-up after investigation of our engineers
Thanks
Mike

ricardo.simoes
Novice
Posts: 4
Liked: never
Joined: Jun 29, 2017 4:05 pm
Contact:

Re: 9.5 Hyper-V 2016 Known Issues

Post by ricardo.simoes » Dec 07, 2017 2:50 pm

Hi @Mike Resseler. Yes, you're right. My Host and VM have installed Win2k16.

However. I've been testing the following:

I disabled the "Enable application-aware processing" option and the backup job ran successfully.
When I re-enable the "Enable application-aware processing" option, it continues to not back up. Does it fail because of the SQL instance created by System Center ?!

It also had the same problem with creating VM checkpoints in Hyper-V (problem identified by @Afterburned). After the workaround has been performing Checkpoints but the backup job continues to fail.

After these, Veeam returns the following error:

"Unable to subscribe to guest processing components: Can not do HvPrepareSnapshot Error: Failed to call RPC function 'Vss.HvPrepareSnapshot': Error code: 0x80004005 Failed to invoke func [HvPrepareSnapshot]: Unspecified error Failed to inject VSS plug-in. PID is NULL.
Failed to create VM recovery checkpoint (mode: Veeam application-aware processing) Details: Failed to prepare guests for volume snapshot.
Error: Failed to prepare guests for volume snapshot. "

I apologize for my English...
Best regards

Post Reply

Who is online

Users browsing this forum: No registered users and 3 guests