9.5 Hyper-V 2016 Known Issues

Discussions specific to Microsoft Hyper-V hypervisor

Re: 9.5 Hyper-V 2016 Known Issues

Veeam Logoby nmdange » Mon Oct 30, 2017 5:41 pm

It is possible to open AD Explorer without AAP. In the Veeam console, go to Backups->Disks in the left navigation pane. Then search for the DC VM you want to restore from. Select the VM and then on the top menu in the Backup tab, select "Applications Items" and then "Microsoft Active Directory".

However, you still need to have Hyper-V Native Quiescence enabled with production checkpoints to get a consistent backup. I have not had any issues backing up 2012 R2 DCs on a 2016 host with this setup.
nmdange
Expert
 
Posts: 298
Liked: 71 times
Joined: Thu Aug 20, 2015 9:30 pm

Re: 9.5 Hyper-V 2016 Known Issues

Veeam Logoby mkaec » Mon Oct 30, 2017 8:35 pm

nmdange wrote:...go to Backups->Disks in the left navigation pane. Then search for the DC VM you want to restore from. Select the VM and then on the top menu in the Backup tab, select "Applications Items" and then "Microsoft Active Directory".


Thanks. I had tried this earlier, but I was in the wrong job at the time. It worked when I tried it again from the correct job.

However, you still need to have Hyper-V Native Quiescence enabled with production checkpoints to get a consistent backup. I have not had any issues backing up 2012 R2 DCs on a 2016 host with this setup.


First, if you have production checkpoints enabled, try manually making a checkpoint in Hyper-V Manager. It should fail.

Then check the Microsoft-Windows-Hyper-V-VMMS-Admin event log around the time of your backups. You should see some events in there being logged about failing checkpoints.

And in your backup job, you might want to try leaving guest quiescence enabled, but turning off the "crash consistent backup" fallback option. This should cause the backup to fail.

From what I can tell, the quiescence checkpoint is failing and the backup is falling back to a crash consistent backup. Veeam just isn't letting us know in the job log that there were problems.
mkaec
Expert
 
Posts: 222
Liked: 49 times
Joined: Thu Jul 16, 2015 1:31 pm
Full Name: Marc K

Re: 9.5 Hyper-V 2016 Known Issues

Veeam Logoby nmdange » Mon Oct 30, 2017 9:12 pm

Well you're definitely right about production checkpoints failing. Looks like I might have to open a ticket with Microsoft on that one :)
nmdange
Expert
 
Posts: 298
Liked: 71 times
Joined: Thu Aug 20, 2015 9:30 pm

Re: 9.5 Hyper-V 2016 Known Issues

Veeam Logoby nmdange » Mon Nov 06, 2017 3:01 pm

Well according to Microsoft this specific issue (2012 R2 DC on 2016 host) is a known bug. Waiting on some more details but it sounds like the fix is scheduled to be included in the January Monthly update.
nmdange
Expert
 
Posts: 298
Liked: 71 times
Joined: Thu Aug 20, 2015 9:30 pm

Re: 9.5 Hyper-V 2016 Known Issues

Veeam Logoby mkaec » Mon Nov 06, 2017 3:10 pm

That would be excellent. Did the source confirm the year of the expected fix? Given that the issue was known in Nov 2016, I would have thought a fix would have been delivered by Jan 2017.
mkaec
Expert
 
Posts: 222
Liked: 49 times
Joined: Thu Jul 16, 2015 1:31 pm
Full Name: Marc K

Re: 9.5 Hyper-V 2016 Known Issues

Veeam Logoby nmdange » Mon Nov 06, 2017 3:52 pm

I meant next January :)
nmdange
Expert
 
Posts: 298
Liked: 71 times
Joined: Thu Aug 20, 2015 9:30 pm

Re: 9.5 Hyper-V 2016 Known Issues

Veeam Logoby gotsch » Thu Nov 09, 2017 6:05 am

Hi everyone, I would like to confirm this great MS Bug, show our situation and maybe someone has a workaround until MS fix this.

Situation:
- 2016 HyperV Cluster, 2 nodes, 105 VMs, most of them 2012R2
- VMs are different customers, so they are all seperated with VLANs. The Cluster and Backup Server itself have no network connection within the VMs.
- Backup Server is a dedicated Server 2016 Machine with Veeam 9.5, fully patched
- Storage is a HBA SAN attached to the cluster nodes

What happens:
- Since our Backup and Cluster has no Network Connection in the VMs, we use HyperV Quiescence for years now, without any problem until 2016 HypeV Hosts. HyperV Quiescence is set in the Backupjob with the fallback option enabled.
- Some VMs Backup may failling, VMs couldn't be accessed anymore, VMs are stucked in Cluster/HyperV with the status "Creating Snapshot..." or "Merging Snapshot...."
- Only Solution ist to move all uneffected VMs to the other Host and then hard reset the failed Host with the failed VM. It is not possible to kill the vmms.exe from the failing VM
- Last two situations where this happens: today in the early hours and 2 days a go. Affected VMs: 2012R2
- It doesn't makes a different if Production Snapshot or Standard Snapshot is set for the VM

MS Support:
MS has confirmed this Bug in our case, the technican has no ETA of a bugfix, so we are all a little bit f***ed up by Microsoft.

Veeam Suport:
Escelated, but they are far behind MS (which means a lot :-D ), the last thing the told us was that it is not the known bug... sure... so no help here.

What I tried:
- I disabled the CSV Cache. I found a forum where some did this with success, but in our case, It didn't help.

What I maybe will do:
- Put the DCs (2012R2) in a seperate Backupjob and disable the fallback option in HyperV Quiescence. <-- anyone tried this? Will the disabled fallback help?

Someone... has any other ideas? Application Aware Backup is no option an as I read it, it will fail anyway.

Thx for your help in advance!

Patrick
gotsch
Novice
 
Posts: 7
Liked: never
Joined: Thu Nov 09, 2017 5:42 am
Full Name: Patrick Gotsch

Re: 9.5 Hyper-V 2016 Known Issues

Veeam Logoby nmdange » Thu Nov 09, 2017 3:21 pm

It sounds like your issue is different. Microsoft did provide me with a temporary workaround to get production checkpoints working on 2012 R2 DCs. The issue will come back but it does fix it temporarily. Not sure if it will help in your case or not. I have observed the hung vmms.exe a few times but I hadn't connected it with the 2012 R2 DC issue.

Issue: Production Checkpoints failing for 2012 R2 DC on 2016 host

This is a known issue and a bug is filed on the same. Microsoft is currently working on this and would be releasing fix in upcoming months.

Bug number: 13748809

As a workaround, kindly restart the below services in a sequence in VM and then try the checkpoint:

COM+ System Application Service
Com+ Event System
Cryptographic Services
Volume Shadow Copy Service
Microsoft Software Shadow copy provider
nmdange
Expert
 
Posts: 298
Liked: 71 times
Joined: Thu Aug 20, 2015 9:30 pm

Re: 9.5 Hyper-V 2016 Known Issues

Veeam Logoby mkaec » Thu Nov 09, 2017 5:36 pm

gotsch wrote:- Put the DCs (2012R2) in a seperate Backupjob and disable the fallback option in HyperV Quiescence. <-- anyone tried this? Will the disabled fallback help?

I had tried this with my backup job and this caused the backup to fail outright. My conclusion was that quiescence was failing and falling back every time, Veeam just wasn't telling me about this in its job log. I ended up turning off quiescence altogether, but hope that Microsoft will come through with a patch in January and I'll be able to turn it back on.
mkaec
Expert
 
Posts: 222
Liked: 49 times
Joined: Thu Jul 16, 2015 1:31 pm
Full Name: Marc K

Re: 9.5 Hyper-V 2016 Known Issues

Veeam Logoby gotsch » Thu Nov 09, 2017 6:53 pm

mkaec wrote:I had tried this with my backup job and this caused the backup to fail outright. My conclusion was that quiescence was failing and falling back every time, Veeam just wasn't telling me about this in its job log. I ended up turning off quiescence altogether, but hope that Microsoft will come through with a patch in January and I'll be able to turn it back on.



Ty for this info, I will disable it for my seperate DC Job also.


nmdange wrote:It sounds like your issue is different. Microsoft did provide me with a temporary workaround to get production checkpoints working on 2012 R2 DCs. The issue will come back but it does fix it temporarily. Not sure if it will help in your case or not. I have observed the hung vmms.exe a few times but I hadn't connected it with the 2012 R2 DC issue.

Issue: Production Checkpoints failing for 2012 R2 DC on 2016 host

This is a known issue and a bug is filed on the same. Microsoft is currently working on this and would be releasing fix in upcoming months.

Bug number: 13748809

As a workaround, kindly restart the below services in a sequence in VM and then try the checkpoint:

COM+ System Application Service
Com+ Event System
Cryptographic Services
Volume Shadow Copy Service
Microsoft Software Shadow copy provider


We got the same "workaround" from Microsoft:

Microsoft wrote:Mögliche Lösungsansätze /zur Zeit ist noch kein Fix von MS erlassen/

1.Host auf die letzte Version aktualisieren /bereits gemacht?/:
https://support.microsoft.com/en-us/help/4038782
2.Auf Gast folgenden KB installieren oder aber Gäste durch Windows Updates mit neuesten Updates versehen: https://support.microsoft.com/en-us/hel ... view-hosts
3.Hyper-V Rolle auf Host deinstallieren und erneut installieren /optional, ich würde dies nicht empfehlen/

Der Ansatz, der erfahrungsgemäß am Besten funktioniert hat– wie im Veeam Forum von einem Kunden beschrieben – ist es, die folgenden Kommandos in Batch zusammen zu fassen, bzw. vor jedem Backup-Job /scheduled task?/ auf allen DC-Gästen ausführen:
net stop BITS
net stop SENS
net stop NtFrs
net stop DFSR
net stop EventSystem
net start EventSystem
net start DFSR
net start NtFrs
net start SENS
net start BITS
net stop VSS
net start VSS
gotsch
Novice
 
Posts: 7
Liked: never
Joined: Thu Nov 09, 2017 5:42 am
Full Name: Patrick Gotsch

Re: 9.5 Hyper-V 2016 Known Issues

Veeam Logoby ricardo.simoes » Tue Nov 21, 2017 3:24 pm

Hi everyone. I have the same problem.

Host -> w2k16 (VBR 9.5)
VM -> w2k16 (VM with System Center 2016)

When I run the backup it gives the following error:
Failed to create VM recovery checkpoint (mode: Veeam application-aware processing) Details: Job failed (''). Error code: '32772'. Failed to create VM recovery snapshot, VM ID 'fd18471c-6ff3-4178-952f-da111b51b246'.

Both OS are updated and I have multiple VM on the same host and I have no problem backing up the remaining.
I have already moved the VM to other Hosts and the problem remains.
At the moment I have 17 VM and I only have problems with 1. :x

Help...
ricardo.simoes
Novice
 
Posts: 4
Liked: never
Joined: Thu Jun 29, 2017 4:05 pm

Re: 9.5 Hyper-V 2016 Known Issues

Veeam Logoby Afterburned » Wed Nov 29, 2017 10:12 am

I have the same issue (2012r2 vm on 2016 host)

My workaround for this is:
Check this is the issue by taking a snapshot in Hyper-V manager - should fail.
Edit the VM settings in cluster manager - take off the backup integration service.
Take a checkpoint from Hyper-V manager (should work) - then delete the checkpoint.
Edit the VM settings in cluster manager - turn the backup integration service back on.
Take a checkpoint - should work. Delete it again just to be neat.

Next backup works. Please note I am not using Veeam but have the same issue with another backup software and this thread has helped focus my investigations (MS bug) so am sharing.
Afterburned
Lurker
 
Posts: 1
Liked: never
Joined: Wed Nov 29, 2017 10:07 am
Full Name: Aaron Davies

Re: 9.5 Hyper-V 2016 Known Issues

Veeam Logoby Mike Resseler » Thu Nov 30, 2017 8:30 am

Aaron,

First: Welcome to the forums!
Second: Thanks for sharing your tip! Really appreciate it and I am rooting for you to get further with MSFT. This bug exists for too long imho ;-)
Mike Resseler
Veeam Software
 
Posts: 4168
Liked: 467 times
Joined: Fri Feb 08, 2013 3:08 pm
Location: Belgium, the land of the fries, the beer, the chocolate and the diamonds...
Full Name: Mike Resseler

Re: 9.5 Hyper-V 2016 Known Issues

Veeam Logoby Mike Resseler » Thu Nov 30, 2017 8:31 am

Ricardo,

Am I right in seeing that you have the issue with a 2k16 VM and not with a 2k12 VM? In that case something else is wrong and I would advice you to create a support call. Post the support case ID here and the follow-up after investigation of our engineers
Thanks
Mike
Mike Resseler
Veeam Software
 
Posts: 4168
Liked: 467 times
Joined: Fri Feb 08, 2013 3:08 pm
Location: Belgium, the land of the fries, the beer, the chocolate and the diamonds...
Full Name: Mike Resseler

Re: 9.5 Hyper-V 2016 Known Issues

Veeam Logoby ricardo.simoes » Thu Dec 07, 2017 2:50 pm

Hi @Mike Resseler. Yes, you're right. My Host and VM have installed Win2k16.

However. I've been testing the following:

I disabled the "Enable application-aware processing" option and the backup job ran successfully.
When I re-enable the "Enable application-aware processing" option, it continues to not back up. Does it fail because of the SQL instance created by System Center ?!

It also had the same problem with creating VM checkpoints in Hyper-V (problem identified by @Afterburned). After the workaround has been performing Checkpoints but the backup job continues to fail.

After these, Veeam returns the following error:

"Unable to subscribe to guest processing components: Can not do HvPrepareSnapshot Error: Failed to call RPC function 'Vss.HvPrepareSnapshot': Error code: 0x80004005 Failed to invoke func [HvPrepareSnapshot]: Unspecified error Failed to inject VSS plug-in. PID is NULL.
Failed to create VM recovery checkpoint (mode: Veeam application-aware processing) Details: Failed to prepare guests for volume snapshot.
Error: Failed to prepare guests for volume snapshot. "

I apologize for my English...
Best regards
ricardo.simoes
Novice
 
Posts: 4
Liked: never
Joined: Thu Jun 29, 2017 4:05 pm

PreviousNext

Return to Microsoft Hyper-V



Who is online

Users browsing this forum: No registered users and 7 guests