Host-based backup of Microsoft Hyper-V VMs.
Post Reply
frankive
Service Provider
Posts: 1092
Liked: 134 times
Joined: May 14, 2013 8:35 pm
Full Name: Frank Iversen
Location: Norway
Contact:

Re: 9.5 Hyper-V 2016 Known Issues

Post by frankive »

+1
derek
Novice
Posts: 3
Liked: never
Joined: Oct 28, 2016 5:05 pm

Re: 9.5 Hyper-V 2016 Known Issues

Post by derek »

I ran across this issue today and did some searching.

I found a post https://superuser.com/questions/1070464 ... 0x800423f4 Where IntelXX mentioned that if you set that VM's checkpointing to Standard instead of the new Production type it will work.
I was able to take a checkpoint using the HyperV interface after changing this.
I'm running a Veeam backup job now with the new setting for my 2008R2 DC VM.
frankive
Service Provider
Posts: 1092
Liked: 134 times
Joined: May 14, 2013 8:35 pm
Full Name: Frank Iversen
Location: Norway
Contact:

Re: 9.5 Hyper-V 2016 Known Issues

Post by frankive »

Worked like a champ!
Veeam, agree this is an good solution for your backups?
AlexLeadingEdge
Veteran
Posts: 472
Liked: 59 times
Joined: Dec 14, 2015 9:42 pm
Contact:

Re: 9.5 Hyper-V 2016 Known Issues

Post by AlexLeadingEdge »

I tried it on the overnight backups but it didn't work for us :cry:

Still seeing these warning messages each night for the SBS 2011 VM on a Server 2016 Hyper-V Host, being backed up from a non-VM Windows 10 backup server (bold is the error messages).

We need Application Aware Processing as it truncates the Exchange logs each night.

It has been eight months and we still have no solution :cry:

Case # 02039027

Code: Select all

[b]10/07/2017 11:55:34 p.m. :: Failed to create VM recovery checkpoint (mode: Veeam application-aware processing with failover) Details: Job failed ('Checkpoint operation for 'ServerNameRedacted' failed. (Virtual machine ID AC036EA6-E0F8-44C5-9E21-4340898C97FA)'). Error code: '32768'.
Failed to create VM recovery snapshot, VM ID 'ac036ea6-e0f8-44c5-9e21-4340898c97fa'.[/b]
10/07/2017 11:55:47 p.m. :: Retrying snapshot creation attempt (Failed to create production checkpoint.)  
10/07/2017 11:55:49 p.m. :: Required backup infrastructure resources have been assigned  
[b]10/07/2017 11:56:00 p.m. :: Guest processing skipped (check guest OS VSS state and hypervisor integration components version)[/b]
10/07/2017 11:56:00 p.m. :: Creating VM recovery checkpoint (mode: Crash consistent)  
10/07/2017 11:56:18 p.m. :: Using source proxy 192.168.9.25 (onhost)  
10/07/2017 11:56:54 p.m. :: Saving config.wmi  
10/07/2017 11:57:11 p.m. :: VMRS configuration file (60.1 KB) 60.1 KB read at 60 KB/s 
10/07/2017 11:57:16 p.m. :: VMCX configuration file (166.3 KB) 166.3 KB read at 166 KB/s 
10/07/2017 11:57:47 p.m. :: Hard disk 1 (571.0 GB) 37.1 GB read at 65 MB/s [CBT] 
10/07/2017 11:57:51 p.m. :: Hard disk 2 (281.0 GB) 8.1 GB read at 45 MB/s [CBT] 
11/07/2017 12:02:30 a.m. :: Hard disk 3 (150.0 GB) 2.9 GB read at 38 MB/s [CBT] 
11/07/2017 12:04:58 a.m. :: Hard disk 4 (300.0 GB) 2.2 GB read at 100 MB/s [CBT] 
11/07/2017 12:08:05 a.m. :: Finalizing  
11/07/2017 12:08:07 a.m. :: Swap file blocks skipped: 15.6 GB  
11/07/2017 12:08:07 a.m. :: Deleted file blocks skipped: 3.2 GB  
11/07/2017 12:08:07 a.m. :: Busy: Source 42% > Proxy 11% > Network 9% > Target 55%  
11/07/2017 12:08:07 a.m. :: Primary bottleneck: Target  
11/07/2017 12:08:07 a.m. :: Network traffic verification detected no corrupted blocks  
11/07/2017 12:08:07 a.m. :: Processing finished with warnings at 11/07/2017 12:08:07 a.m.
frankive
Service Provider
Posts: 1092
Liked: 134 times
Joined: May 14, 2013 8:35 pm
Full Name: Frank Iversen
Location: Norway
Contact:

Re: 9.5 Hyper-V 2016 Known Issues

Post by frankive »

I am no engineer expert on Veeam, but doesn't this log says you take a "production checkpoint"? (Line 4).
I think it should say standard checkpoint after you have changed this in the checkpoint settings for the VM, but I may be wrong.
AlexLeadingEdge
Veteran
Posts: 472
Liked: 59 times
Joined: Dec 14, 2015 9:42 pm
Contact:

Re: 9.5 Hyper-V 2016 Known Issues

Post by AlexLeadingEdge »

Good spotting. I have just changed it back to Standard and run it again but it doesn't seem to change that message. I can test it again on the overnight backup but I don't expect that it will change (it finished with a warning the other night).
jed-hyper
Enthusiast
Posts: 39
Liked: 4 times
Joined: Feb 26, 2014 4:42 am
Full Name: Jed Parkes
Contact:

Re: 9.5 Hyper-V 2016 Known Issues

Post by jed-hyper »

derek wrote:I ran across this issue today and did some searching.

I found a post https://superuser.com/questions/1070464 ... 0x800423f4 Where IntelXX mentioned that if you set that VM's checkpointing to Standard instead of the new Production type it will work.
I was able to take a checkpoint using the HyperV interface after changing this.
I'm running a Veeam backup job now with the new setting for my 2008R2 DC VM.
I have tried changing the checkpoints setting from 'Production checkpoints' to Standard checkpoints' on the virtual machine but I'm still having the same issue.
I'm Running Windows Server 2016 Hyper-V Host with Windows 2012 R2 virtual machine which is a DC with Veeam application-aware processing enabled.
Veeam error: Failed to create VM recovery checkpoint (mode: Veeam application-aware processing) Details: Job failed (''). Error code: '32768'.
Have Microsoft or Veeam acknowledged this issue yet ?
Tallwood
Novice
Posts: 6
Liked: 1 time
Joined: Sep 06, 2012 7:23 pm
Full Name: Tom Allwood
Contact:

Re: 9.5 Hyper-V 2016 Known Issues

Post by Tallwood »

Also having issues with backing up 2016 DC's on 2016 hyper v. I must say Veeam supports response is dreadful, basically just fobbed it off as an MS issue, no evidence that is in fact an MS issue no internal KB that this is a known issue with veeam and no links to bug ID's with MS either basically just left to sort it out ourselves.

Worst of all when veeam has attempted to backup some of these VM's its created checkpoints which then fail to merge when the VM is shutdown and then completely bork the vm itself.
Backup software that doesnt backup supported applications, corrupts VM's and support which isnt particularly proactive, nice one veeam!

Support ID's 02218107 & 02213337
nmdange
Veteran
Posts: 528
Liked: 144 times
Joined: Aug 20, 2015 9:30 pm
Contact:

Re: 9.5 Hyper-V 2016 Known Issues

Post by nmdange »

If the issue is fixed when Veeam App-Aware processing is disabled and Hyper-V Native Quiescence is used instead, then I'm inclined to side with Microsoft that this is a Veeam issue.
pedropgracio
Lurker
Posts: 1
Liked: never
Joined: Jul 12, 2017 5:06 pm
Full Name: Pedro Pinheiro
Contact:

Re: 9.5 Hyper-V 2016 Known Issues

Post by pedropgracio »

Case ID# 02198270 and 02211099

Hi Mike, greetings to you all,

same problem here Hyper-V 2016 Cluster with 2012r2 DCs, Veeam support was not very helpful. The support engineer had made remote support session only to prove that the problem is related to Microsoft not appointing any solution. On the other hand I have a similar problem with the replica Job of my SQL server, when it tries to create a checkpoint, on the GuestVM, it hangs on Merging Checkpoint State and crash my Hyper-V Node, problem reported to Microsoft and there is no related bug reported.
TT-DL
Influencer
Posts: 21
Liked: 2 times
Joined: Feb 14, 2017 2:50 am
Full Name: Daniel L
Contact:

Re: 9.5 Hyper-V 2016 Known Issues

Post by TT-DL »

jed-hyper wrote: Have Microsoft or Veeam acknowledged this issue yet ?
No. I have an open support case with them in relation to backing up Exchange 2010 or 2k8r2 and the ADAM VSS writer failing. Microsoft are busy referring me from department to department.
TT-DL
Influencer
Posts: 21
Liked: 2 times
Joined: Feb 14, 2017 2:50 am
Full Name: Daniel L
Contact:

Re: 9.5 Hyper-V 2016 Known Issues

Post by TT-DL » 1 person likes this post

nmdange wrote:If the issue is fixed when Veeam App-Aware processing is disabled and Hyper-V Native Quiescence is used instead, then I'm inclined to side with Microsoft that this is a Veeam issue.
I have done alot of work on this issue with both Veeam and Microsoft. Full VSS trace logs have been taken and anaylsed. The summary of that analysis from Veeam is the following:

<Start Snip>
I found more information in the vsstrace. I got some help from a colleague and he was able to find something that will probably be useful.

At the end of the snapshot process, there is an event called onPostSnapshot. At this point the ADAM writer tries to open its own files inside the shadow copy. This is whats failing and ultimately causing the writer to fail. writers sometimes do try to access snapshots like this. But for this the snapshot needs to have an attribute autorecovery. in Hyper-V 2016 this attribute is not set for these snapshots and the snapshot is made read only at the previous event onThaw.

It looks like at least ADAM writer is not working as its suppose to or its not aware of changes related to Hyper-V 2016. this attempt is what we see in the event viewer logs as the file being locked. I'm guessing this is the same issue for the NTDS writer as well.
</End Snip>
hintzen_comito
Lurker
Posts: 2
Liked: never
Joined: Jul 14, 2017 7:29 am
Full Name: Philippe Hintzen
Contact:

Re: 9.5 Hyper-V 2016 Known Issues

Post by hintzen_comito »

Hi,

I have this error with a full 2016 system.
Strange thing is, everathing else can be backuped with veeam, but the error still happens on the Backup Server itself (A virtualised server 2016)
All servers in this network are still at version 1607.
hintzen_comito
Lurker
Posts: 2
Liked: never
Joined: Jul 14, 2017 7:29 am
Full Name: Philippe Hintzen
Contact:

Re: 9.5 Hyper-V 2016 Known Issues

Post by hintzen_comito »

So I have a error with a Server 2016, that is virtualised with hyper-v
everything else, espect this server, can be backuped without an error, but at this server I get the 32770 error, so I can't make a backup of my backupserver.
All servers, also the Hyper-V, are on server 2016 with build 1607.
Don'T know what the problem is. Does someone know a workaround or fix
JTT
Service Provider
Posts: 116
Liked: 5 times
Joined: Jan 02, 2017 7:31 am
Full Name: JTT
Contact:

Re: 9.5 Hyper-V 2016 Known Issues

Post by JTT »

How long has this issue been up?? At this point i as a customer, do not really care who´s fault it is!
Veeam, i have bought Your software, i have paid support for support and it is also Your responsibility to fix it, even if it is Microsofts fault ,then work together, to get this issue resolved.
It seems absurd, that today with Server 2016, i cannot do application aware backup of my most critical servers with DC/Exchange/SQL.
nandyol
Novice
Posts: 3
Liked: 1 time
Joined: Jul 05, 2016 11:22 pm
Contact:

Re: 9.5 Hyper-V 2016 Known Issues

Post by nandyol »

So this problem is still not fixed then?
  • Application-aware processing of Active Directory domain controllers running on guest OS other than Windows Server 2016 fails with the “Failed to create VM recovery checkpoint” error (32770). This is a Hyper-V 2016 issue that Microsoft is planning to address in the future update.
nvolkmer
Novice
Posts: 3
Liked: never
Joined: Jul 20, 2017 7:32 am
Full Name: Nils Volkmer
Contact:

Re: 9.5 Hyper-V 2016 Known Issues

Post by nvolkmer »

Hi,
we've got the same issue with the backing up VMs on WS2016 and AAP.

The only chance is to disable it. After you got your backups, you can still use AAP to restore items (just open the LDAP files and so on..)
After a Little bit of Research and contacting our Microsoft partner for this matter, in August there will be a public hotfix for this, as this is already be testing in a 1:1 lab enviornment.

But i agree, that this i a really bad behavior.. we are using a 2-node Cluster with around 120 VMs.

Regards
frankive
Service Provider
Posts: 1092
Liked: 134 times
Joined: May 14, 2013 8:35 pm
Full Name: Frank Iversen
Location: Norway
Contact:

Re: 9.5 Hyper-V 2016 Known Issues

Post by frankive »

beside changing from production to stnadard checkpoints I also use VAW inside of my domain controllers. Then I always have a 100% nice backup of the DC database, and can also do item-level restore by freshring the repository we use for it directly in the VBR console.
dredd123
Novice
Posts: 7
Liked: never
Joined: Feb 24, 2016 10:10 am
Full Name: David Walsh
Contact:

Re: 9.5 Hyper-V 2016 Known Issues

Post by dredd123 »

I've given up waiting for a solution and replaced our guest DC's with Windows Server 2016. Now backing up OK using production checkpoints.
TT-DL
Influencer
Posts: 21
Liked: 2 times
Joined: Feb 14, 2017 2:50 am
Full Name: Daniel L
Contact:

Re: 9.5 Hyper-V 2016 Known Issues

Post by TT-DL »

nvolkmer wrote:Hi,

After a Little bit of Research and contacting our Microsoft partner for this matter, in August there will be a public hotfix for this, as this is already be testing in a 1:1 lab enviornment.
Are you able to provide any links to KB articles or similiar of said hotfix? I have open support cases with MSft about this and they are still failing to acknowledge it as an issue. So if you have any references I could share with them that would be good. Thanks in advance.
Mike Resseler
Product Manager
Posts: 8191
Liked: 1322 times
Joined: Feb 08, 2013 3:08 pm
Full Name: Mike Resseler
Location: Belgium
Contact:

Re: 9.5 Hyper-V 2016 Known Issues

Post by Mike Resseler »

Nils,
If you have anything public I would like to know that also. I am trying from my side to get some acknowledge from MSFT but... :-(
frankive
Service Provider
Posts: 1092
Liked: 134 times
Joined: May 14, 2013 8:35 pm
Full Name: Frank Iversen
Location: Norway
Contact:

Re: 9.5 Hyper-V 2016 Known Issues

Post by frankive »

a temporary solution is also to use a VAW inside a DC until this is fixed..
nmdange
Veteran
Posts: 528
Liked: 144 times
Joined: Aug 20, 2015 9:30 pm
Contact:

Re: 9.5 Hyper-V 2016 Known Issues

Post by nmdange »

frankive wrote:a temporary solution is also to use a VAW inside a DC until this is fixed..
A better solution is to just disable Application Aware Processing. You do not lose any functionality by doing this with DCs (or any other VMs other than SQL/Exchange/Oracle)
frankive
Service Provider
Posts: 1092
Liked: 134 times
Joined: May 14, 2013 8:35 pm
Full Name: Frank Iversen
Location: Norway
Contact:

Re: 9.5 Hyper-V 2016 Known Issues

Post by frankive »

A better solution is to just disable Application Aware Processing. You do not lose any functionality by doing this with DCs (or any other VMs other than SQL/Exchange/Oracle)


I don't think this is a good solution. Active Directory also need the Application Aware Processing to take a reliable backup. Veeam, please correct me if I am a wrong regarding this.
In this matter I find it more safe to also have a VAW agent installed on of my DC if you are disabling AAP.
nmdange
Veteran
Posts: 528
Liked: 144 times
Joined: Aug 20, 2015 9:30 pm
Contact:

Re: 9.5 Hyper-V 2016 Known Issues

Post by nmdange » 3 people like this post

Both Hyper-V Native Quiescence and Veeam AAP will perform an application-consistent VSS snapshot within the VM. AAP is NOT required for application-consistent backups. With Native Hyper-V Quiescence, the only difference is that the Hyper-V host is using the integration services to tell the VM to do the VSS snapshot rather than Veeam connecting over the network. In fact this has some advantages because you can do application-consistent backups without Veeam needing to have admin credentials or even network connectivity to the guest VM. This is especially important with DCs as you can forego storing Domain Admin credentials within Veeam.

Application-Aware processing does provide functionality for truncating transaction logs for SQL/Exchange/Oracle if you need that. On the other hand, if your SQL server databases are all in SIMPLE recovery mode, then Application-Aware processing is not needed there either because there are no logs to truncate.

Disabling AAP does not prevent any of the Veeam Explorer tools from working either. I have it disabled on my DCs and Veeam Explorer for AD still works fine.
mkaec
Veteran
Posts: 465
Liked: 136 times
Joined: Jul 16, 2015 1:31 pm
Full Name: Marc K
Contact:

Re: 9.5 Hyper-V 2016 Known Issues

Post by mkaec »

Thanks nmdange! I've been trying to figure this out for a while. The documentation makes it sound like AAP is essential for proper operation.

I really like the idea about not needing to assign domain admin credentials to Veeam.
Taasadi
Lurker
Posts: 1
Liked: never
Joined: Jul 31, 2017 9:38 pm
Contact:

Re: 9.5 Hyper-V 2016 Known Issues

Post by Taasadi »

Any update on production checkpoint issue with windows 2012 R2 virtual machines? I couldn't downgrade the integration services as some suggested! for me running checkpoint from the Hyper-v manager succeed but exchange 2016 report that the backup was canceled by the client!
JTT
Service Provider
Posts: 116
Liked: 5 times
Joined: Jan 02, 2017 7:31 am
Full Name: JTT
Contact:

Re: 9.5 Hyper-V 2016 Known Issues

Post by JTT »

nvolkmer wrote: After a Little bit of Research and contacting our Microsoft partner for this matter, in August there will be a public hotfix for this, as this is already be testing in a 1:1 lab enviornment.
Any update on this public hotfix?
dschuler
Novice
Posts: 8
Liked: never
Joined: Aug 29, 2016 4:35 pm
Full Name: Dann Schuler
Contact:

Re: 9.5 Hyper-V 2016 Known Issues

Post by dschuler »

Do we know if this was supposed to be a hotfix or was it going to be included in the August CU? I am willing to patch my 2016 Hyper-V Cluster with the CU and try to backup a guest cluster to test.
TT-DL
Influencer
Posts: 21
Liked: 2 times
Joined: Feb 14, 2017 2:50 am
Full Name: Daniel L
Contact:

Re: 9.5 Hyper-V 2016 Known Issues

Post by TT-DL »

Hi Guys,

From my conversations with 2nd/3rd level support with both Veeam and Microsoft I don't think any 2k16 updates will resolve this issue. The fix is going to need to be specific to the ADAM VSS writer on whatever flavor or exchange you are running or the NTDS VSS writer for AD. Though in relation to AD as pointed out in this forum you don't actually need AAP for AD backups. You only need AAP if you are trying to flush logs ie SQL/Exchange. My SQL is working fine with AAP at present.

Getting production checkpoints working for AD/Exchange is the same fix.
Post Reply

Who is online

Users browsing this forum: Google [Bot] and 11 guests