9.5 Hyper-V 2016 Known Issues

[frankive]

+1

[derek]

I ran across this issue today and did some searching.

I found a post https://superuser.com/questions/1070464 ... 0x800423f4 Where IntelXX mentioned that if you set that VM's checkpointing to Standard instead of the new Production type it will work.
I was able to take a checkpoint using the HyperV interface after changing this.
I'm running a Veeam backup job now with the new setting for my 2008R2 DC VM.

[frankive]

Worked like a champ!
Veeam, agree this is an good solution for your backups?

[AlexLeadingEdge]

I tried it on the overnight backups but it didn't work for us

Still seeing these warning messages each night for the SBS 2011 VM on a Server 2016 Hyper-V Host, being backed up from a non-VM Windows 10 backup server (bold is the error messages).

We need Application Aware Processing as it truncates the Exchange logs each night.

It has been eight months and we still have no solution

Case # 02039027

Code: Select all

[b]10/07/2017 11:55:34 p.m. :: Failed to create VM recovery checkpoint (mode: Veeam application-aware processing with failover) Details: Job failed ('Checkpoint operation for 'ServerNameRedacted' failed. (Virtual machine ID AC036EA6-E0F8-44C5-9E21-4340898C97FA)'). Error code: '32768'.
Failed to create VM recovery snapshot, VM ID 'ac036ea6-e0f8-44c5-9e21-4340898c97fa'.[/b]
10/07/2017 11:55:47 p.m. :: Retrying snapshot creation attempt (Failed to create production checkpoint.)  
10/07/2017 11:55:49 p.m. :: Required backup infrastructure resources have been assigned  
[b]10/07/2017 11:56:00 p.m. :: Guest processing skipped (check guest OS VSS state and hypervisor integration components version)[/b]
10/07/2017 11:56:00 p.m. :: Creating VM recovery checkpoint (mode: Crash consistent)  
10/07/2017 11:56:18 p.m. :: Using source proxy 192.168.9.25 (onhost)  
10/07/2017 11:56:54 p.m. :: Saving config.wmi  
10/07/2017 11:57:11 p.m. :: VMRS configuration file (60.1 KB) 60.1 KB read at 60 KB/s 
10/07/2017 11:57:16 p.m. :: VMCX configuration file (166.3 KB) 166.3 KB read at 166 KB/s 
10/07/2017 11:57:47 p.m. :: Hard disk 1 (571.0 GB) 37.1 GB read at 65 MB/s [CBT] 
10/07/2017 11:57:51 p.m. :: Hard disk 2 (281.0 GB) 8.1 GB read at 45 MB/s [CBT] 
11/07/2017 12:02:30 a.m. :: Hard disk 3 (150.0 GB) 2.9 GB read at 38 MB/s [CBT] 
11/07/2017 12:04:58 a.m. :: Hard disk 4 (300.0 GB) 2.2 GB read at 100 MB/s [CBT] 
11/07/2017 12:08:05 a.m. :: Finalizing  
11/07/2017 12:08:07 a.m. :: Swap file blocks skipped: 15.6 GB  
11/07/2017 12:08:07 a.m. :: Deleted file blocks skipped: 3.2 GB  
11/07/2017 12:08:07 a.m. :: Busy: Source 42% > Proxy 11% > Network 9% > Target 55%  
11/07/2017 12:08:07 a.m. :: Primary bottleneck: Target  
11/07/2017 12:08:07 a.m. :: Network traffic verification detected no corrupted blocks  
11/07/2017 12:08:07 a.m. :: Processing finished with warnings at 11/07/2017 12:08:07 a.m.

[frankive]

I am no engineer expert on Veeam, but doesn't this log says you take a "production checkpoint"? (Line 4).
I think it should say standard checkpoint after you have changed this in the checkpoint settings for the VM, but I may be wrong.

[AlexLeadingEdge]

Good spotting. I have just changed it back to Standard and run it again but it doesn't seem to change that message. I can test it again on the overnight backup but I don't expect that it will change (it finished with a warning the other night).

[jed-hyper]

I ran across this issue today and did some searching.

I found a post https://superuser.com/questions/1070464 ... 0x800423f4 Where IntelXX mentioned that if you set that VM's checkpointing to Standard instead of the new Production type it will work.
I was able to take a checkpoint using the HyperV interface after changing this.
I'm running a Veeam backup job now with the new setting for my 2008R2 DC VM.

I have tried changing the checkpoints setting from 'Production checkpoints' to Standard checkpoints' on the virtual machine but I'm still having the same issue.
I'm Running Windows Server 2016 Hyper-V Host with Windows 2012 R2 virtual machine which is a DC with Veeam application-aware processing enabled.
Veeam error: Failed to create VM recovery checkpoint (mode: Veeam application-aware processing) Details: Job failed (''). Error code: '32768'.
Have Microsoft or Veeam acknowledged this issue yet ?

[Tallwood]

Also having issues with backing up 2016 DC's on 2016 hyper v. I must say Veeam supports response is dreadful, basically just fobbed it off as an MS issue, no evidence that is in fact an MS issue no internal KB that this is a known issue with veeam and no links to bug ID's with MS either basically just left to sort it out ourselves.

Worst of all when veeam has attempted to backup some of these VM's its created checkpoints which then fail to merge when the VM is shutdown and then completely bork the vm itself.
Backup software that doesnt backup supported applications, corrupts VM's and support which isnt particularly proactive, nice one veeam!

Support ID's 02218107 & 02213337

[nmdange]

If the issue is fixed when Veeam App-Aware processing is disabled and Hyper-V Native Quiescence is used instead, then I'm inclined to side with Microsoft that this is a Veeam issue.

[pedropgracio]

Case ID# 02198270 and 02211099

Hi Mike, greetings to you all,

same problem here Hyper-V 2016 Cluster with 2012r2 DCs, Veeam support was not very helpful. The support engineer had made remote support session only to prove that the problem is related to Microsoft not appointing any solution. On the other hand I have a similar problem with the replica Job of my SQL server, when it tries to create a checkpoint, on the GuestVM, it hangs on Merging Checkpoint State and crash my Hyper-V Node, problem reported to Microsoft and there is no related bug reported.

[TT-DL]

Have Microsoft or Veeam acknowledged this issue yet ?

No. I have an open support case with them in relation to backing up Exchange 2010 or 2k8r2 and the ADAM VSS writer failing. Microsoft are busy referring me from department to department.

[TT-DL]

If the issue is fixed when Veeam App-Aware processing is disabled and Hyper-V Native Quiescence is used instead, then I'm inclined to side with Microsoft that this is a Veeam issue.

I have done alot of work on this issue with both Veeam and Microsoft. Full VSS trace logs have been taken and anaylsed. The summary of that analysis from Veeam is the following:

<Start Snip>
I found more information in the vsstrace. I got some help from a colleague and he was able to find something that will probably be useful.

At the end of the snapshot process, there is an event called onPostSnapshot. At this point the ADAM writer tries to open its own files inside the shadow copy. This is whats failing and ultimately causing the writer to fail. writers sometimes do try to access snapshots like this. But for this the snapshot needs to have an attribute autorecovery. in Hyper-V 2016 this attribute is not set for these snapshots and the snapshot is made read only at the previous event onThaw.

It looks like at least ADAM writer is not working as its suppose to or its not aware of changes related to Hyper-V 2016. this attempt is what we see in the event viewer logs as the file being locked. I'm guessing this is the same issue for the NTDS writer as well.
</End Snip>

[hintzen_comito]

Hi,

I have this error with a full 2016 system.
Strange thing is, everathing else can be backuped with veeam, but the error still happens on the Backup Server itself (A virtualised server 2016)
All servers in this network are still at version 1607.

[hintzen_comito]

So I have a error with a Server 2016, that is virtualised with hyper-v
everything else, espect this server, can be backuped without an error, but at this server I get the 32770 error, so I can't make a backup of my backupserver.
All servers, also the Hyper-V, are on server 2016 with build 1607.
Don'T know what the problem is. Does someone know a workaround or fix

[JTT]

How long has this issue been up?? At this point i as a customer, do not really care who´s fault it is!
Veeam, i have bought Your software, i have paid support for support and it is also Your responsibility to fix it, even if it is Microsofts fault ,then work together, to get this issue resolved.
It seems absurd, that today with Server 2016, i cannot do application aware backup of my most critical servers with DC/Exchange/SQL.

[nandyol]

So this problem is still not fixed then?

• Application-aware processing of Active Directory domain controllers running on guest OS other than Windows Server 2016 fails with the “Failed to create VM recovery checkpoint” error (32770). This is a Hyper-V 2016 issue that Microsoft is planning to address in the future update.

[nvolkmer]

Hi,
we've got the same issue with the backing up VMs on WS2016 and AAP.

The only chance is to disable it. After you got your backups, you can still use AAP to restore items (just open the LDAP files and so on..)
After a Little bit of Research and contacting our Microsoft partner for this matter, in August there will be a public hotfix for this, as this is already be testing in a 1:1 lab enviornment.

But i agree, that this i a really bad behavior.. we are using a 2-node Cluster with around 120 VMs.

Regards

[frankive]

beside changing from production to stnadard checkpoints I also use VAW inside of my domain controllers. Then I always have a 100% nice backup of the DC database, and can also do item-level restore by freshring the repository we use for it directly in the VBR console.

[dredd123]

I've given up waiting for a solution and replaced our guest DC's with Windows Server 2016. Now backing up OK using production checkpoints.

[TT-DL]

Hi,

After a Little bit of Research and contacting our Microsoft partner for this matter, in August there will be a public hotfix for this, as this is already be testing in a 1:1 lab enviornment.

Are you able to provide any links to KB articles or similiar of said hotfix? I have open support cases with MSft about this and they are still failing to acknowledge it as an issue. So if you have any references I could share with them that would be good. Thanks in advance.

[Mike Resseler]

Nils,
If you have anything public I would like to know that also. I am trying from my side to get some acknowledge from MSFT but...

[frankive]

a temporary solution is also to use a VAW inside a DC until this is fixed..

[nmdange]

a temporary solution is also to use a VAW inside a DC until this is fixed..

A better solution is to just disable Application Aware Processing. You do not lose any functionality by doing this with DCs (or any other VMs other than SQL/Exchange/Oracle)

[frankive]

A better solution is to just disable Application Aware Processing. You do not lose any functionality by doing this with DCs (or any other VMs other than SQL/Exchange/Oracle)

I don't think this is a good solution. Active Directory also need the Application Aware Processing to take a reliable backup. Veeam, please correct me if I am a wrong regarding this.
In this matter I find it more safe to also have a VAW agent installed on of my DC if you are disabling AAP.

[nmdange]

Both Hyper-V Native Quiescence and Veeam AAP will perform an application-consistent VSS snapshot within the VM. AAP is NOT required for application-consistent backups. With Native Hyper-V Quiescence, the only difference is that the Hyper-V host is using the integration services to tell the VM to do the VSS snapshot rather than Veeam connecting over the network. In fact this has some advantages because you can do application-consistent backups without Veeam needing to have admin credentials or even network connectivity to the guest VM. This is especially important with DCs as you can forego storing Domain Admin credentials within Veeam.

Application-Aware processing does provide functionality for truncating transaction logs for SQL/Exchange/Oracle if you need that. On the other hand, if your SQL server databases are all in SIMPLE recovery mode, then Application-Aware processing is not needed there either because there are no logs to truncate.

Disabling AAP does not prevent any of the Veeam Explorer tools from working either. I have it disabled on my DCs and Veeam Explorer for AD still works fine.

[mkaec]

Thanks nmdange! I've been trying to figure this out for a while. The documentation makes it sound like AAP is essential for proper operation.

I really like the idea about not needing to assign domain admin credentials to Veeam.

[Taasadi]

Any update on production checkpoint issue with windows 2012 R2 virtual machines? I couldn't downgrade the integration services as some suggested! for me running checkpoint from the Hyper-v manager succeed but exchange 2016 report that the backup was canceled by the client!

[JTT]

After a Little bit of Research and contacting our Microsoft partner for this matter, in August there will be a public hotfix for this, as this is already be testing in a 1:1 lab enviornment.

Any update on this public hotfix?

[dschuler]

Do we know if this was supposed to be a hotfix or was it going to be included in the August CU? I am willing to patch my 2016 Hyper-V Cluster with the CU and try to backup a guest cluster to test.

[TT-DL]

Hi Guys,

From my conversations with 2nd/3rd level support with both Veeam and Microsoft I don't think any 2k16 updates will resolve this issue. The fix is going to need to be specific to the ADAM VSS writer on whatever flavor or exchange you are running or the NTDS VSS writer for AD. Though in relation to AD as pointed out in this forum you don't actually need AAP for AD backups. You only need AAP if you are trying to flush logs ie SQL/Exchange. My SQL is working fine with AAP at present.

Getting production checkpoints working for AD/Exchange is the same fix.