Failed to create VM recovery checkpoint

[AlexLeadingEdge]

Hi guys!

We have just finished moving Hyper-V VMs from our old Windows Server 2012 R2 Hyper-V host to our new Windows Server 2016 Hyper-V host. All the VMs work exceptionally well but when we go to do a backup of the SBS 2011 server using B&R 9.5 it spits up this error after 11 seconds:

15/01/2017 11:47:48 p.m. :: Failed to create VM recovery checkpoint (mode: Veeam application-aware processing with failover) Details: Job failed ('Checkpoint operation for 'SBS-SERVER' failed. (Virtual machine ID B854900D-2DA6-44D0-9E3A-24D5D4700368)'). Error code: '32768'.
Failed to create VM recovery snapshot, VM ID 'b854900d-2da6-44d0-9e3a-24d5d4700368'.

15/01/2017 11:48:00 p.m. :: Retrying snapshot creation attempt (Failed to create production checkpoint.)

I will open a case when I can and update this ticket.

(For privacy reasons I have changed the name of the SBS server in the error message)

[AlexLeadingEdge]

I see here that there is a known issue with Server 2016 and B&R 9.5 (different error message though!):

microsoft-hyper-v-f25/9-5-hyper-v-known ... 38927.html

Application-aware processing of Active Directory domain controllers running on guest OS other than Windows Server 2016 fails with the “Failed to create VM recovery checkpoint” error (32770). This is a Hyper-V 2016 issue that Microsoft is planning to address in the future update.

[AlexLeadingEdge]

Case # 02039027

[Mike Resseler]

Alex,

For my understanding, your SBS server is running as a VM on a hyper-v 2016 host? In that SBS server is normally a domain controller so while the message might be different, I do think that you should look in that direction. But since I don't have any logs, please keep working with your support engineer to find the root cause

Thanks
Mike

[AlexLeadingEdge]

Could you please check if you can manually create a Checkpoint?

Also, verify if the VSS writers on the host are stable. You can run the command
Vssadmin list writers

I can manually create a checkpoint on the Hyper-V Host.

VSS writers are stable on both the backup server and the VM (the SBS Server).

The backup last night again spat up this same error.

[AlexLeadingEdge]

For my understanding, your SBS server is running as a VM on a hyper-v 2016 host?

Correct.

In that SBS server is normally a domain controller so while the message might be different, I do think that you should look in that direction. But since I don't have any logs, please keep working with your support engineer to find the root cause.

Will do. The backup last night again spat up this same error.

[Perry]

I have a job that backs up 6 VMs two of which are domain controllers. While the non domain controllers back up fine the domain controllers fail with this error:

Code: Select all

Failed to create VM recovery checkpoint (mode: Veeam application-aware processing) Details: Job failed (''). Error code: '32768'. Failed to create VM recovery snapshot, VM ID '50dfbcce-ca37-426c-a280-a83ac9d2e454'.
Retrying snapshot creation attempt (Failed to create production checkpoint.)
Task has been rescheduled
Queued for processing at 1/25/2017 1:41:29 AM
Unable to allocate processing resources. Error: Job failed (''). Error code: '32768'. Failed to create VM recovery snapshot, VM ID '50dfbcce-ca37-426c-a280-a83ac9d2e454'.

Any ideas?

[Perry]

I want to add that the backups for the DCs work fine if "Application-Aware Processing" is disabled.

[veremin]

Do you use Hyper-V 2016? Does DC run on OS other than Windows Server 2016? If so, be aware that you've come across known Hyper-V 2016 issue. Thanks.

[Perry]

Since application-aware processing has to be disabled for Domain Contoller VMs until the bug is fixed, should we enable guest quiescence? If so, how do we enable this at the individual VM level? Enabling quiescence seems to be global to the entire backup.

[veremin]

I don't think enabling Hyper-V quiescence is likely to help here, as it would leverage similar framework as AAIP does. Thanks.

[Perry]

So what's the recommended setting for backing up DC servers until the bug is fixed? Completely disable Application-Aware processing?

[veremin]

Yep, either disable AAIP (till the issue gets fixed by MS) or update DC to Windows Server 2016.

[carussell]

I am having the exact same issue but there is a twist in the information.

HV host updated from HV server 2012r2 to server 2016 datacenter. all the VM's that I manually MOVED to the updated host will complete a backup with a warning that they should be updated to hardware version 8. A newly created VM (windows 7 ent x32) will not backup and fails:

Failed to create VM recovery checkpoint (mode: Hyper-V child partition snapshot) Details: Job failed ('Checkpoint operation for 'Win 7 Ent x32' failed. (Virtual machine ID xxxxxxxx)'). Error code: '32768'. Failed to create VM recovery snapshot, VM ID 'xxxxxxxxx'.
Retrying snapshot creation attempt (Failed to create production checkpoint.)
Task has been rescheduled
Queued for processing at 2/1/2017 8:56:02 AM
Unable to allocate processing resources. Error: Job failed ('Checkpoint operation for 'Win 7 Ent x32' failed. (Virtual machine ID xxxxxxxx)'). Error code: '32768'. Failed to create VM recovery snapshot, VM ID 'xxxxxxxxx'.

Domain controllers are windows 2003.... but they are NOT VM's they are separate physical machines. So while this seems to be related it doesn't seem as though the known bug is this issue as that is about older OS DC's as Guests on Win2016 Hosts.

[mvalpreda]

Did this ever get sorted on either the Microsoft or Veeam side? I just did a 2012 R2 to 2016 migration and the 2012 R2 DC is not happy.

[Mike Resseler]

Hi Mark,

No this issue still exists. As written above, you should disable AAIP for the DC. The bug is unfortunately still there and we haven't got any notification that MSFT is solving it

[rsanders]

Still having this issue as I just migrated an SBS server to a 2016 host. It won't be there long as it being decommissioned soon, but it would still be nice to have guest processing.

[AlexLeadingEdge]

Nine months on I'm still having this issue

[mazzu]

Same problem here, same scenario, Hyper-V 2016 with a 2012 R2 Domain Controller VM.
Backups for the DC work fine if "Application-Aware Processing" is disabled.

[sbaltic]

Same problem with 2016 host and 2016 VM (not DC) ....

11.10.2017 20:01:26 :: Failed to create VM recovery checkpoint (mode: Veeam application-aware processing) Details: Job failed (''). Error code: '32772'.
Failed to create VM recovery snapshot, VM ID '6604883e-fc6e-4403-b72d-454d3741377b'.

[IHT_Michael]

Yep, either disable AAIP (till the issue gets fixed by MS) or update DC to Windows Server 2016.

How can we disable AAIP with your scheduled PowerShell script and VB&R Free? I'm assuming I'm seeing this same issue. I have Windows Server 2016 for the Hyper-V host with four virtuals running. The two that are not DCs back up fine. The two that are DCs have this Error 32768. But I'm doing this from scheduled PowerShell scripts.

Thanks,
Michael

[foggy]

Hi Michael, there's no AAIP in the free version.

[cpfleger]

Got the same problem after I v2ved a Win2k12r2 file server from vSphere 6.5 to Hyper-V 2016 (using a Veeam Agent for Windows 2.0 free backup Instant recovering to Hyper-V host and finalizing the Hyper-V vm), had both app-aware and Hyper-V quiescence enabled in the backup job.

Tried again after deinstalling VAW and VMware tools, but same result.

vssadmin list writers shows a bunch of vss writers failing due to timeouts (System Writer, ASR Writer, FSRM Writer, Shadow Copy Optimization Writer, Registry Writer, COM+ REGDB Writer, Dedup Writer, WMI Writer). Will check if I can start/fix the services behind.

Hints and tips welcome.

Regards!

[cpfleger]

System Writer => Service: Cryptographic Services
ASR Writer / Registry Writer / COM+ REGDB Writer / Shadow Copy Optimization Writer => Service: Volume Shadow Copy
FSRM writer => Service: File Server Resource Manager
Dedup Writer => Data Deduplication Volume Shadow Copy Service
WMI Writer => Windows Management Instrumentation

Restarted all those services, restarted vm: all services running
Tried again VBR backup job but failed again, vsslist writers again showed vss writers failed because of timeout. Hmm ...

Disabled application aware processing (without restarting VSS Writer services), now backup has run through.

Ideas still welcome!

Regards!

[Mike Resseler]

Claus,

Is that server up-to-date on hotfixes and updates?

[AlexLeadingEdge]

Disabled application aware processing (without restarting VSS Writer services), now backup has run through.

Hi cpfleger, my understanding is that you really don't want to do that, as this step quietens (quiesces) an active database system such as SQL or an Exchange Server, so that the backup doesn't end up corrupt!

[ahickingbottom]

Just checking to see the status on this issue with everyone else. I am still having the same issues. We are running Hyper-V on Server 2016 and the VM with the issue is a 2012 R2 running Microsoft SQL. We do not want to disable Application-Aware Processing just to get the job to complete because it runs SQL. Any information on a proper fix would be appreciated.

[AlexLeadingEdge]

One year now and still no resolution

[Mike Resseler]

Andrew,

First: Welcome to the forums!

I think your issue might be a bit different. The problem is with DC's. There are MSFT fixes that solves it for other type of workloads. If your 2012 R2 VM is fully up-to-date with MSFT updates, please create a support call. Post the support call ID here and let us know the follow-up after investigation with the engineers
Thanks
Mike

[Bennon Maina]

This might help for Hyper-V users:

- Depending on the VM workload, are you able to create a checkpoint/snapshot manually using Hyper-V Manager/Failover Cluster Manager/SCVMM Console?
If not, then the issue is simply snapshotting. I suggest shutdown the VM (when possible), check the disk. Are you able to edit it? Has the location of the disks recently changed?
Browse to the location of the virtual disk (don't be afraid), map it and then take a manual snapshot. This should work. It definitely has to do with the VHD location of the VM. It's happened to me twice while migrating a VM's storage location.

You should be able to take a snapshot of a VM whether running or not. Veeam should be able to backup a VM that is running or not.

Hope this helps