Host-based backup of Microsoft Hyper-V VMs.
AlexLeadingEdge
Veteran
Posts: 456
Liked: 57 times
Joined: Dec 14, 2015 9:42 pm
Contact:

Failed to create VM recovery checkpoint

Post by AlexLeadingEdge »

Hi guys!

We have just finished moving Hyper-V VMs from our old Windows Server 2012 R2 Hyper-V host to our new Windows Server 2016 Hyper-V host. All the VMs work exceptionally well but when we go to do a backup of the SBS 2011 server using B&R 9.5 it spits up this error after 11 seconds:
15/01/2017 11:47:48 p.m. :: Failed to create VM recovery checkpoint (mode: Veeam application-aware processing with failover) Details: Job failed ('Checkpoint operation for 'SBS-SERVER' failed. (Virtual machine ID B854900D-2DA6-44D0-9E3A-24D5D4700368)'). Error code: '32768'.
Failed to create VM recovery snapshot, VM ID 'b854900d-2da6-44d0-9e3a-24d5d4700368'.
15/01/2017 11:48:00 p.m. :: Retrying snapshot creation attempt (Failed to create production checkpoint.)
I will open a case when I can and update this ticket.

(For privacy reasons I have changed the name of the SBS server in the error message)
AlexLeadingEdge
Veteran
Posts: 456
Liked: 57 times
Joined: Dec 14, 2015 9:42 pm
Contact:

Re: Failed to create VM recovery checkpoint

Post by AlexLeadingEdge »

I see here that there is a known issue with Server 2016 and B&R 9.5 (different error message though!):

microsoft-hyper-v-f25/9-5-hyper-v-known ... 38927.html
Application-aware processing of Active Directory domain controllers running on guest OS other than Windows Server 2016 fails with the “Failed to create VM recovery checkpoint” error (32770). This is a Hyper-V 2016 issue that Microsoft is planning to address in the future update.
AlexLeadingEdge
Veteran
Posts: 456
Liked: 57 times
Joined: Dec 14, 2015 9:42 pm
Contact:

Re: Failed to create VM recovery checkpoint

Post by AlexLeadingEdge »

Case # 02039027
Mike Resseler
Product Manager
Posts: 8042
Liked: 1262 times
Joined: Feb 08, 2013 3:08 pm
Full Name: Mike Resseler
Location: Belgium
Contact:

Re: Failed to create VM recovery checkpoint

Post by Mike Resseler »

Alex,

For my understanding, your SBS server is running as a VM on a hyper-v 2016 host? In that SBS server is normally a domain controller so while the message might be different, I do think that you should look in that direction. But since I don't have any logs, please keep working with your support engineer to find the root cause

Thanks
Mike
AlexLeadingEdge
Veteran
Posts: 456
Liked: 57 times
Joined: Dec 14, 2015 9:42 pm
Contact:

Re: Failed to create VM recovery checkpoint

Post by AlexLeadingEdge »

VEEAM Support wrote:Could you please check if you can manually create a Checkpoint?

Also, verify if the VSS writers on the host are stable. You can run the command
Vssadmin list writers
I can manually create a checkpoint on the Hyper-V Host.

VSS writers are stable on both the backup server and the VM (the SBS Server).

The backup last night again spat up this same error.
AlexLeadingEdge
Veteran
Posts: 456
Liked: 57 times
Joined: Dec 14, 2015 9:42 pm
Contact:

Re: Failed to create VM recovery checkpoint

Post by AlexLeadingEdge »

Mike Resseler wrote:For my understanding, your SBS server is running as a VM on a hyper-v 2016 host?
Correct.
Mike Resseler wrote:In that SBS server is normally a domain controller so while the message might be different, I do think that you should look in that direction. But since I don't have any logs, please keep working with your support engineer to find the root cause.
Will do. The backup last night again spat up this same error.
Perry
Enthusiast
Posts: 43
Liked: never
Joined: Jan 02, 2017 10:19 pm
Full Name: Perry
Contact:

[MERGED] Domain controllers fail to backup

Post by Perry »

I have a job that backs up 6 VMs two of which are domain controllers. While the non domain controllers back up fine the domain controllers fail with this error:

Code: Select all

Failed to create VM recovery checkpoint (mode: Veeam application-aware processing) Details: Job failed (''). Error code: '32768'. Failed to create VM recovery snapshot, VM ID '50dfbcce-ca37-426c-a280-a83ac9d2e454'.
Retrying snapshot creation attempt (Failed to create production checkpoint.)
Task has been rescheduled
Queued for processing at 1/25/2017 1:41:29 AM
Unable to allocate processing resources. Error: Job failed (''). Error code: '32768'. Failed to create VM recovery snapshot, VM ID '50dfbcce-ca37-426c-a280-a83ac9d2e454'.
Any ideas?
Perry
Enthusiast
Posts: 43
Liked: never
Joined: Jan 02, 2017 10:19 pm
Full Name: Perry
Contact:

Re: Domain controllers fail to backup

Post by Perry »

I want to add that the backups for the DCs work fine if "Application-Aware Processing" is disabled.
veremin
Product Manager
Posts: 20261
Liked: 2249 times
Joined: Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin
Contact:

Re: Failed to create VM recovery checkpoint

Post by veremin »

Do you use Hyper-V 2016? Does DC run on OS other than Windows Server 2016? If so, be aware that you've come across known Hyper-V 2016 issue. Thanks.
Perry
Enthusiast
Posts: 43
Liked: never
Joined: Jan 02, 2017 10:19 pm
Full Name: Perry
Contact:

Re: Failed to create VM recovery checkpoint

Post by Perry »

Since application-aware processing has to be disabled for Domain Contoller VMs until the bug is fixed, should we enable guest quiescence? If so, how do we enable this at the individual VM level? Enabling quiescence seems to be global to the entire backup.
veremin
Product Manager
Posts: 20261
Liked: 2249 times
Joined: Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin
Contact:

Re: Failed to create VM recovery checkpoint

Post by veremin »

I don't think enabling Hyper-V quiescence is likely to help here, as it would leverage similar framework as AAIP does. Thanks.
Perry
Enthusiast
Posts: 43
Liked: never
Joined: Jan 02, 2017 10:19 pm
Full Name: Perry
Contact:

Re: Failed to create VM recovery checkpoint

Post by Perry »

So what's the recommended setting for backing up DC servers until the bug is fixed? Completely disable Application-Aware processing?
veremin
Product Manager
Posts: 20261
Liked: 2249 times
Joined: Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin
Contact:

Re: Failed to create VM recovery checkpoint

Post by veremin »

Yep, either disable AAIP (till the issue gets fixed by MS) or update DC to Windows Server 2016.
carussell
Influencer
Posts: 12
Liked: 2 times
Joined: Jan 17, 2017 3:36 pm
Full Name: Chris Russell
Contact:

Re: Failed to create VM recovery checkpoint

Post by carussell »

I am having the exact same issue but there is a twist in the information.

HV host updated from HV server 2012r2 to server 2016 datacenter. all the VM's that I manually MOVED to the updated host will complete a backup with a warning that they should be updated to hardware version 8. A newly created VM (windows 7 ent x32) will not backup and fails:

Failed to create VM recovery checkpoint (mode: Hyper-V child partition snapshot) Details: Job failed ('Checkpoint operation for 'Win 7 Ent x32' failed. (Virtual machine ID xxxxxxxx)'). Error code: '32768'. Failed to create VM recovery snapshot, VM ID 'xxxxxxxxx'.
Retrying snapshot creation attempt (Failed to create production checkpoint.)
Task has been rescheduled
Queued for processing at 2/1/2017 8:56:02 AM
Unable to allocate processing resources. Error: Job failed ('Checkpoint operation for 'Win 7 Ent x32' failed. (Virtual machine ID xxxxxxxx)'). Error code: '32768'. Failed to create VM recovery snapshot, VM ID 'xxxxxxxxx'.


Domain controllers are windows 2003.... but they are NOT VM's they are separate physical machines. So while this seems to be related it doesn't seem as though the known bug is this issue as that is about older OS DC's as Guests on Win2016 Hosts.
mvalpreda
Enthusiast
Posts: 74
Liked: 3 times
Joined: May 06, 2015 10:57 pm
Full Name: Mark Valpreda
Contact:

Re: Failed to create VM recovery checkpoint

Post by mvalpreda »

Did this ever get sorted on either the Microsoft or Veeam side? I just did a 2012 R2 to 2016 migration and the 2012 R2 DC is not happy.
Mike Resseler
Product Manager
Posts: 8042
Liked: 1262 times
Joined: Feb 08, 2013 3:08 pm
Full Name: Mike Resseler
Location: Belgium
Contact:

Re: Failed to create VM recovery checkpoint

Post by Mike Resseler »

Hi Mark,

No this issue still exists. As written above, you should disable AAIP for the DC. The bug is unfortunately still there and we haven't got any notification that MSFT is solving it
rsanders
Service Provider
Posts: 42
Liked: 10 times
Joined: Apr 20, 2016 7:32 pm
Full Name: Ryan Sanders
Contact:

Re: Failed to create VM recovery checkpoint

Post by rsanders »

Still having this issue as I just migrated an SBS server to a 2016 host. It won't be there long as it being decommissioned soon, but it would still be nice to have guest processing.
AlexLeadingEdge
Veteran
Posts: 456
Liked: 57 times
Joined: Dec 14, 2015 9:42 pm
Contact:

Re: Failed to create VM recovery checkpoint

Post by AlexLeadingEdge »

Nine months on I'm still having this issue :(
mazzu
Novice
Posts: 4
Liked: 1 time
Joined: Jul 28, 2014 6:37 am
Full Name: Emilio Mazzucconi
Contact:

Re: Failed to create VM recovery checkpoint

Post by mazzu »

Same problem here, same scenario, Hyper-V 2016 with a 2012 R2 Domain Controller VM.
Backups for the DC work fine if "Application-Aware Processing" is disabled.
sbaltic
Enthusiast
Posts: 31
Liked: 1 time
Joined: Nov 18, 2013 6:10 pm
Full Name: Sebastijan
Contact:

Re: Failed to create VM recovery checkpoint

Post by sbaltic »

Same problem with 2016 host and 2016 VM (not DC) ....

11.10.2017 20:01:26 :: Failed to create VM recovery checkpoint (mode: Veeam application-aware processing) Details: Job failed (''). Error code: '32772'.
Failed to create VM recovery snapshot, VM ID '6604883e-fc6e-4403-b72d-454d3741377b'.
IHT_Michael
Lurker
Posts: 1
Liked: never
Joined: Nov 21, 2017 10:36 pm
Full Name: Michael C Brown
Contact:

Re: Failed to create VM recovery checkpoint

Post by IHT_Michael »

v.Eremin wrote:Yep, either disable AAIP (till the issue gets fixed by MS) or update DC to Windows Server 2016.
How can we disable AAIP with your scheduled PowerShell script and VB&R Free? I'm assuming I'm seeing this same issue. I have Windows Server 2016 for the Hyper-V host with four virtuals running. The two that are not DCs back up fine. The two that are DCs have this Error 32768. But I'm doing this from scheduled PowerShell scripts.

Thanks,
Michael
foggy
Veeam Software
Posts: 21069
Liked: 2115 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Failed to create VM recovery checkpoint

Post by foggy »

Hi Michael, there's no AAIP in the free version.
cpfleger
Certified Trainer
Posts: 367
Liked: 42 times
Joined: Aug 31, 2012 7:30 am
Full Name: Claus Pfleger
Contact:

Re: Failed to create VM recovery checkpoint

Post by cpfleger »

Got the same problem after I v2ved a Win2k12r2 file server from vSphere 6.5 to Hyper-V 2016 (using a Veeam Agent for Windows 2.0 free backup Instant recovering to Hyper-V host and finalizing the Hyper-V vm), had both app-aware and Hyper-V quiescence enabled in the backup job.

Tried again after deinstalling VAW and VMware tools, but same result.

vssadmin list writers shows a bunch of vss writers failing due to timeouts (System Writer, ASR Writer, FSRM Writer, Shadow Copy Optimization Writer, Registry Writer, COM+ REGDB Writer, Dedup Writer, WMI Writer). Will check if I can start/fix the services behind.

Hints and tips welcome.

Regards!
cpfleger
Certified Trainer
Posts: 367
Liked: 42 times
Joined: Aug 31, 2012 7:30 am
Full Name: Claus Pfleger
Contact:

Re: Failed to create VM recovery checkpoint

Post by cpfleger »

System Writer => Service: Cryptographic Services
ASR Writer / Registry Writer / COM+ REGDB Writer / Shadow Copy Optimization Writer => Service: Volume Shadow Copy
FSRM writer => Service: File Server Resource Manager
Dedup Writer => Data Deduplication Volume Shadow Copy Service
WMI Writer => Windows Management Instrumentation

Restarted all those services, restarted vm: all services running
Tried again VBR backup job but failed again, vsslist writers again showed vss writers failed because of timeout. Hmm ...

Disabled application aware processing (without restarting VSS Writer services), now backup has run through.

Ideas still welcome!

Regards!
Mike Resseler
Product Manager
Posts: 8042
Liked: 1262 times
Joined: Feb 08, 2013 3:08 pm
Full Name: Mike Resseler
Location: Belgium
Contact:

Re: Failed to create VM recovery checkpoint

Post by Mike Resseler »

Claus,

Is that server up-to-date on hotfixes and updates?
AlexLeadingEdge
Veteran
Posts: 456
Liked: 57 times
Joined: Dec 14, 2015 9:42 pm
Contact:

Re: Failed to create VM recovery checkpoint

Post by AlexLeadingEdge »

cpfleger wrote:Disabled application aware processing (without restarting VSS Writer services), now backup has run through.
Hi cpfleger, my understanding is that you really don't want to do that, as this step quietens (quiesces) an active database system such as SQL or an Exchange Server, so that the backup doesn't end up corrupt! :shock:
ahickingbottom
Lurker
Posts: 1
Liked: never
Joined: Jan 17, 2018 2:35 pm
Full Name: Andrew Hickingbottom
Contact:

Re: Failed to create VM recovery checkpoint

Post by ahickingbottom »

Just checking to see the status on this issue with everyone else. I am still having the same issues. We are running Hyper-V on Server 2016 and the VM with the issue is a 2012 R2 running Microsoft SQL. We do not want to disable Application-Aware Processing just to get the job to complete because it runs SQL. Any information on a proper fix would be appreciated.
AlexLeadingEdge
Veteran
Posts: 456
Liked: 57 times
Joined: Dec 14, 2015 9:42 pm
Contact:

Re: Failed to create VM recovery checkpoint

Post by AlexLeadingEdge »

One year now and still no resolution :(
Mike Resseler
Product Manager
Posts: 8042
Liked: 1262 times
Joined: Feb 08, 2013 3:08 pm
Full Name: Mike Resseler
Location: Belgium
Contact:

Re: Failed to create VM recovery checkpoint

Post by Mike Resseler »

Andrew,

First: Welcome to the forums!

I think your issue might be a bit different. The problem is with DC's. There are MSFT fixes that solves it for other type of workloads. If your 2012 R2 VM is fully up-to-date with MSFT updates, please create a support call. Post the support call ID here and let us know the follow-up after investigation with the engineers
Thanks
Mike
Bennon Maina
Novice
Posts: 9
Liked: 7 times
Joined: Mar 07, 2018 3:23 pm
Full Name: Bennon Maina
Contact:

Re: Failed to create VM recovery checkpoint

Post by Bennon Maina » 1 person likes this post

This might help for Hyper-V users:

- Depending on the VM workload, are you able to create a checkpoint/snapshot manually using Hyper-V Manager/Failover Cluster Manager/SCVMM Console?
If not, then the issue is simply snapshotting. I suggest shutdown the VM (when possible), check the disk. Are you able to edit it? Has the location of the disks recently changed?
Browse to the location of the virtual disk (don't be afraid), map it and then take a manual snapshot. This should work. It definitely has to do with the VHD location of the VM. It's happened to me twice while migrating a VM's storage location.

You should be able to take a snapshot of a VM whether running or not. Veeam should be able to backup a VM that is running or not.

Hope this helps

:D
Post Reply

Who is online

Users browsing this forum: No registered users and 8 guests