Failed to create VM recovery checkpoint

[wishr]

Hi Fernando,

Correct, with the "Take crash-consistent backup instead of suspending VM" option the application running inside will not be prepared for backup and that might cause issues with restores, so you may want to leave this setting unchecked. However, since it's pfSense, you may want to take a look at this thread mentioning a few possible issues with restores that are related to the initial pfSense deployment method. Additionally, I'd recommend checking if BIS driver package is installed and up to date.

Please, also note that AAIP is not supposed to work with FreeBSD since it's not supported by Microsoft VSS, so Hyper-V Quiescence should be the way to go in this case.

Thanks!

[Rabbit]

Hi experts, I have run the back up for two weeks and everything is fine for the first week after some setting has been updated. But starting from this week, the backup for one of the virtual machines failed. Below is the errror:

Code: Select all

3/2/2020 7:53:02 PM :: Failed to create VM recovery checkpoint (mode: Veeam application-aware processing) Details: Job failed ('Checkpoint operation for 'XXX' failed. (Virtual machine ID xxxxxxxxx)
Production checkpoints cannot be created for 'xxxxxxxxx'. (Virtual machine ID xxxxxxxx)'). Error code: '32770'.
Failed to create VM recovery snapshot, VM ID 'xxxxxxxxxx'. 
3/2/2020 7:53:13 PM :: Transaction logs will not be truncated 
3/2/2020 7:53:18 PM :: Retrying snapshot creation attempt (Failed to create production checkpoint.) 
3/2/2020 7:53:18 PM :: Task has been rescheduled 
3/2/2020 7:53:18 PM :: Queued for processing at 3/2/2020 7:53:18 PM 
3/2/2020 7:53:23 PM :: Unable to allocate processing resources. Error: Job failed ('Checkpoint operation for 'XXX' failed. (Virtual machine ID xxxxxxxx.

What I have done differently this week is to attach two drives on the host. When adding the backup repositories, the attached drives were formatted to REFS 64KB. Please advise how to fix the error! Thank you!

[HannesK]

Hello,
I merged your question with the existing thread. Maybe you find something useful.

Please advise how to fix the error!

please contact support and post a case number for reference. Please understand that this is not a support forum

Best regards,
Hannes

[wishr]

Hello Julia,

If you are unable to create a production checkpoint manually and keep getting the same error, we would strongly recommend raising a ticket with Microsoft support.

Thanks

[fsr]

Hi,

I just had this issue with one of the remaining Windows Server 2008 R2 VMs, and i could circunvent this issue by shutting down the guest, then the snapshots and backups work. And when i turned the guest on again, the production snapshot did also work! No need to restart the Hyper-V Host. (just restarting the guest doesn't fix it, you need to take a snapshot with the guest off).

Regards

[Egor Yakovlev]

Interesting workaround, thanks for sharing!
/Cheers!

[kunniyoor]

Hi,

Production having Hyper-V 2012 and 2016 cluster

DR having Hyper-V 2012 and 2019 cluster.

Currently we are successfully running Backup jobs for VMs running on both hyper-v cluster (2012 & 2016) and replication jobs between Hyper-V 2012 at Production & DR hyper-v 2012 servers without Production checkpoints.

But when we run the replication job for the production hyper-v 2016 cluster VMs (with Production Checkpoint option unchecked) to replicate to DR (Windows 2019 hyper-v cluster) it throws below error

Code: Select all

Failed to process replication task Error: Failed to create VM (ID: cd5c08ac-4023-4598-900e-02dd81a0b091) snapshot. 
Job failed ('Checkpoint operation for 'SVR*****01' failed. (Virtual machine ID CD5C08AC-4023-4598-900E-02DD81A0B091)
Checkpoints have been disabled for 'SVR*****01'. (Virtual machine ID CD5C08AC-4023-4598-900E-02DD81A0B091).'). Error code: '32770'.

What could be the reason? :

Veeam support is working parallel with this issue and asking us to enable "Production Checkpoint" for 2016 Hyper-v Cluster VMs. We cant enable this option as we faced multiple vm crashes due to snapshot AVHD got utilized all allocated storage space.

Any suggestions?

[wishr]

Hi Krishnakuamr,

B&R only uses production checkpoints in Hyper-V 2016 and later, so the right way to get it resolved would be to investigate VM crashes you mentioned.

Could you please share your support case ID with us, as it is necessary when posting about any technical issue on the forums?

Thanks

[kunniyoor]

Veeam Support - Case # 04285270

But when we did the below work around it went all well.

. Initiated the replication by enabling replica seed as usual.
. Got said Error after a while. but the job created replica VM with out any virtual disk in the target cluster (windows 2019).
. Modified the replica VM setting and enabled "Production Checkpoint" option.
. Restored the virtual disk and attached to the replica VM.
. Rerun the replication and it got created the replica VM successfully.

We didn't enable production check for the source vm in any case.

Krish

[wishr]

Hi Krishnakuamr,

I have checked the case details and would like to say that regardless of the configured setting in Hyper-V, B&R does not attempt to create standard checkpoints in Hyper-V 2016 and later. So it would be great to try out the suggestions offered by our engineer and see what will be the outcome.

Thanks

[drumtek2000]

I rebooted but still failed.. I shut off application aware setting and backup completed.. If that helps with finding resolution..

[wishr]

It makes sense since because without AAIP enabled Microsoft VSS is not used, however, it is not a direct solution because the application consistency is not guaranteed in that case. We encourage everyone experiencing similar issues and being unable to create a production checkpoint manually from Hyper-V to raise a ticket with Microsoft support.

Thanks

[fsr]

I rebooted but still failed.. I shut off application aware setting and backup completed.. If that helps with finding resolution..

Try this:

• Shut down the affected VM.

• Take snapshots/backups while the VM is off. They should work, as nothing is running on the VM, so it's not neccesary (nor possible) to "freeze" running applications. But this also seems to reset any error condition about production snapshots that were preventing them from working.

• Turn on the affected VM. Now, production snapshots and backups should work again with the VM running. Al least, it worked that way for me on Windows Server 2016 Hyper-V hosts.

This only circunvents the issue, however. If you can contact Microsoft's support, you should do that.

Regards

[davidkillingsworth]

Veeam Support Case# 05021079

We are using Veeam Community Edition to backup 4 VM guests on Microsoft Windows 2019 Hyper-V server.
No Windows AD domain involved.
Just 4 VM guests (1 Windows 2016, 2 x Windows 2019, and 1 x Ubuntu 20.04 LTS).

Every month or so, the backup job gets stuck with the following error message on the same 2 VM guests:

Code: Select all

9/13/2021 9:00:50 PM :: Failed to create VM recovery checkpoint (mode: Veeam application-aware processing) Details: Failed to create VM (ID: 8c1fb6a9-4a76-465d-b099-46e3334424b1) recovery checkpoint. Failed to call wmi method 'CreateSnapshot'. Wmi error: '32775'

and

Code: Select all

9/13/2021 9:00:50 PM :: Failed to create VM recovery checkpoint (mode: Veeam application-aware processing) Details: Failed to create VM (ID: 62968b66-669a-477b-8be1-7f5430b4b122) recovery checkpoint. Failed to call wmi method 'CreateSnapshot'. Wmi error: '32775' 

When we go to the Hyper-V Manager the next morning, we find the 2 of the Windows 2019 servers are both stuck at creating their checkpoints at about 16%.

I am unable to shutdown the Veeam VM guest, the other VM guests, or the Hyper-V Manager.

The whole Hyper-V server becomes hung and we have to do a warm boot of the physical server.

This happens about every 30 days.

Here are the fail dates:
June 24, 2021
July 9, 2021
August 6, 2021
September 14, 2021

I have created a support case and re-posting here for visibility since we are using the community edition.

[wishr]

Hello Davidkillingsworth,

Please, give this suggestion a try and review this thread for additional information. Most likely it's an issue on the hypervisor layer.

Thanks

[petben]

I have a backup job that goes out to our cloud provider and is a mix or Windows, Linux servers. A few of these servers (3 x Linux, 2 x MS Remote Desktop servers) are erroring with:
Failed to create VM recovery checkpoint (mode: Veeam application-aware processing with failover)

I have followed the Veeam guide what to do when there's a mix of OS w.r.t Application Aware processing and Hyper V quiescence
https://helpcenter.veeam.com/archive/ba ... ce_hv.html

Basically on the Linux VMs (and the RDS servers as I think there is a bug with Veeam and RDS servers?) I have enabled both application-aware processing and Microsoft Hyper-V guest quiescence and set them to continue and Ignore application processing failures.

can anyone please advise what I can do as I am still getting the error.

[wishr]

Hi Pete,

I'm merging your post with an existing topic. Please, take a look. Most likely, it's a Hyper-V issue and you can easily confirm that by attempting to create a production checkpoint manually.

Thanks

[petben]

I can recreate this in Hyper-V, it hangs creating checkpoint at 9%.

This isn't a problem however when I am backing up the same VM's to out Local Veeam server, but this job is just the terminal servers (these are 2 of the servers with the error) and this job has both Application Aware Processing and Hyper-V quiescence disabled. I don't think I can disable Hyper-V quiescence on an individual VM basis?
any ideas?

Thanks

[wishr]

Hi Pete,

Thanks for answering.

In this case, we recommend creating a support case with Microsoft. There are also a few other suggestions mentioned in this thread.

Regarding the quiescence, yes, it works on the job level, but I would not recommend this workaround because not having an application-consistent backup may cause issues with application restores. The ideal solution would be to investigate the root cause on the Hyper-V end and eliminate it.

Thanks

[akraker]

We have had a few customers recently getting failed backups due to production checkpoints failing on the Hyper-V host. We can not even manually create a production checkpoint on the host. This common thing between all of them is Hyper-V 2019 hosts and the problem VM is a 2019 Domain Controller.
I can create standard checkpoints without issue but production checkpoints will fail. See the event log from the HV Host below.
I have checked all VSS writers and they look good. I can go production checkpoints on other VM's just fine.


The description for Event ID 3280 from source Microsoft-Windows-Hyper-V-Worker cannot be found. Either the component that raises this event is not installed on your local computer or the installation is corrupted. You can install or repair the component on the local computer.

If the event originated on another computer, the display information had to be saved with the event.

The following information was included with the event:

B1C50A40-17B9-49AF-8217-CAB3EA3CBB51
%%2147754996
0x800423F4

The locale specific resource for the desired message is not present

[wishr]

Hi Andrew,

No doubt, it's a Hyper-V/VSS/driver issue, so we would recommend opening a case with Microsoft support.

I'm merging your topic with an existing discussion about issues with production checkpoints. Also, there are several articles on the internet about the error 0x800423F4 - worth checking.

Thanks

[dasfliege]

Just stumbled across this discussion as i recently see a lot of our backups fail because of this error. We backup more then 1000 VMs and i have to restart VSS services for a few of them daily as they fail to backup properly. Restarting "COM+ Event System", "Volume Shadow Copy" and "Microsoft Software Shadow Copy Provider" services inside the gust VM solves the problem for a few days. Problems started after updating to Veeam v11a. We didn't performed any Hyper-V updates nor major updates inside the guest operating systems.

[wishr]

Hi Dasfliege,

Sad to hear you've also faced this issue.

It's easy to confirm whether the root cause lies on Veeam or Hyper-V layer by attempting to create a production checkpoint manually. Please, refer to the existing posts in this thread for more information. I'm sure the issue is not related to the upgrade - likely, it's just a coincidence.

Thanks

[mgaze]

Just stumbled across this discussion as i recently see a lot of our backups fail because of this error. We backup more then 1000 VMs and i have to restart VSS services for a few of them daily as they fail to backup properly. Restarting "COM+ Event System", "Volume Shadow Copy" and "Microsoft Software Shadow Copy Provider" services inside the gust VM solves the problem for a few days. Problems started after updating to Veeam v11a. We didn't performed any Hyper-V updates nor major updates inside the guest operating systems.

We too recently upgraded to 11a and I can confirm I am also seeing this issue on a couple of our VMs. This issue was not present before the upgrade to v11. Hosts are 2016.

Restarting the VM or the Hyper-V Volume Shadow Copy Requestor service clears the error, but as noted, only for a few days

[wishr]

Hello,

Please, refer to the comments above that explain how to confirm that the issue lies on the Hyper-V side. Once confirmed, we recommend opening a support ticket with Microsoft support.

Thanks

[HErceg]

I have similar issue if I try to use VeeamZIP.Checkpoints for all VM-s are disabled, on both WS2016 and WS2019 hosts.I tried to do it while VM in question was off, and to enable/disable quiescence, same result.After I enabled checkpoints everything was fine.
Regular backup works flawlessly, regardless of disabled checkpoints.

Funny thing is VeeamZIP for same VM worked about 14 months ago, so seems something changed in underlying VSS.

[ejenner]

I suspect this may be something to do with whether or not the VM has automatic updates enabled. I admit I have not done even half the research of the others in this thread. However I just think it makes logical sense. If the error goes after a reboot and comes back a while later.

When it is automatically updating it is probably making a snapshot of sorts with the updates in it. This interferes with the process of creating a VSS snapshot for backup.

It's just a hunch.

How many who have experienced the issue had their automatic updates enabled?