Failed to create VM recovery checkpoint

[bcrusa]

Case #03651343

Dear support team and forum members!

We are using Veeam Backup and Replication Community edition, version 9.5 (update 4a). 1st of all let me say that the product is great and thumbs up for all the work you did there!

We are trying to find the root of problem with failed VM backups.
We have 3 virtual machines (2 linux machines and 1 Windows OS 2016) on Hyper-V host (OS 2019)
Backup runs ok for 2 days but after that only restarting the Hyper-V host helps that Veeam can create next backups.

Error message that Veeam B&R reports when backup fails:

Failed to create VM recovery checkpoint (mode: Crash consistent) Details: Job failed ('Checkpoint operation for 'LINUX' failed. (Virtual machine ID DD9D9847-7EFE-4195-852A-C34F71B15D5E) Checkpoint operation for 'LINUX' was cancelled. (Virtual machine ID DD9D9847-7EFE-4195-852A-C34F71B15D5E) 'LINUX' could not initiate a checkpoint operation: The process cannot access the file because it is being used by another process. (0x80070020). (Virtual machine ID DD9D9847-7EFE-4195-852A-C34F71B15D5E) 'LINUX
Retrying snapshot creation attempt (Failed to create production checkpoint.)
Task has been rescheduled
Queued for processing at 4. 07. 2019 22:43:03
Unable to allocate processing resources. Error: Job failed ('Checkpoint operation for 'LINUX' failed. (Virtual machine ID DD9D9847-7EFE-4195-852A-C34F71B15D5E) Checkpoint operation for 'LINUX' was cancelled. (Virtual machine ID DD9D9847-7EFE-4195-852A-C34F71B15D5E) 'LINUX' could not initiate a checkpoint operation: The process cannot access the file because it is being used by another process. (0x80070020). (Virtual machine ID DD9D9847-7EFE-4195-852A-C34F71B15D5E) 'LINUX' could not create aut


What we have tried already:
1. If we restart Hyper-V Host (OS2019) where Virtual Machines are located backup is running fine for 2 days but after that error repeats and only rebooting Hyper-V host helps.
2. Restore points on Backup Jobs are set to 3 days and we had tried changing from incremental to reverse incremental but error repeats in both cases.
3. When error repeats we cannot even make Active Full backups at that moment.
4. When error repeats not even Windows Server Backup can make the backup of a virtual machines.
5. Restarting all Veeam services on Host does not solve the problem, we need to restart Hyper-V Host (OS 2019) completely.
6. Veeam Backup and Replication software and Hyper-v role is installed on the same server. We have even tried with installing Veeam Backup and Replicaton community edition on a separated host and recreated backup jobs but the problem remains the same.
7. When backup fails, we also see that we are unable to edit any disks on virtual machines. They are visible as avhdx files instead of vhdx. Get-VmSnapshot does not return any results.
8. We had also posted question on Microsoft Technet Forum (https://social.technet.microsoft.com/Fo ... rverhyperv ) and still searching for solution.

The problem started to appear after we have moved this 3 Virtual Machines from one Hyper-V host to other.
We had moved virtual machines from 2012 R2 Hyper-v host to 2019 Hyper-v host.

Before migrating VM's from one to other host we had turned off virtual machines, so they were moved as offline.

After the live migration move completed, we had upgraded Hyper-V Configuration version on all the three VM's to version 9 and powered on VM's and everything seems to be OK except backups are failing after 2 days.

Hyper-V Host (OS 2019) was a fresh OS installation with only Hyper-V role and Veeam Backup and Replication software.
Server has latest windows patches.

Any ideas how to solve this problem?

[wishr]

Hi Bostjan,

Could you please let us know if you able to create a production checkpoint manually?

P.S. Veeam Technical Support team members rarely look at this forum and we recommended contacting them directly in case of any technical issues. The forum is mostly managed by Veeam Product Management team.

Thanks

[bcrusa]

Hi.
Thank you for quick reply.
I need to wait for problem to repeat. Yesterday I have rebooted server. Now probably the problem will repeat either today, or either tomorrow.

[wishr]

Hello,

Not a problem. Please let us know what result you'll get.

Thanks

[AlexandreD]

Hi

I don't know if I have the same issue, but it's look like mine

In veeam, replication job failed with:
Failed to create VM recovery checkpoint (mode: Crash consistent) Details: Failed to call wmi method 'CreateSnapshot'. Wmi error: '32775' Failed to create VM recovery snapshot, VM ID '4c9d3f65-b731-487f-a2c6-4002e018103c'.
Retrying snapshot creation attempt (Failed to create production checkpoint.)
Task has been rescheduled
Queued for processing at 04/07/2019 20:50:46
Unable to allocate processing resources. Error: Failed to call wmi method 'CreateSnapshot'. Wmi error: '32775' Failed to create VM recovery snapshot, VM ID '4c9d3f65-b731-487f-a2c6-4002e018103c'.
This is my environnement:

- Hyper-V Windows Server 2019 (1809 version 17763.437)
- Veeam B&R update 4 9.5.4.2753
- Hyper-V Host are directly connect by Broadcom NetXtreme E-Series 10GB Base T ( For replication job every 15 minutes)

Sometimes backup and replication jobs fail and in Hyper-V manager, snapshot creation was stuck on 9%. During this problem i cannot:
- Launch New backup or replication job
- In hyper-v, cannot boot, reboot or modify VM.
- Cannot restart Hyper-v VMMS service

The only solution is to HARD REBOOT the Hyper-v.. After reboot, backup and replication work fine until new problem.

I do several update on my servers :
- update Windows Server 2019
- update Veeam B&R u4 in update A
- update NIC card, Bios and firmware (PowerEdge Dell R540)
- Changed production checkpoint to standard checkpoint

In veeam backup :
- I split VMs in several Jobs (By OS) customer had old OS like 2000, 2003, XP. Also Debian 9.
- For this VMs, in veeam job, I enable the option "Hyper-v Guest Tool quiescence and Take Crash consistent backup instead of suspending VM" for the old OS job and linux.

After modifications, i have no problem during 3 weeks until last night... Replication failed, snapshot creation stuck 9% and backup jobs failed too.


I found similar technet post, where i post my problem
https://social.technet.microsoft.com/Fo ... f=required
No solution found

Here, same blog post, he move to veeam 9.5 update 4 and have the problem.
http://www.checkyourlogs.net/?p=60293


I opened a case 03618396 in June 2019 when i had the problem for the second time, and support answered me :

"Hi Alexandre,

Veeam has no control over checkpoint or snapshot creation. We do send requests to hosts asking for snapshots to be created, but from there it's up to the host (and its storage if its using hardware VSS) to accomplish the task of snapshotting a VM so we can continue with our backup or replication of the VM.

I did find this thread of several other admins discussing a similar problem with various root causes and solutions, and it may have some suggestions that would be useful to you. I'm seeing problems as far ranging as NIC firmware being out of date to corrupted Hyper-v server instances that needed to be rebuilt.

https://social.technet.microsoft.com/Fo ... rverhyperv

Please let me know if you have any further questions."

[wishr]

Hi Alexandre,

Have you had a chance to discuss this issue with Microsoft support?

Please note that I've merged your posts to an existing discussion - please take a look.

Thanks

[AlexandreD]

Hi

Unfortunately not. we have no silver or gold partner with microsoft.

I have a another customer with Hyper-V 2019 and veeam 9.5 u3, no backup problem.
i'm in discussion with veeam support. I open a new case.

[wishr]

Hi Alexandre,

The comments made by our support engineer are correct - we do not have control over Hyper-V snapshot creation behavior. Since the root cause lies on the Hyper-V end (you may confirm it by trying to create a production checkpoint manually - the procedure can be found above), it would be beneficial to open a case with Microsoft.

Thanks

[mb1811]

Hi!

Veeam Case # 03676525

We ran into this issue 5 days ago. The only way for us was to try to shutdown all VMs and then reboot the server.
Yesterday we ran into this 9% issue again.

We run a 2019 Datacenter on an brand new Fujitsu 2540M5 with 2x Intel 10G X710 DA2 NICs.
Server 2019 always said that the newest MS drivers for this cards were installed. After we disconnected that server from out WSUS - WindowsUpdate found some brand new Intel drivers.

We also set all Snapshots from productive to standard.

Fingers crossed!


Does anybody has a ticket opened @Microsoft? Seems to me that this issue is known since Server 2016 and not fixed yet?

As we - and many other - do not have silver or gold support @Microsoft - is there any change that Veeam can use it's connections? Or maybe keep track of that issue?


Thanks!
Matt

[wishr]

Hi Matt,

Just to remind - we do not utilize legacy checkpoints at all when backing up VMs @ Hyper-V 2016 and later. Only production checkpoints are used.

Regarding our interaction with MS on this issue as well as others, please refer to that post by Gostev. Long story short, more cases we raise - more chances Microsoft will release a fix for any issue on their end. Keep in mind that the root causes could be different across even just cases mentioned in this particular thread, so it's definitely recommended to have a separate Microsoft case raised independently on each situation. While it may sound a bit frustrating - this is the best approach that can be taken currently to ensure you get a proper long-term solution.

Thanks

[AlexandreD]

Hi

I have open a case @microsoft ... i have no silver or gold support so i paid for one ticket (350€). For now, really disapointed, no constructive return. The MS Support engineer said at phone without any diagnostic "it's veeam". Great job, and i paid for this.

I have always a open case @veeam 03651597 , the support engineer did a really good job, he try to identify the problem with me.
it is undoubtedly a problem hyper-v but without the good microsoft support, the problem will not be solved.

I have to hard reboot the server every 2 weeks. For the moment I have not yet encountered any problem in VMs..
If I have one, I should restore the data before the date the problem occurs, my client could lose data because of that...
I have to manually check every night that snapshot creations are not blocked.

I hope that my microsoft ticket will have a good continuation

Alexandre

[churchthedead]

Case #03689147

When trying to backup or replicate a specific VM I get the following error:

Code: Select all

Failed to create VM recovery checkpoint (mode: Veeam application-aware processing) Details: Job failed ('Checkpoint operation for 'VM' failed. (Virtual machine ID E397EC81-9B8C-435F-95C9-F9950638BF28) Checkpoint operation for 'VM' was cancelled. (Virtual machine ID E397EC81-9B8C-435F-95C9-F9950638BF28) 'VM' could not initiate a checkpoint operation: Element not found. (0x80070490). (Virtual machine ID E397EC81-9B8C-435F-95C9-F9950638BF28)'). Error code: '32789'. Failed to create VM recov
Retrying snapshot creation attempt (Failed to create production checkpoint.)
Task has been rescheduled
Queued for processing at 7/30/2019 11:46:59 AM
Unable to allocate processing resources. Error: Job failed ('Checkpoint operation for 'VM failed. (Virtual machine ID E397EC81-9B8C-435F-95C9-F9950638BF28) Checkpoint operation for 'VM' was cancelled. (Virtual machine ID E397EC81-9B8C-435F-95C9-F9950638BF28) 'VM' could not initiate a checkpoint operation: Element not found. (0x80070490). (Virtual machine ID E397EC81-9B8C-435F-95C9-F9950638BF28)'). Error code: '32789'. Failed to create VM recovery snapshot, VM ID 'e397ec81-9b8c-435f-95c9

I have two Hyper-V hosts, both Server 2016 Standard, with Server 2016 Standard VMs, on one host I can backup and replicate the VM with app aware processing just fine, the other gives me this error when trying to backup or replicate the VM.
I cannot create a manual production checkpoint, you can fairly reliably recreate the issue by creating a new Server 2016 VM, create a production checkpoint, then delete the checkpoint subtree, production checkpoints now fail.

On the Host where I'm getting the failure, I did a production checkpoint as a test, then deleted the checkpoint subtree (since I know Veeam requires production checkpoints for application aware processing), on the Host that's operating correctly, I did not do a test checkpoint/sub-tree delete.

I'm aware this is likely a MS issue, but just wanted to see if anyone may have any insight. I've tried exporting/importing the VM (with a new unique ID), restarting VSS and HVVSS, rebooting the VM, rebooting the Host etc.

[wishr]

Hi Travis,

Is there any specific reason for deleting a subtree since it sounds like a root cause. Of course, it should not cause the overall mechanism to fail

The general recommendation is to raise a support case with Microsoft. There is some info about this error available on the internet, but I cannot recommend any article in particular.

P.S. I merged your topic with an existing discussion around checkpoint creation issues in latest Hyper-V versions.

Thanks

[churchthedead]

Honestly there isn't any particular reason other than I didn't need it. I created the checkpoint as a test and then deleted it because it was successful, and everything merged back down successfully as far as I know, I didn't get any errors or anything.

I have the question up on technet and I'll be raising a case with MS as soon as I get info on our support contract with them for our partner agreement. I'll update here as I go incase it helps someone else.

Of note - I can make checkpoints / backup / replicate when the vm is powered down, I can edit the disk, I have no orphan avhdx files, I can export the VM, and both the host and the vm are fully updated.

[wishr]

Hi Travis,

Thanks for sharing the details.

Sounds like the root cause is related to the interaction between Hyper-V and guest OS of the VM, maybe to VSS components. Have you had a chance to take a look at the VSS event log?

Please keep us posted.

Regards,
Fedor

[churchthedead]

Nothing in VSS logs, VSS writers of host and guest report as stable and ok.

[wishr]

I'm out of ideas

Let's see what Microsoft will respond. This is your technet thread, right?

[churchthedead]

https://social.technet.microsoft.com/Fo ... rverhyperv is my Technet thread

[vertices]

I'm in the middle of a VMware to Hyper-V 2016 migration. All of my 2016 or 2019 VMs back up just fine. I migrated a SQL Server running Windows Server 2012 R2 and hit this bug. Veeam can't back it up. Hyper-V can't make a Production checkpoint manually. I can make a Production checkpoint if the VM is off, and I can make a Standard checkpoint if it's on or off. Integration services are up to date and functioning fine. So I can't back it up with Veeam unless I power it down I suppose, will check tonight.

I'm considering open a MS case for this. Is that just pointless? It sounds like this is a known issue with no known resolution?

[vertices]

Well, I resolved my issue. It was due to the Message Queuing Service on the guest. Uninstalling and reinstalling it resolved my inability to backup the 2012 R2 VM. The VSS writer for that service would fail upon trying to back it up. Bouncing services around eventually would get it to work. Uninstalling and reinstalling seems to have fixed it for now.

Prior to that, I had also recreated the vhdx's as fixed as I saw that had fixed it for someone else. So between both of those, issue fixed.

[bcrusa]

I would just like to share my experience that issue was resolved with updating NIC drivers

[saban]

hi
i have a stand alone hyper v server 2016
with veeam installed on it

it was running ok for a long time. but i have backup issue for a while
and it probably started after a windows updates

i started getting issues with checkpoint creation errors

i get this kind of errors on all of my virtual machines

"
Failed to create VM recovery checkpoint (mode: Veeam application-aware processing with failover) Details: Job failed ('Checkpoint operation for '*' failed. (Virtual machine ID *) Production checkpoints cannot be created for 'Terminal'. (Virtual machine ID *)'). Error code: '32770'. Failed to create VM recovery snapshot, VM ID '*'. "

if i disable application aware backup or select to ignore vss errors
than backups are working. but i rather it will work with vss

i read here in the forums about sevral issues with microsoft bugs

i tried updating the server with all cumulative updates
and also update veeam to the latest availvble update (4b)
but the issue still pressist

wanted to know is there any workaround ?
or is it a confirmed microsoft bug and veeam can't fix it ?

[wishr]

Hello Saban,

Looks like a Hyper-V issue. Could you please confirm it by attempting to create a production checkpoint manually?

I'm merging your topic with the related thread - please take a look.

Thanks

[saban]

hi wishr
if i disable "create standard checkpoint if it's not possbile to create...."
than yes production checkpoint fail

it it's enabled it create a standart checkpoint

is there any fix ?
do you know which update cause that ? maybe it can be removed

[wishr]

Hello Saban,

In this case, we strongly suggest raising a support case with Microsoft to get it sorted since the root cause lies on the Hyper-V end. There are also several suggestions above in the current thread - might be worth checking.

Thanks

[JRBeaver]

I have 3 instances of Windows 2012 standard Domain Controller VMs on Hyper-V 2016 or 2019 hosts where the checkpoint fails.

The creation of Hyper-V checkpoints on the host also fails when "Production" checkpoints are enabled in the VM settings. If I change this to "Standard" checkpoints I can create Hyper-V checkpoints, however the Veeam backups still fail.

In all cases the issue seems to be that the VSS Writer NTDS fails
Restarting the COM+ Event Sytem service (and the related services) and the Volume Shadow Copy service then allows the Veeam backup to complete without error
The strange point here is that after restarting these services and rerunning vssadmin list writers, the NTDS service no longer shows in the list of Writers at all. So I am not even sure if this is then a consistent VSS aware backup of AD.

After the VM is next rebooted the backups fail again with the NTDS writer showing as stable before the backup and failed after the backup

There are no issues with Windows 2012 R2 on 2016 or 2019 hosts

I can and do allow the backups to run using the Ignore failures option in the VSS section of the Application Aware Processing Options, which of course generates warnings for the backup job

The above is really only background and I have read quite a lot through these and other forums to see that this is an ongoing issue that is unlikely to be resolved, although I do live in hope that someone may have a solution. My real concern is that I want to ensure that these domain controller backups are consistent and able to be restored in the event of any issues. I would also like to eliminate warnings as our management solution generates tickets for every failure or warning.

As I am sure there must be other users backing up 2012 standard domain controllers on 2016 or 2019 hosts, does anyone have any suggestions on resolving the errors, and if not have you verified that your backups are indeed good (when ignoring VSS errors). Finally if all else fails have you been able to configure the backup so as not to produce warnings. And yes, I have considered upgrading to R2 and this may be the only solution.

[wishr]

Hi John,

I'm merging your topic with the existing one, please take a look.

I should add that AAIP is strongly recommended when backing up DCs to ensure AD DB will not become corrupted, so investigating that behavior with Microsoft support is strongly recommended (ofc, if the manual creation of production checkpoint fails also). On the other hand, we've seen cases when users were able to restore DCs from backups with no AAIP - while this is not the recommended approach, you may use SureBackup capabilities to make sure your workloads are recoverable, as a temporary solution.

Thanks

[mapate]

I have a v17 Ubuntu VM hosted in a 3-node Hyper-V Server 2019 cluster. VBR is successfully backing up several Windows VM in the cluster but am unable to successfully backup the Ubuntu VM. The error in the job is:

Failed to create VM recovery checkpoint (mode: Veeam application-aware
processing) Details: Job failed ('Checkpoint operation for '00A-Web04 - 10.0.0.58' failed. (Virtual machine ID
45A835D4-320C-4177-9D34-90B11600C1BE) Production checkpoints cannot be created for '00A-Web04 - 10.0.0.58'. (Virtual machine ID 45A835D4-320C-4177-9D34-90B11600C1BE)'). Error code: '32770'

On the Hyper-V host, I ran "get-vmintegrationservice -VMName (vmhostname)'
and see that VSS is installed and showing as OK:

VMName Name Enabled PrimaryStatusDescription SecondaryStatusDescription
------ ---- ------- ------------------------ --------------------------
00A-Web04 - 10.0.0.58 Guest Service Interface True OK
00A-Web04 - 10.0.0.58 Heartbeat True OK
00A-Web04 - 10.0.0.58 Key-Value Pair Exchange True No Contact
00A-Web04 - 10.0.0.58 Shutdown True OK
00A-Web04 - 10.0.0.58 Time Synchronization True OK
00A-Web04 - 10.0.0.58 VSS True OK The protocol version of the component...

I'm at a loss to determine how to resolve this issue. Has anyone experienced this problem and resolved it?

Case ID: 03930009

[Egor Yakovlev]

Hi Mike.
I have merged your post into existing thread. Please check it from the start, as Checkpoints Error 32770 discussed here from various angles.
Also, try disabling Guest Processing for that VM and see if backup works.
/Cheers!

[fsr]

I just found this problem with a freebsd (pfSense) VM on a Windows Server 2016 HOST. Other VMs work fine. Checkpoint also fails in Hyper-V manager if i force it to take a production checkpoint (uncheck "create standard checkpoint if it's not possible..."). Standard checkpoints work fine, but Veeam doesn't use them when the OS supports production checkpoints (which makes sense, as "production" would be the way to go for backups).

However, it seems that i found a work-around: on Veeam, you go to the job properties, then [storage] [advanced] [hyper-v], and select [enable hyper-v quiescence] and also [take crash consistent backup instead of suspending VM]. With that options, i can see that a temporary checkpoint is created during backup. The job log says "Creating VM recovery checkpoint (mode: Hyper-V child partition snapshot)". I suppose that that's as reliable as any standard checkpoint?