Discussions specific to the Microsoft Hyper-V hypervisor
bcrusa
Novice
Posts: 5
Liked: 1 time
Joined: Dec 10, 2017 2:15 pm
Full Name: bcrusa
Contact:

[MERGED] Veeam B&R 9.5 Update 4.a (VM failed backups on Hyper-V OS 2019) - The process cannot access the file

Post by bcrusa » Jul 05, 2019 7:51 am

Case #03651343

Dear support team and forum members!

We are using Veeam Backup and Replication Community edition, version 9.5 (update 4a). 1st of all let me say that the product is great and thumbs up for all the work you did there!

We are trying to find the root of problem with failed VM backups.
We have 3 virtual machines (2 linux machines and 1 Windows OS 2016) on Hyper-V host (OS 2019)
Backup runs ok for 2 days but after that only restarting the Hyper-V host helps that Veeam can create next backups.

Error message that Veeam B&R reports when backup fails:

Failed to create VM recovery checkpoint (mode: Crash consistent) Details: Job failed ('Checkpoint operation for 'LINUX' failed. (Virtual machine ID DD9D9847-7EFE-4195-852A-C34F71B15D5E) Checkpoint operation for 'LINUX' was cancelled. (Virtual machine ID DD9D9847-7EFE-4195-852A-C34F71B15D5E) 'LINUX' could not initiate a checkpoint operation: The process cannot access the file because it is being used by another process. (0x80070020). (Virtual machine ID DD9D9847-7EFE-4195-852A-C34F71B15D5E) 'LINUX
Retrying snapshot creation attempt (Failed to create production checkpoint.)
Task has been rescheduled
Queued for processing at 4. 07. 2019 22:43:03
Unable to allocate processing resources. Error: Job failed ('Checkpoint operation for 'LINUX' failed. (Virtual machine ID DD9D9847-7EFE-4195-852A-C34F71B15D5E) Checkpoint operation for 'LINUX' was cancelled. (Virtual machine ID DD9D9847-7EFE-4195-852A-C34F71B15D5E) 'LINUX' could not initiate a checkpoint operation: The process cannot access the file because it is being used by another process. (0x80070020). (Virtual machine ID DD9D9847-7EFE-4195-852A-C34F71B15D5E) 'LINUX' could not create aut


What we have tried already:
1. If we restart Hyper-V Host (OS2019) where Virtual Machines are located backup is running fine for 2 days but after that error repeats and only rebooting Hyper-V host helps.
2. Restore points on Backup Jobs are set to 3 days and we had tried changing from incremental to reverse incremental but error repeats in both cases.
3. When error repeats we cannot even make Active Full backups at that moment.
4. When error repeats not even Windows Server Backup can make the backup of a virtual machines.
5. Restarting all Veeam services on Host does not solve the problem, we need to restart Hyper-V Host (OS 2019) completely.
6. Veeam Backup and Replication software and Hyper-v role is installed on the same server. We have even tried with installing Veeam Backup and Replicaton community edition on a separated host and recreated backup jobs but the problem remains the same.
7. When backup fails, we also see that we are unable to edit any disks on virtual machines. They are visible as avhdx files instead of vhdx. Get-VmSnapshot does not return any results.
8. We had also posted question on Microsoft Technet Forum (https://social.technet.microsoft.com/Fo ... rverhyperv ) and still searching for solution.

The problem started to appear after we have moved this 3 Virtual Machines from one Hyper-V host to other.
We had moved virtual machines from 2012 R2 Hyper-v host to 2019 Hyper-v host.

Before migrating VM's from one to other host we had turned off virtual machines, so they were moved as offline.

After the live migration move completed, we had upgraded Hyper-V Configuration version on all the three VM's to version 9 and powered on VM's and everything seems to be OK except backups are failing after 2 days.

Hyper-V Host (OS 2019) was a fresh OS installation with only Hyper-V role and Veeam Backup and Replication software.
Server has latest windows patches.

Any ideas how to solve this problem?

wishr
Veeam Software
Posts: 965
Liked: 101 times
Joined: Aug 07, 2018 3:11 pm
Full Name: Fedor Maslov
Contact:

Re: Veeam B&R 9.5 Update 4.a (VM failed backups on Hyper-V OS 2019) - The process cannot access the file

Post by wishr » Jul 05, 2019 8:29 am

Hi Bostjan,

Could you please let us know if you able to create a production checkpoint manually?

P.S. Veeam Technical Support team members rarely look at this forum and we recommended contacting them directly in case of any technical issues. The forum is mostly managed by Veeam Product Management team.

Thanks

bcrusa
Novice
Posts: 5
Liked: 1 time
Joined: Dec 10, 2017 2:15 pm
Full Name: bcrusa
Contact:

Re: Veeam B&R 9.5 Update 4.a (VM failed backups on Hyper-V OS 2019) - The process cannot access the file

Post by bcrusa » Jul 05, 2019 8:43 am

Hi.
Thank you for quick reply.
I need to wait for problem to repeat. Yesterday I have rebooted server. Now probably the problem will repeat either today, or either tomorrow.

wishr
Veeam Software
Posts: 965
Liked: 101 times
Joined: Aug 07, 2018 3:11 pm
Full Name: Fedor Maslov
Contact:

Re: Veeam B&R 9.5 Update 4.a (VM failed backups on Hyper-V OS 2019) - The process cannot access the file

Post by wishr » Jul 05, 2019 9:04 am

Hello,

Not a problem. Please let us know what result you'll get.

Thanks

AlexandreD
Service Provider
Posts: 14
Liked: 1 time
Joined: Jan 22, 2019 4:21 pm
Full Name: ALEXANDRE D
Contact:

[MERGED] Re: Veeam B&R 9.5 Update 4.a (VM failed backups on Hyper-V OS 2019) - The process cannot access the file

Post by AlexandreD » Jul 05, 2019 9:16 am

Hi

I don't know if I have the same issue, but it's look like mine

In veeam, replication job failed with:
Failed to create VM recovery checkpoint (mode: Crash consistent) Details: Failed to call wmi method 'CreateSnapshot'. Wmi error: '32775' Failed to create VM recovery snapshot, VM ID '4c9d3f65-b731-487f-a2c6-4002e018103c'.
Retrying snapshot creation attempt (Failed to create production checkpoint.)
Task has been rescheduled
Queued for processing at 04/07/2019 20:50:46
Unable to allocate processing resources. Error: Failed to call wmi method 'CreateSnapshot'. Wmi error: '32775' Failed to create VM recovery snapshot, VM ID '4c9d3f65-b731-487f-a2c6-4002e018103c'.
This is my environnement:

- Hyper-V Windows Server 2019 (1809 version 17763.437)
- Veeam B&R update 4 9.5.4.2753
- Hyper-V Host are directly connect by Broadcom NetXtreme E-Series 10GB Base T ( For replication job every 15 minutes)

Sometimes backup and replication jobs fail and in Hyper-V manager, snapshot creation was stuck on 9%. During this problem i cannot:
- Launch New backup or replication job
- In hyper-v, cannot boot, reboot or modify VM.
- Cannot restart Hyper-v VMMS service

The only solution is to HARD REBOOT the Hyper-v.. After reboot, backup and replication work fine until new problem.

I do several update on my servers :
- update Windows Server 2019
- update Veeam B&R u4 in update A
- update NIC card, Bios and firmware (PowerEdge Dell R540)
- Changed production checkpoint to standard checkpoint

In veeam backup :
- I split VMs in several Jobs (By OS) customer had old OS like 2000, 2003, XP. Also Debian 9.
- For this VMs, in veeam job, I enable the option "Hyper-v Guest Tool quiescence and Take Crash consistent backup instead of suspending VM" for the old OS job and linux.

After modifications, i have no problem during 3 weeks until last night... Replication failed, snapshot creation stuck 9% and backup jobs failed too.


I found similar technet post, where i post my problem
https://social.technet.microsoft.com/Fo ... f=required
No solution found

Here, same blog post, he move to veeam 9.5 update 4 and have the problem.
http://www.checkyourlogs.net/?p=60293


I opened a case 03618396 in June 2019 when i had the problem for the second time, and support answered me :

"Hi Alexandre,

Veeam has no control over checkpoint or snapshot creation. We do send requests to hosts asking for snapshots to be created, but from there it's up to the host (and its storage if its using hardware VSS) to accomplish the task of snapshotting a VM so we can continue with our backup or replication of the VM.

I did find this thread of several other admins discussing a similar problem with various root causes and solutions, and it may have some suggestions that would be useful to you. I'm seeing problems as far ranging as NIC firmware being out of date to corrupted Hyper-v server instances that needed to be rebuilt.

https://social.technet.microsoft.com/Fo ... rverhyperv

Please let me know if you have any further questions."

wishr
Veeam Software
Posts: 965
Liked: 101 times
Joined: Aug 07, 2018 3:11 pm
Full Name: Fedor Maslov
Contact:

Re: Veeam B&R 9.5 Update 4.a (VM failed backups on Hyper-V OS 2019) - The process cannot access the file

Post by wishr » Jul 05, 2019 9:30 am

Hi Alexandre,

Have you had a chance to discuss this issue with Microsoft support?

Please note that I've merged your posts to an existing discussion - please take a look.

Thanks

AlexandreD
Service Provider
Posts: 14
Liked: 1 time
Joined: Jan 22, 2019 4:21 pm
Full Name: ALEXANDRE D
Contact:

Re: Failed to create VM recovery checkpoint

Post by AlexandreD » Jul 05, 2019 11:38 am

Hi

Unfortunately not. we have no silver or gold partner with microsoft.

I have a another customer with Hyper-V 2019 and veeam 9.5 u3, no backup problem.
i'm in discussion with veeam support. I open a new case.

wishr
Veeam Software
Posts: 965
Liked: 101 times
Joined: Aug 07, 2018 3:11 pm
Full Name: Fedor Maslov
Contact:

Re: Failed to create VM recovery checkpoint

Post by wishr » Jul 05, 2019 12:33 pm

Hi Alexandre,

The comments made by our support engineer are correct - we do not have control over Hyper-V snapshot creation behavior. Since the root cause lies on the Hyper-V end (you may confirm it by trying to create a production checkpoint manually - the procedure can be found above), it would be beneficial to open a case with Microsoft.

Thanks

mb1811
Influencer
Posts: 15
Liked: 2 times
Joined: Apr 29, 2014 7:08 am
Full Name: MB1811
Contact:

Re: Failed to create VM recovery checkpoint

Post by mb1811 » Jul 24, 2019 6:18 am

Hi!

Veeam Case # 03676525

We ran into this issue 5 days ago. The only way for us was to try to shutdown all VMs and then reboot the server.
Yesterday we ran into this 9% issue again.

We run a 2019 Datacenter on an brand new Fujitsu 2540M5 with 2x Intel 10G X710 DA2 NICs.
Server 2019 always said that the newest MS drivers for this cards were installed. After we disconnected that server from out WSUS - WindowsUpdate found some brand new Intel drivers.

We also set all Snapshots from productive to standard.

Fingers crossed!


Does anybody has a ticket opened @Microsoft? Seems to me that this issue is known since Server 2016 and not fixed yet?

As we - and many other - do not have silver or gold support @Microsoft - is there any change that Veeam can use it's connections? Or maybe keep track of that issue?


Thanks!
Matt

wishr
Veeam Software
Posts: 965
Liked: 101 times
Joined: Aug 07, 2018 3:11 pm
Full Name: Fedor Maslov
Contact:

Re: Failed to create VM recovery checkpoint

Post by wishr » Jul 24, 2019 9:56 am

Hi Matt,

Just to remind - we do not utilize legacy checkpoints at all when backing up VMs @ Hyper-V 2016 and later. Only production checkpoints are used.

Regarding our interaction with MS on this issue as well as others, please refer to that post by Gostev. Long story short, more cases we raise - more chances Microsoft will release a fix for any issue on their end. Keep in mind that the root causes could be different across even just cases mentioned in this particular thread, so it's definitely recommended to have a separate Microsoft case raised independently on each situation. While it may sound a bit frustrating - this is the best approach that can be taken currently to ensure you get a proper long-term solution.

Thanks

AlexandreD
Service Provider
Posts: 14
Liked: 1 time
Joined: Jan 22, 2019 4:21 pm
Full Name: ALEXANDRE D
Contact:

Re: Failed to create VM recovery checkpoint

Post by AlexandreD » Jul 29, 2019 6:54 am

Hi

I have open a case @microsoft ... i have no silver or gold support so i paid for one ticket (350€). For now, really disapointed, no constructive return. The MS Support engineer said at phone without any diagnostic "it's veeam". Great job, and i paid for this.

I have always a open case @veeam 03651597 , the support engineer did a really good job, he try to identify the problem with me.
it is undoubtedly a problem hyper-v but without the good microsoft support, the problem will not be solved.

I have to hard reboot the server every 2 weeks. For the moment I have not yet encountered any problem in VMs..
If I have one, I should restore the data before the date the problem occurs, my client could lose data because of that...
I have to manually check every night that snapshot creations are not blocked.

I hope that my microsoft ticket will have a good continuation

Alexandre

churchthedead
Novice
Posts: 4
Liked: never
Joined: Jul 30, 2019 3:37 pm
Full Name: Travis Riffe
Contact:

[MERGED] Server 2016 VM backup/replication failure: Element Not Found

Post by churchthedead » Jul 30, 2019 4:05 pm

Case #03689147

When trying to backup or replicate a specific VM I get the following error:

Code: Select all

Failed to create VM recovery checkpoint (mode: Veeam application-aware processing) Details: Job failed ('Checkpoint operation for 'VM' failed. (Virtual machine ID E397EC81-9B8C-435F-95C9-F9950638BF28) Checkpoint operation for 'VM' was cancelled. (Virtual machine ID E397EC81-9B8C-435F-95C9-F9950638BF28) 'VM' could not initiate a checkpoint operation: Element not found. (0x80070490). (Virtual machine ID E397EC81-9B8C-435F-95C9-F9950638BF28)'). Error code: '32789'. Failed to create VM recov
Retrying snapshot creation attempt (Failed to create production checkpoint.)
Task has been rescheduled
Queued for processing at 7/30/2019 11:46:59 AM
Unable to allocate processing resources. Error: Job failed ('Checkpoint operation for 'VM failed. (Virtual machine ID E397EC81-9B8C-435F-95C9-F9950638BF28) Checkpoint operation for 'VM' was cancelled. (Virtual machine ID E397EC81-9B8C-435F-95C9-F9950638BF28) 'VM' could not initiate a checkpoint operation: Element not found. (0x80070490). (Virtual machine ID E397EC81-9B8C-435F-95C9-F9950638BF28)'). Error code: '32789'. Failed to create VM recovery snapshot, VM ID 'e397ec81-9b8c-435f-95c9
I have two Hyper-V hosts, both Server 2016 Standard, with Server 2016 Standard VMs, on one host I can backup and replicate the VM with app aware processing just fine, the other gives me this error when trying to backup or replicate the VM.
I cannot create a manual production checkpoint, you can fairly reliably recreate the issue by creating a new Server 2016 VM, create a production checkpoint, then delete the checkpoint subtree, production checkpoints now fail.

On the Host where I'm getting the failure, I did a production checkpoint as a test, then deleted the checkpoint subtree (since I know Veeam requires production checkpoints for application aware processing), on the Host that's operating correctly, I did not do a test checkpoint/sub-tree delete.

I'm aware this is likely a MS issue, but just wanted to see if anyone may have any insight. I've tried exporting/importing the VM (with a new unique ID), restarting VSS and HVVSS, rebooting the VM, rebooting the Host etc.

wishr
Veeam Software
Posts: 965
Liked: 101 times
Joined: Aug 07, 2018 3:11 pm
Full Name: Fedor Maslov
Contact:

Re: Failed to create VM recovery checkpoint

Post by wishr » Jul 30, 2019 4:19 pm

Hi Travis,

Is there any specific reason for deleting a subtree since it sounds like a root cause. Of course, it should not cause the overall mechanism to fail :)

The general recommendation is to raise a support case with Microsoft. There is some info about this error available on the internet, but I cannot recommend any article in particular.

P.S. I merged your topic with an existing discussion around checkpoint creation issues in latest Hyper-V versions.

Thanks

churchthedead
Novice
Posts: 4
Liked: never
Joined: Jul 30, 2019 3:37 pm
Full Name: Travis Riffe
Contact:

Re: Failed to create VM recovery checkpoint

Post by churchthedead » Jul 30, 2019 5:46 pm

Honestly there isn't any particular reason other than I didn't need it. I created the checkpoint as a test and then deleted it because it was successful, and everything merged back down successfully as far as I know, I didn't get any errors or anything.

I have the question up on technet and I'll be raising a case with MS as soon as I get info on our support contract with them for our partner agreement. I'll update here as I go incase it helps someone else.

Of note - I can make checkpoints / backup / replicate when the vm is powered down, I can edit the disk, I have no orphan avhdx files, I can export the VM, and both the host and the vm are fully updated.

wishr
Veeam Software
Posts: 965
Liked: 101 times
Joined: Aug 07, 2018 3:11 pm
Full Name: Fedor Maslov
Contact:

Re: Failed to create VM recovery checkpoint

Post by wishr » Jul 31, 2019 9:00 am

Hi Travis,

Thanks for sharing the details.

Sounds like the root cause is related to the interaction between Hyper-V and guest OS of the VM, maybe to VSS components. Have you had a chance to take a look at the VSS event log?

Please keep us posted.

Regards,
Fedor

churchthedead
Novice
Posts: 4
Liked: never
Joined: Jul 30, 2019 3:37 pm
Full Name: Travis Riffe
Contact:

Re: Failed to create VM recovery checkpoint

Post by churchthedead » Jul 31, 2019 4:14 pm

Nothing in VSS logs, VSS writers of host and guest report as stable and ok.

wishr
Veeam Software
Posts: 965
Liked: 101 times
Joined: Aug 07, 2018 3:11 pm
Full Name: Fedor Maslov
Contact:

Re: Failed to create VM recovery checkpoint

Post by wishr » Aug 01, 2019 11:19 am

I'm out of ideas :(

Let's see what Microsoft will respond. This is your technet thread, right?

churchthedead
Novice
Posts: 4
Liked: never
Joined: Jul 30, 2019 3:37 pm
Full Name: Travis Riffe
Contact:

Re: Failed to create VM recovery checkpoint

Post by churchthedead » Aug 01, 2019 4:16 pm


vertices
Enthusiast
Posts: 35
Liked: 3 times
Joined: Oct 05, 2010 3:27 pm
Full Name: Rob Miller
Contact:

Re: Failed to create VM recovery checkpoint

Post by vertices » Aug 13, 2019 5:13 pm

I'm in the middle of a VMware to Hyper-V 2016 migration. All of my 2016 or 2019 VMs back up just fine. I migrated a SQL Server running Windows Server 2012 R2 and hit this bug. Veeam can't back it up. Hyper-V can't make a Production checkpoint manually. I can make a Production checkpoint if the VM is off, and I can make a Standard checkpoint if it's on or off. Integration services are up to date and functioning fine. So I can't back it up with Veeam unless I power it down I suppose, will check tonight.

I'm considering open a MS case for this. Is that just pointless? It sounds like this is a known issue with no known resolution? :(

vertices
Enthusiast
Posts: 35
Liked: 3 times
Joined: Oct 05, 2010 3:27 pm
Full Name: Rob Miller
Contact:

Re: Failed to create VM recovery checkpoint

Post by vertices » Aug 15, 2019 6:25 pm 1 person likes this post

Well, I resolved my issue. It was due to the Message Queuing Service on the guest. Uninstalling and reinstalling it resolved my inability to backup the 2012 R2 VM. The VSS writer for that service would fail upon trying to back it up. Bouncing services around eventually would get it to work. Uninstalling and reinstalling seems to have fixed it for now.

Prior to that, I had also recreated the vhdx's as fixed as I saw that had fixed it for someone else. So between both of those, issue fixed.

Post Reply

Who is online

Users browsing this forum: No registered users and 5 guests