Host-based backup of Microsoft Hyper-V VMs.
Post Reply
rogerdu
Expert
Posts: 148
Liked: 11 times
Joined: Aug 20, 2013 1:16 pm
Full Name: Roger Dufour
Contact:

Re: 9.5 Hyper-V 2016 Known Issues

Post by rogerdu »

Annnnddd... Dynamic Disks... its all I'm gonna say. They're not supported for Production Snapshots...

One fixed, the customer wants to leave the other, so no Application Aware backups for you!
mb1811
Influencer
Posts: 17
Liked: 2 times
Joined: Apr 29, 2014 7:08 am
Full Name: MB1811
Contact:

Re: 9.5 Hyper-V 2016 Known Issues

Post by mb1811 » 1 person likes this post

Veeam Case # 03676525

This case ist related to Server 2019 but it hits since Server 2016.

I call it the 9% Issue:
While perfoming a Snapshot via ANY backup tool, VM's get stuck in a state: "Creating Snapshot 9%"

https://social.technet.microsoft.com/Fo ... rum=ws2016

microsoft-hyper-v-f25/failed-to-create- ... 98-30.html

Hardly shutdown the VM's and reboot the Hyper-V host...
MDS_UK
Lurker
Posts: 1
Liked: never
Joined: Aug 06, 2019 8:25 am
Full Name: Matt Smith
Contact:

Re: 9.5 Hyper-V 2016 Known Issues

Post by MDS_UK »

ChristineAlexa
Enthusiast
Posts: 44
Liked: 6 times
Joined: Aug 26, 2019 7:04 am
Full Name: Christine Boersen
Contact:

Re: 9.5 Hyper-V 2016 Known Issues

Post by ChristineAlexa » 1 person likes this post

Another fix I didn't see in here, and will add to the list to help save someone else a few hours of hunting

- Symptom - Errors Similar to the below, where %1 is the VM name and %2 is the Virtual Machine ID)

'Checkpoint operation for %1 failed. (Virtual machine %2) Checkpoint operation for '%1' was cancelled. (Virtual machine ID %2) '%1' could not initiate a checkpoint operation: Element not found. (0x80070490). Error code: '32789'. Failed


If you have MPIO enabled on the VM,
- ensure you didn't accidently check "Add Support for SAS Devices".
- You may see errors in event log for volumes that don't exist not mounting.

To uncheck that box (since you can't in the interface), (You most likely will need to reboot afterwards, even if not prompted)
- Open a Powershell session in the VM
- Type the following command: Disable-MSDSMAutomaticClaim -BusType SAS

Once you have reboooted, follow the normal settings for Production Checkpoints, and it should work finally.
Quibus
Novice
Posts: 4
Liked: never
Joined: May 09, 2019 3:06 pm
Full Name: Niclas Mann
Contact:

Re: 9.5 Hyper-V 2016 Known Issues

Post by Quibus »

Does anyone know a smooth script that can merge snapshots together after an incident like this?
I have a VM that has 213 .avhdx files ... sigh...
nmdange
Veteran
Posts: 527
Liked: 142 times
Joined: Aug 20, 2015 9:30 pm
Contact:

Re: 9.5 Hyper-V 2016 Known Issues

Post by nmdange » 1 person likes this post

I had to modify the script to do what I needed, but this should be a good starting point
https://gallery.technet.microsoft.com/s ... e-72a96c17
Quibus
Novice
Posts: 4
Liked: never
Joined: May 09, 2019 3:06 pm
Full Name: Niclas Mann
Contact:

Re: 9.5 Hyper-V 2016 Known Issues

Post by Quibus »

Thank you nmdange!
This was what is was looking for :)
bcrusa
Novice
Posts: 7
Liked: 2 times
Joined: Dec 10, 2017 2:15 pm
Full Name: bcrusa
Contact:

Re: 9.5 Hyper-V 2016 Known Issues

Post by bcrusa »

I would like to share my experience that I had solved the issue with updating NIC drivers.
fsr
Enthusiast
Posts: 30
Liked: 1 time
Joined: Mar 27, 2019 5:28 pm
Full Name: Fernando Rapetti
Contact:

Re: 9.5 Hyper-V 2016 Known Issues

Post by fsr »

So, still no patch from Microsoft for the "Checkpoint operation failed" issue? The only option continues to be restarting the two services every time the VM starts?
Wilian
Lurker
Posts: 1
Liked: never
Joined: Aug 24, 2020 2:41 pm
Full Name: Wilian Camargo
Contact:

Re: 9.5 Hyper-V 2016 Known Issues

Post by Wilian »

Hi All,

Does anyone know how to fix that?
I`m facing exactly the same problem with Windows Server 2016?
Vitaliy S.
VP, Product Management
Posts: 27055
Liked: 2710 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: 9.5 Hyper-V 2016 Known Issues

Post by Vitaliy S. »

Wilian, that's a pretty long thread with all sorts of issues, so can you please clarify what problem do you have? Thanks!
davidkillingsworth
Enthusiast
Posts: 56
Liked: 10 times
Joined: Nov 21, 2011 5:41 am
Location: Hong Kong
Contact:

Re: 9.5 Hyper-V 2016 Known Issues

Post by davidkillingsworth » 1 person likes this post

I just want to add that I fixed this issue with a brand new setup with Hyper-V 2019 and Veeam 10.

New Dell PowerEdge Server stand alone server.
New Windows Server 2019 Hyper-V host stand alone server.
3 New Windows Server 2019 VM guests
1 New Ubuntu 20.04 Server VM guest
Veeam Backup & Replication 10.0 Community Edition.

The issue I was having was that the backup and checkpoint would get to 7% or 50% and freeze. The backup job froze, then VM guests froze. I was not able to gracefully shut down the VM guests. When I tried to reboot the Hyper-V host, it froze.

I had to use the Dell iDrac card to physically restart the Hyper-V host.

After changing the checkpoint type to Standard in the VM's settings, then rebooted all VM guests and VM host, I ran the backup again and the checkpoint succeeded in a few seconds and the backup ran without any problems.
sunnyIT20
Lurker
Posts: 2
Liked: never
Joined: Dec 28, 2020 7:36 pm
Contact:

Re: 9.5 Hyper-V 2016 Known Issues

Post by sunnyIT20 »

Having this issue myself. One guest alone has 335 avhdx files sized 5g-20gb for a total of 2.5tb, just made aware as storage low alerts triggered. Have a couple other guests with same issue on a 2nd hyper v host. Backups have run successfully without error for last 3 years. No errors creating new checkpoints in hyperv mgr. Hosts and guest are all server2021R2, have all KBs mentioned in these posts. Veeam 9.5u4. Have not tried and would like to avoid manual merge process. Any advice or recent updates to this issue? Any reason why it wouldnt be easier to just create a replica than use that and delete old avhdx?

If not my attempts will consist of these two links mentioned here prior:
https://docs.microsoft.com/en-us/troubl ... -vms-fails
https://www.altaro.com/hyper-v/clean-up ... heckpoint/
https://gallery.technet.microsoft.com/s ... e-72a96c17
wishr
Veteran
Posts: 3077
Liked: 453 times
Joined: Aug 07, 2018 3:11 pm
Full Name: Fedor Maslov
Contact:

Re: 9.5 Hyper-V 2016 Known Issues

Post by wishr »

Hi Sunny,

Welcome to the forums!

What particular issue out of those discussed in this thread are you referring to?

Thanks
LarsBorris
Lurker
Posts: 1
Liked: never
Joined: Jun 30, 2021 7:53 am
Full Name: Lars Borris
Contact:

Re: 9.5 Hyper-V 2016 Known Issues

Post by LarsBorris »

ChristineAlexa wrote: Aug 26, 2019 7:22 am Another fix I didn't see in here, and will add to the list to help save someone else a few hours of hunting

- Symptom - Errors Similar to the below, where %1 is the VM name and %2 is the Virtual Machine ID)

'Checkpoint operation for %1 failed. (Virtual machine %2) Checkpoint operation for '%1' was cancelled. (Virtual machine ID %2) '%1' could not initiate a checkpoint operation: Element not found. (0x80070490). Error code: '32789'. Failed


If you have MPIO enabled on the VM,
- ensure you didn't accidently check "Add Support for SAS Devices".
- You may see errors in event log for volumes that don't exist not mounting.

To uncheck that box (since you can't in the interface), (You most likely will need to reboot afterwards, even if not prompted)
- Open a Powershell session in the VM
- Type the following command: Disable-MSDSMAutomaticClaim -BusType SAS

Once you have reboooted, follow the normal settings for Production Checkpoints, and it should work finally.
Thank you SO much! This was exactly my issue.
You saved my day. Have a great one. Cheers.
Post Reply

Who is online

Users browsing this forum: No registered users and 25 guests