Discussions specific to the Microsoft Hyper-V hypervisor
Post Reply
Eugen Fournes
Lurker
Posts: 2
Liked: 1 time
Joined: Sep 06, 2019 4:10 pm
Full Name: Eugen Fournes
Contact:

Solved: Server 2019 Hyper-V new checkpoint stuck issue

Post by Eugen Fournes » Sep 06, 2019 4:20 pm 1 person likes this post

Keywords: Server 2019 Hyper-V checkpoint stuck broken Veeam backup AVHDX

Hopefully that's enough for our friend Google to find this.

WARNING: Some of these instructions are potentially harmful and can cause data loss, even if they are followed correctly. Use at your own risk. If the VM is running and it has critical data that has not been backed up, use another method (e.g. Windows Server Backup, manual copy, etc. from within the VM) to back it up somewhere else before following these instructions. You've been warned.

Problem: Backup runs normally for one run, then fails because the checkpoints can't be created after that. The first backup left behind checkpoints that are halfway merged.

Cause: The default permissions that Hyper-V assigns to the checkpoint files aren't sufficient for the checkpoint service to delete the old files, so it crashes after completing the merge on the first VHDX file. (Root cause: when setting up the folder structure for the VM storage in a non-default location, Hyper-V isn't smart enough to adjust permissions on the folders.)

Good solution: Set the Hyper-V service GUID "account" to have FULL ACCESS rights to the folders where the VHDX files are stored. (It should have full access to the entire VM.) In PowerShell started with Run As Administrator:

First, get the GUID of the VMs on your Hyper-V server. (You can also get these from the permissions on the "stuck" AVHDX files.)

Code: Select all

  get-vm | fl name, id

Output Example:

Code: Select all

    Name : MyVM
    Id   : d3599536-222a-4d6e-bb10-a6019c3f2b9b
 
    Name : TheirVM
    Id   : a0af7903-94b4-4a2c-b3b3-16050d5f80f

Second, grant full access to that GUID to the VM's hierarchy:

Code: Select all

  icacls <Folder with VHDS> /grant "NT VIRTUAL MACHINE\<VM GUID>":(OI)F

Example:

Code: Select all

    icacls "E:\Hyper-V\Virtual Machines\MyVM" /grant "NT VIRTUAL MACHINE\d3599536-222a-4d6e-bb10-a6019c3f2b9b":(OI)F
    icacls "Q:\Hyper-V\Virtual Machines\TheirVM" /grant "NT VIRTUAL MACHINE\a0af7903-94b4-4a2c-b3b3-16050d5f80f2":(OI)F
Bad solution: If you're in a hurry, like I was, just set the VM folders' permissions to grant FULL ACCESS to "Everyone". Then set the permissions as above after everything's fixed up.


To fix the broken checkpoint merges:

First, set the permissions as detailed above. Then shut down the virtual machine. If you're lucky, the merge should begin after a moment and complete successfully. It might go by quickly if the AVHDX's aren't very large. Check the folder to see if the checkpoint files are gone. If they are, the VM is good to go.

If the merge doesn't happen automatically, or seems to get stuck partway through, start up the VM again. If the merge still doesn't complete on its own, you'll have to merge it manually.

Manual merge: *** USE WITH CAUTION ***

Shut down the VM once more.

Open PowerShell with Run As Administrator.

Run the command:

Code: Select all

  Merge-VHD -Path <checkpoint file path> -Destination <parent file path>

CAUTION: IF THERE IS MORE THAN ONE LEVEL OF CHECKPOINT, YOU NEED TO MERGE THE CHECKPOINT FILES IN THE CORRECT ORDER OR YOU *WILL* LOSE DATA!

Find out which checkpoints are the Parent of the others by using the "Inspect" option on the virtual disk in the VM's settings in Hyper-V. You'll need to start with the last checkpoint file--that's likely the one that's showing as the active virtual disk.

Example (merge one checkpoint into another and then into the main VM file):

Code: Select all

  Merge-VHD -Path "E:\Hyper-V\Virtual Machines\MyVM\Virtual Hard Disks\MyVM-Disk1_1508F969-7684-48DC-9B9C-2DC6C5A3CEB7.avhdx" -Destination "E:\Hyper-V\Virtual Machines\MyVM\Virtual Hard Disks\MyVM-Disk1_295A473C-1134-37F7-3783-B3A7820D478F.avhdx"

  Merge-VHD -Path "E:\Hyper-V\Virtual Machines\MyVM\Virtual Hard Disks\MyVM-Disk1_295A473C-1134-37F7-3783-B3A7820D478F.avhdx" -Destination "E:\Hyper-V\Virtual Machines\MyVM\Virtual Hard Disks\MyVM-Disk1.vhdx"  

If this gives you a "file in use" error, check Hyper-V again, it might have started a merge on its own--let it run.
If this gives you a permissions error, check that the files have the correct permissions, as detailed above, plus the Administrator rights.
If this gives you some other error, sorry, that's outside the scope of this document.

After they're merged, go back into Hyper-V, remove the entire virtual disk (if you try to replace the existing one, it still thinks there's a checkpoint pending). Click "Continue" on the warning about checkpoints. Re-add the base disk to the VM. Repeat for any additional virtual disks on that VM.

After starting up the VM again, you may have to put the second (third, fourth, etc.) disks online manually, within the VM, using Administration Tools -> Computer Management -> Disk Management. Everything should be back to normal after that.


References:

https://support.microsoft.com/en-ca/hel ... ack-up-vms
https://docs.microsoft.com/en-us/powers ... w=win10-ps


Note: I didn't bother opening a Veeam support case because this issue wasn't specific to Veeam, but with Hyper-V, though it affects Veeam and other backup software that uses Hyper-V checkpoints, so figured I'd post the solution here.

HannesK
Veeam Software
Posts: 3814
Liked: 458 times
Joined: Sep 01, 2014 11:46 am
Location: Austria
Contact:

Re: Solved: Server 2019 Hyper-V new checkpoint stuck issue

Post by HannesK » Sep 10, 2019 5:04 am

Hello,
and welcome to the forum. Thanks for sharing with the community.

Best regards,
Hannes

Eugen Fournes
Lurker
Posts: 2
Liked: 1 time
Joined: Sep 06, 2019 4:10 pm
Full Name: Eugen Fournes
Contact:

Re: NOT Solved: Server 2019 Hyper-V new checkpoint stuck issue

Post by Eugen Fournes » Sep 13, 2019 3:12 pm

<sigh> I'm afraid I jumped the gun. After few days, the problem came back.

All permissions are as recommended, and we're still getting the orphaned checkpoint files.

Time to open a ticket with Veeam and probably Microsoft... I'll keep everyone informed how it goes.

Post Reply

Who is online

Users browsing this forum: No registered users and 11 guests