Host-based backup of Microsoft Hyper-V VMs.
Post Reply
joelg
Influencer
Posts: 18
Liked: 7 times
Joined: Jan 16, 2023 3:13 pm
Full Name: Joel G
Contact:

AVHDX not merging after backups

Post by joelg »

Having an issue where avhdx's are filling up our host drive and crashing VM's. This looks to be an issue with Hyper-V not being able to merge the avhdx after backup is complete. I opened a support ticket with Veeam, but it appears to be more a Microsoft/Hyper-V issue (I don't disagree, also opened a ticket with Microsoft), just wondering if anyone else has had this experience and found a solution.

1. Veeam requests Hyper-V VSS to create a cohesive point-in-time copy of a VM
2. Hyper-V VSS creates a VM checkpoint
3. Backup completes
4. Disk merge starts - Event ID 19070
5. Disk merge gets interrupted - Event ID 19090
6. At this point, the avhdx continues to grow throughout the day. Eventually filling the host disk and causing the VM(s) to crash.
7. Running "Get-VMSnapshot -VMName VM01" showed no outstanding checkpoints
8. No checkpoints show in Hyper-V Manager
9. Shutting the VM down will kick off the disk merge through Hyper-V Manager automatically, however that isn't practical on production VM's
10. If I don't shut down the VM to trigger the merge, I'm seeing the following in the next backup job log:
a. Cannot use CBT: The process cannot access the file because it is being used by another process. Failed to open VHD/VHDX/VHDS disk 'E:\VM01\Virtual Hard Disks\VM01_....avhdx' in readonly mode.
11. Sometimes after the next backup the disk merge is successful (Event ID 19080)

I’m seeing other related errors in Hyper-V-VMMS Admin event log:
• Event ID 15268 - Failed to get the disk information.
• Event ID 12240 - '': Attachment 'E:\VM01\Virtual Hard Disks\VM01_....avhdx' not found. Error: 'The system cannot find the file specified.' (0x80070002). (Virtual machine ID )
• Event ID 14260 - User 'LOCAL SERVICE' does not have the permission required to perform the operation on 'VM01'. (Virtual machine ID …)

Any advice would be appreciated,
Joel
david.domask
Veeam Software
Posts: 2334
Liked: 554 times
Joined: Jun 28, 2016 12:12 pm
Contact:

Re: AVHDX not merging after backups

Post by david.domask »

Hi Joel,

Thank you for the detailed description of the issue. May I ask you to share the Veeam Support case number? It sounds like you and Support are approaching the same conclusion, but would be helpful just to track the issue.

Your solution in step 9 likely will be required regrettably unless Microsoft Support is able to assist; I would advise simply show the stacked up avhdx files to the Microsoft representative on your Support Case and then the checkpoint list in the Hyper-V manager (and the Powershell output) and ask them to explain the discrepancy.

"The process cannot access the file because it is being used by another process"

This + the permissions event error starts me thinking about anti-virus interference, but that's mostly a guess; the above error is an OS error that Veeam Backup and Replication returns when it cannot access something due to lock conflicts; VBR tracks the resources it locks during any operation and will give a more informative message if there is another Veeam operation locking the same resource, and usually AV is the culprit.

However, as noted please post the case number and show your Microsoft case representative the same issues with the avhdx files existing on disk but not shown in Powershell/Hyper-V manager.
David Domask | Product Management: Principal Analyst
joelg
Influencer
Posts: 18
Liked: 7 times
Joined: Jan 16, 2023 3:13 pm
Full Name: Joel G
Contact:

Re: AVHDX not merging after backups

Post by joelg »

Sure, case 07212387.

I'm still waiting to hear back from Microsoft (I did have a screen share with them, but the issue was slightly different - I could see a checkpoint, just couldn't delete or apply it).

We are using Microsoft Defender for Business as our AV, and have implemented all the Microsoft recommended exclusions as noted in https://learn.microsoft.com/en-us/troub ... er-v-hosts

This is a fairly intermittent issue, not always the same VM, and sometimes it clears itself with the next backup. It is happening mutliple times a week though.

Any suggestions appreciated,
Joel
david.domask
Veeam Software
Posts: 2334
Liked: 554 times
Joined: Jun 28, 2016 12:12 pm
Contact:

Re: AVHDX not merging after backups

Post by david.domask »

Thanks for the update and the case number Joel; let's see what the Microsoft representative says on the subjects mentioned. No specific suggestions right now if the AV exclusions are set, but if it's happening frequently, it might help Support to get fresh logs from the job and Hyper-V host just so they have a "fresh" view of the incident. The same Event logs likely would help the Microsoft representative also.
David Domask | Product Management: Principal Analyst
Veliki Kokuruz
Lurker
Posts: 1
Liked: never
Joined: Apr 26, 2024 2:11 pm
Full Name: Kenan Redzic
Contact:

Re: AVHDX not merging after backups

Post by Veliki Kokuruz »

Hey guys, also interested in the solution that you end up finding, for now the only work-around that we have found here is to restart the Hyper-V Machine Management service before the backups are scheduled to kick off which usually allows the file to merge after the job is completed. However, this isn't a perfect fix as we still have situations where it fails to merge after the job is completed, so we would also greatly appreciate hearing what Microsoft has to say on this issue.
mroberts@fewpb.com
Lurker
Posts: 1
Liked: never
Joined: Aug 22, 2024 11:15 am
Full Name: Mark Roberts
Contact:

Re: AVHDX not merging after backups

Post by mroberts@fewpb.com »

Has there been any update on this issue from MS? We are experiencing the same thing sporadically, but frustratingly.
david.domask
Veeam Software
Posts: 2334
Liked: 554 times
Joined: Jun 28, 2016 12:12 pm
Contact:

Re: AVHDX not merging after backups

Post by david.domask »

Hi Mark, welcome to the forums.

For the case mentioned in this topic, the case was closed without resolution due to no further updates on the case.

Such issues tend to be fairly specific to a given environment, so please open a Support Case and include logs from one of the affected jobs and allow Veeam Support time to review the logs. (Use the 1st radio button during log export and select one of the affected jobs)

Please share your case number once created.

Thanks!
David Domask | Product Management: Principal Analyst
joelg
Influencer
Posts: 18
Liked: 7 times
Joined: Jan 16, 2023 3:13 pm
Full Name: Joel G
Contact:

Re: AVHDX not merging after backups

Post by joelg » 1 person likes this post

I am still experiencing this issue intermittently. Veeam support responded:

"However, as this issue resides on the Microsoft side of things, I am unable to provide specific troubleshooting steps to resolve this. If the job is successful, production checkpoints are being taken and removed as Veeam requires. Veeam Backup & Replication requests Microsoft Hyper-V VSS to create a cohesive point-in-time copy of a VM. Microsoft Hyper-V VSS creates a volume snapshot or VM checkpoint, depending on the hypervisor version. Unfortunately, I am unable to assist with troubleshooting expanding AVHDX files on the Hyper-V side."

I opened a support case with Microsoft, but was unable to repro for 2 weeks and that case was closed. My workaround is moving the VM's to SAN storage with plenty of free space to handle the AVHDX until they merge on their own. Not ideal, but at least the VM's aren't crashing.

Joel
briani
Novice
Posts: 8
Liked: 1 time
Joined: Sep 17, 2024 3:17 pm
Full Name: Brian I
Contact:

Re: AVHDX not merging after backups

Post by briani »

We see this same intermittent behavior with a completely different backup vendor. And we're also left with a similar comment that it's up to MS to troubleshoot and resolve. However, that MS 'support' is completely lacking and never fully resolves anything for us. So we're left with workarounds like restarting the VMMS service or Live Migrating the guest to another host to trigger the merge again.

This is something that is either going to take a large Corporate client or a combined effort by the vendors to drag the actual MS engineers back to the table to resolve their bug.
briani
Novice
Posts: 8
Liked: 1 time
Joined: Sep 17, 2024 3:17 pm
Full Name: Brian I
Contact:

Re: AVHDX not merging after backups

Post by briani »

This issue resolved on its own after a host crashed. Whatever locks on those files was finally freed.

Now after our backups from the end of December, the issue has started again. Same errors as before, and the usual workarounds are working.

I wonder if it's somehow related to the files becoming stuck in buffered mode as in this other thread:

microsoft-hyper-v-f25/windows-server-20 ... 62112.html
joelg
Influencer
Posts: 18
Liked: 7 times
Joined: Jan 16, 2023 3:13 pm
Full Name: Joel G
Contact:

Re: AVHDX not merging after backups

Post by joelg »

I think the two issues are quite different, but we are experiencing that issue as well. There are no extra files created with the other issue, and live migration seems to be a workaround for it. Where this issue only merges the AVHDX when the VM is powered off or the next backup runs.
Looks like we'll be upgrading to Server 2022 to resolve the slow I/O issue, maybe we'll get lucky and this will resolve as well.

If it is of any help, I did create an automated Task Scheduler task to run a PowerShell script that e-mails me every day with a list of outstanding AVHDX:

Code: Select all

# check for existence of AVHDX
# Joel Goudreau - Sept 6, 2024

# Start PowerShell Logging
Start-Transcript -Path .\TaskLogs.txt

# Cleanup and set variables
Write-Host "Creating variables and cleaning up"
$recipients = ("name1@domain.com","name2@domain.com)
if (Test-Path .\Hyper-V_AVHDX.csv) { Remove-Item .\Hyper-V_AVHDX.csv }

# Collect events
Write-Host "Checking Clustered Shared Volumes"
Get-ChildItem -Path \\cluster.domain.local\ClusterStorage$ -Filter *.avhdx -Recurse | Select-Object -ExpandProperty FullName | Out-File .\Hyper-V_AVHDX.csv -Append

Write-Host "Checking Server1 local disks"
Get-ChildItem -Path \\server1\D$ -Filter *.avhdx -Recurse | Select-Object -ExpandProperty FullName | Out-File .\Hyper-V_AVHDX.csv -Append

# E-mail log
Write-Host "E-mailing results"
if ((Get-Item '.\Hyper-V_AVHDX.csv').length -eq 0) {
    $sendMailMessageSplat = @{
    From = 'NAME <NAME@domain.com>'
    To = $recipients
    Subject = 'AVHDX Daily Summary'
    Body = 'There are no outstanding AVHDX.'
    SmtpServer = 'mailserv.domain.local'
	}
} else {
    $sendMailMessageSplat = @{
    From = 'NAME <NAME@domain.com>'
    To = $recipients
    Subject = 'AVHDX Daily Summary'
    Body = 'The attached VMs have outstanding AVHDX.'
    Attachments = '.\Hyper-V_AVHDX.csv'
    SmtpServer = 'mailserv.domain.local'
	}
}
Send-MailMessage @sendMailMessageSplat

# Stop PowerShell logging
Write-Host "Script complete, stopping transcript"
Stop-Transcript
Joel
briani
Novice
Posts: 8
Liked: 1 time
Joined: Sep 17, 2024 3:17 pm
Full Name: Brian I
Contact:

Re: AVHDX not merging after backups

Post by briani »

My thought was that if there's also some kind of redirect going on during the backup cycle, it could explain the delay Windows has releasing/unmounting the disks when the backup solution sends its command.
Post Reply

Who is online

Users browsing this forum: No registered users and 22 guests