-
- Influencer
- Posts: 18
- Liked: 7 times
- Joined: Jan 16, 2023 3:13 pm
- Full Name: Joel G
- Contact:
AVHDX not merging after backups
Having an issue where avhdx's are filling up our host drive and crashing VM's. This looks to be an issue with Hyper-V not being able to merge the avhdx after backup is complete. I opened a support ticket with Veeam, but it appears to be more a Microsoft/Hyper-V issue (I don't disagree, also opened a ticket with Microsoft), just wondering if anyone else has had this experience and found a solution.
1. Veeam requests Hyper-V VSS to create a cohesive point-in-time copy of a VM
2. Hyper-V VSS creates a VM checkpoint
3. Backup completes
4. Disk merge starts - Event ID 19070
5. Disk merge gets interrupted - Event ID 19090
6. At this point, the avhdx continues to grow throughout the day. Eventually filling the host disk and causing the VM(s) to crash.
7. Running "Get-VMSnapshot -VMName VM01" showed no outstanding checkpoints
8. No checkpoints show in Hyper-V Manager
9. Shutting the VM down will kick off the disk merge through Hyper-V Manager automatically, however that isn't practical on production VM's
10. If I don't shut down the VM to trigger the merge, I'm seeing the following in the next backup job log:
a. Cannot use CBT: The process cannot access the file because it is being used by another process. Failed to open VHD/VHDX/VHDS disk 'E:\VM01\Virtual Hard Disks\VM01_....avhdx' in readonly mode.
11. Sometimes after the next backup the disk merge is successful (Event ID 19080)
I’m seeing other related errors in Hyper-V-VMMS Admin event log:
• Event ID 15268 - Failed to get the disk information.
• Event ID 12240 - '': Attachment 'E:\VM01\Virtual Hard Disks\VM01_....avhdx' not found. Error: 'The system cannot find the file specified.' (0x80070002). (Virtual machine ID )
• Event ID 14260 - User 'LOCAL SERVICE' does not have the permission required to perform the operation on 'VM01'. (Virtual machine ID …)
Any advice would be appreciated,
Joel
1. Veeam requests Hyper-V VSS to create a cohesive point-in-time copy of a VM
2. Hyper-V VSS creates a VM checkpoint
3. Backup completes
4. Disk merge starts - Event ID 19070
5. Disk merge gets interrupted - Event ID 19090
6. At this point, the avhdx continues to grow throughout the day. Eventually filling the host disk and causing the VM(s) to crash.
7. Running "Get-VMSnapshot -VMName VM01" showed no outstanding checkpoints
8. No checkpoints show in Hyper-V Manager
9. Shutting the VM down will kick off the disk merge through Hyper-V Manager automatically, however that isn't practical on production VM's
10. If I don't shut down the VM to trigger the merge, I'm seeing the following in the next backup job log:
a. Cannot use CBT: The process cannot access the file because it is being used by another process. Failed to open VHD/VHDX/VHDS disk 'E:\VM01\Virtual Hard Disks\VM01_....avhdx' in readonly mode.
11. Sometimes after the next backup the disk merge is successful (Event ID 19080)
I’m seeing other related errors in Hyper-V-VMMS Admin event log:
• Event ID 15268 - Failed to get the disk information.
• Event ID 12240 - '': Attachment 'E:\VM01\Virtual Hard Disks\VM01_....avhdx' not found. Error: 'The system cannot find the file specified.' (0x80070002). (Virtual machine ID )
• Event ID 14260 - User 'LOCAL SERVICE' does not have the permission required to perform the operation on 'VM01'. (Virtual machine ID …)
Any advice would be appreciated,
Joel
-
- Veeam Software
- Posts: 2334
- Liked: 554 times
- Joined: Jun 28, 2016 12:12 pm
- Contact:
Re: AVHDX not merging after backups
Hi Joel,
Thank you for the detailed description of the issue. May I ask you to share the Veeam Support case number? It sounds like you and Support are approaching the same conclusion, but would be helpful just to track the issue.
Your solution in step 9 likely will be required regrettably unless Microsoft Support is able to assist; I would advise simply show the stacked up avhdx files to the Microsoft representative on your Support Case and then the checkpoint list in the Hyper-V manager (and the Powershell output) and ask them to explain the discrepancy.
"The process cannot access the file because it is being used by another process"
This + the permissions event error starts me thinking about anti-virus interference, but that's mostly a guess; the above error is an OS error that Veeam Backup and Replication returns when it cannot access something due to lock conflicts; VBR tracks the resources it locks during any operation and will give a more informative message if there is another Veeam operation locking the same resource, and usually AV is the culprit.
However, as noted please post the case number and show your Microsoft case representative the same issues with the avhdx files existing on disk but not shown in Powershell/Hyper-V manager.
Thank you for the detailed description of the issue. May I ask you to share the Veeam Support case number? It sounds like you and Support are approaching the same conclusion, but would be helpful just to track the issue.
Your solution in step 9 likely will be required regrettably unless Microsoft Support is able to assist; I would advise simply show the stacked up avhdx files to the Microsoft representative on your Support Case and then the checkpoint list in the Hyper-V manager (and the Powershell output) and ask them to explain the discrepancy.
"The process cannot access the file because it is being used by another process"
This + the permissions event error starts me thinking about anti-virus interference, but that's mostly a guess; the above error is an OS error that Veeam Backup and Replication returns when it cannot access something due to lock conflicts; VBR tracks the resources it locks during any operation and will give a more informative message if there is another Veeam operation locking the same resource, and usually AV is the culprit.
However, as noted please post the case number and show your Microsoft case representative the same issues with the avhdx files existing on disk but not shown in Powershell/Hyper-V manager.
David Domask | Product Management: Principal Analyst
-
- Influencer
- Posts: 18
- Liked: 7 times
- Joined: Jan 16, 2023 3:13 pm
- Full Name: Joel G
- Contact:
Re: AVHDX not merging after backups
Sure, case 07212387.
I'm still waiting to hear back from Microsoft (I did have a screen share with them, but the issue was slightly different - I could see a checkpoint, just couldn't delete or apply it).
We are using Microsoft Defender for Business as our AV, and have implemented all the Microsoft recommended exclusions as noted in https://learn.microsoft.com/en-us/troub ... er-v-hosts
This is a fairly intermittent issue, not always the same VM, and sometimes it clears itself with the next backup. It is happening mutliple times a week though.
Any suggestions appreciated,
Joel
I'm still waiting to hear back from Microsoft (I did have a screen share with them, but the issue was slightly different - I could see a checkpoint, just couldn't delete or apply it).
We are using Microsoft Defender for Business as our AV, and have implemented all the Microsoft recommended exclusions as noted in https://learn.microsoft.com/en-us/troub ... er-v-hosts
This is a fairly intermittent issue, not always the same VM, and sometimes it clears itself with the next backup. It is happening mutliple times a week though.
Any suggestions appreciated,
Joel
-
- Veeam Software
- Posts: 2334
- Liked: 554 times
- Joined: Jun 28, 2016 12:12 pm
- Contact:
Re: AVHDX not merging after backups
Thanks for the update and the case number Joel; let's see what the Microsoft representative says on the subjects mentioned. No specific suggestions right now if the AV exclusions are set, but if it's happening frequently, it might help Support to get fresh logs from the job and Hyper-V host just so they have a "fresh" view of the incident. The same Event logs likely would help the Microsoft representative also.
David Domask | Product Management: Principal Analyst
-
- Lurker
- Posts: 1
- Liked: never
- Joined: Apr 26, 2024 2:11 pm
- Full Name: Kenan Redzic
- Contact:
Re: AVHDX not merging after backups
Hey guys, also interested in the solution that you end up finding, for now the only work-around that we have found here is to restart the Hyper-V Machine Management service before the backups are scheduled to kick off which usually allows the file to merge after the job is completed. However, this isn't a perfect fix as we still have situations where it fails to merge after the job is completed, so we would also greatly appreciate hearing what Microsoft has to say on this issue.
-
- Lurker
- Posts: 1
- Liked: never
- Joined: Aug 22, 2024 11:15 am
- Full Name: Mark Roberts
- Contact:
Re: AVHDX not merging after backups
Has there been any update on this issue from MS? We are experiencing the same thing sporadically, but frustratingly.
-
- Veeam Software
- Posts: 2334
- Liked: 554 times
- Joined: Jun 28, 2016 12:12 pm
- Contact:
Re: AVHDX not merging after backups
Hi Mark, welcome to the forums.
For the case mentioned in this topic, the case was closed without resolution due to no further updates on the case.
Such issues tend to be fairly specific to a given environment, so please open a Support Case and include logs from one of the affected jobs and allow Veeam Support time to review the logs. (Use the 1st radio button during log export and select one of the affected jobs)
Please share your case number once created.
Thanks!
For the case mentioned in this topic, the case was closed without resolution due to no further updates on the case.
Such issues tend to be fairly specific to a given environment, so please open a Support Case and include logs from one of the affected jobs and allow Veeam Support time to review the logs. (Use the 1st radio button during log export and select one of the affected jobs)
Please share your case number once created.
Thanks!
David Domask | Product Management: Principal Analyst
-
- Influencer
- Posts: 18
- Liked: 7 times
- Joined: Jan 16, 2023 3:13 pm
- Full Name: Joel G
- Contact:
Re: AVHDX not merging after backups
I am still experiencing this issue intermittently. Veeam support responded:
"However, as this issue resides on the Microsoft side of things, I am unable to provide specific troubleshooting steps to resolve this. If the job is successful, production checkpoints are being taken and removed as Veeam requires. Veeam Backup & Replication requests Microsoft Hyper-V VSS to create a cohesive point-in-time copy of a VM. Microsoft Hyper-V VSS creates a volume snapshot or VM checkpoint, depending on the hypervisor version. Unfortunately, I am unable to assist with troubleshooting expanding AVHDX files on the Hyper-V side."
I opened a support case with Microsoft, but was unable to repro for 2 weeks and that case was closed. My workaround is moving the VM's to SAN storage with plenty of free space to handle the AVHDX until they merge on their own. Not ideal, but at least the VM's aren't crashing.
Joel
"However, as this issue resides on the Microsoft side of things, I am unable to provide specific troubleshooting steps to resolve this. If the job is successful, production checkpoints are being taken and removed as Veeam requires. Veeam Backup & Replication requests Microsoft Hyper-V VSS to create a cohesive point-in-time copy of a VM. Microsoft Hyper-V VSS creates a volume snapshot or VM checkpoint, depending on the hypervisor version. Unfortunately, I am unable to assist with troubleshooting expanding AVHDX files on the Hyper-V side."
I opened a support case with Microsoft, but was unable to repro for 2 weeks and that case was closed. My workaround is moving the VM's to SAN storage with plenty of free space to handle the AVHDX until they merge on their own. Not ideal, but at least the VM's aren't crashing.
Joel
-
- Novice
- Posts: 8
- Liked: 1 time
- Joined: Sep 17, 2024 3:17 pm
- Full Name: Brian I
- Contact:
Re: AVHDX not merging after backups
We see this same intermittent behavior with a completely different backup vendor. And we're also left with a similar comment that it's up to MS to troubleshoot and resolve. However, that MS 'support' is completely lacking and never fully resolves anything for us. So we're left with workarounds like restarting the VMMS service or Live Migrating the guest to another host to trigger the merge again.
This is something that is either going to take a large Corporate client or a combined effort by the vendors to drag the actual MS engineers back to the table to resolve their bug.
This is something that is either going to take a large Corporate client or a combined effort by the vendors to drag the actual MS engineers back to the table to resolve their bug.
-
- Novice
- Posts: 8
- Liked: 1 time
- Joined: Sep 17, 2024 3:17 pm
- Full Name: Brian I
- Contact:
Re: AVHDX not merging after backups
This issue resolved on its own after a host crashed. Whatever locks on those files was finally freed.
Now after our backups from the end of December, the issue has started again. Same errors as before, and the usual workarounds are working.
I wonder if it's somehow related to the files becoming stuck in buffered mode as in this other thread:
microsoft-hyper-v-f25/windows-server-20 ... 62112.html
Now after our backups from the end of December, the issue has started again. Same errors as before, and the usual workarounds are working.
I wonder if it's somehow related to the files becoming stuck in buffered mode as in this other thread:
microsoft-hyper-v-f25/windows-server-20 ... 62112.html
-
- Influencer
- Posts: 18
- Liked: 7 times
- Joined: Jan 16, 2023 3:13 pm
- Full Name: Joel G
- Contact:
Re: AVHDX not merging after backups
I think the two issues are quite different, but we are experiencing that issue as well. There are no extra files created with the other issue, and live migration seems to be a workaround for it. Where this issue only merges the AVHDX when the VM is powered off or the next backup runs.
Looks like we'll be upgrading to Server 2022 to resolve the slow I/O issue, maybe we'll get lucky and this will resolve as well.
If it is of any help, I did create an automated Task Scheduler task to run a PowerShell script that e-mails me every day with a list of outstanding AVHDX:
Joel
Looks like we'll be upgrading to Server 2022 to resolve the slow I/O issue, maybe we'll get lucky and this will resolve as well.
If it is of any help, I did create an automated Task Scheduler task to run a PowerShell script that e-mails me every day with a list of outstanding AVHDX:
Code: Select all
# check for existence of AVHDX
# Joel Goudreau - Sept 6, 2024
# Start PowerShell Logging
Start-Transcript -Path .\TaskLogs.txt
# Cleanup and set variables
Write-Host "Creating variables and cleaning up"
$recipients = ("name1@domain.com","name2@domain.com)
if (Test-Path .\Hyper-V_AVHDX.csv) { Remove-Item .\Hyper-V_AVHDX.csv }
# Collect events
Write-Host "Checking Clustered Shared Volumes"
Get-ChildItem -Path \\cluster.domain.local\ClusterStorage$ -Filter *.avhdx -Recurse | Select-Object -ExpandProperty FullName | Out-File .\Hyper-V_AVHDX.csv -Append
Write-Host "Checking Server1 local disks"
Get-ChildItem -Path \\server1\D$ -Filter *.avhdx -Recurse | Select-Object -ExpandProperty FullName | Out-File .\Hyper-V_AVHDX.csv -Append
# E-mail log
Write-Host "E-mailing results"
if ((Get-Item '.\Hyper-V_AVHDX.csv').length -eq 0) {
$sendMailMessageSplat = @{
From = 'NAME <NAME@domain.com>'
To = $recipients
Subject = 'AVHDX Daily Summary'
Body = 'There are no outstanding AVHDX.'
SmtpServer = 'mailserv.domain.local'
}
} else {
$sendMailMessageSplat = @{
From = 'NAME <NAME@domain.com>'
To = $recipients
Subject = 'AVHDX Daily Summary'
Body = 'The attached VMs have outstanding AVHDX.'
Attachments = '.\Hyper-V_AVHDX.csv'
SmtpServer = 'mailserv.domain.local'
}
}
Send-MailMessage @sendMailMessageSplat
# Stop PowerShell logging
Write-Host "Script complete, stopping transcript"
Stop-Transcript
-
- Novice
- Posts: 8
- Liked: 1 time
- Joined: Sep 17, 2024 3:17 pm
- Full Name: Brian I
- Contact:
Re: AVHDX not merging after backups
My thought was that if there's also some kind of redirect going on during the backup cycle, it could explain the delay Windows has releasing/unmounting the disks when the backup solution sends its command.
Who is online
Users browsing this forum: No registered users and 22 guests