Case 05188307 - Windows 2019 / Veeam 11 / .avhdx being leftover after backup

[willomatic2022]

Hi folks,

I've done a fair bit of searching around this problem and came across some stuff relating to the issue on Windows 2016. As of yet I can't find a way of resolving the problem on windows 2019.

The issue in a nutshell:

Windows Server hyper-v cluster with 3 hosts.
A windows 2012 R2 guest running SQL (approx 1.3 TB in size in total)
Veeam B&R 11.0.1.1261

We have this 1 SQL guest that is misbehaving. The problem is sporadic and doesn't always occur.

The SQL host has its own dedicated backup job that runs @1900 everyday. Its application aware. After the backup job finishes, instead of merging the .avhdx files into the parent .vhdx file, they remain on the disk and this begins to fill up the disk. It doesn't happen all the time, just now and again, but because of the size of the server and it being busy, the resulting .avhx files can be extremely large.

I have done the following to troubleshoot:

- Shut the host down, merged all the disks. Taken snapshot, then exported the snapshot and recreated the VM.
- Merging checkpoints using powershell
- Changed the backup start time a few times
- Given the host its own dedicated job

There is a ticket open with Veeam about it but they are being slow.

Has anyone else had this problem of leftover avhdx files?? How did you fix it?

[HannesK]

Hello,
and welcome to the forums.

As far as I see, that case is complex, which might explain why it's open that long and still not solved (also it looks like support sometimes had to wait for logs from your side). If you feel that things are stuck with support, did you already use the "talk to manager" button and ask what's the reason for not finding a solution?

Best regards,
Hannes

[willomatic2022]

I'll try the "talk to manager" option. Thanks.

[nmdange]

Is there a recovery snapshot visible on the VM if you do Get-VMSnapshot in Powershell? I occasionally have this happen and just remove the snapshot manually through Powershell. I've never had to manually merge avhdx files.

[willomatic2022]

Snapshots look like this:

VMName Name SnapshotType CreationTime ParentSnapshotName
------ ---- ------------ ------------ ------------------
CXGSQL01_new CXGSQL01_new - (12/02/2022 - 19:04:17) Recovery 12/02/2022 19:04:28
CXGSQL01_new CXGSQL01_new - (12/02/2022 - 19:07:07) Recovery 12/02/2022 19:07:19 CXGSQL01_new - (12/02/2022 - 19:04:17)
CXGSQL01_new CXGSQL01_new - (12/02/2022 - 19:09:57) Recovery 12/02/2022 19:10:13 CXGSQL01_new - (12/02/2022 - 19:07:07)
CXGSQL01_new CXGSQL01_new - (12/02/2022 - 19:12:47) Recovery 12/02/2022 19:13:00 CXGSQL01_new - (12/02/2022 - 19:09:57)
CXGSQL01_new CXGSQL01_new - (13/02/2022 - 19:04:01) Recovery 13/02/2022 19:04:12 CXGSQL01_new - (12/02/2022 - 19:12:47)
CXGSQL01_new CXGSQL01_new - (15/02/2022 - 01:04:19) Recovery 15/02/2022 01:04:25 CXGSQL01_new - (13/02/2022 - 19:04:01)

[willomatic2022]

Whats alarming is that the folder size where all the snapshots are is roughly doubled! The VM is only about 1.2TB in size, but the Clustered Shared Volumes folder where the .vhdx + .avhdx files are residing is 2.4TB now!

PS C:\ClusterStorage\Volume6\CXGSQL01_exported> (gci . | measure length -s).sum /1tb
2.44767934083939

If we don't manually clear them eventually the CSV fills and then all havoc breaks loose.

[nmdange]

This may be a Hyper-V/Microsoft issue, but that aside, this is a good guide on how to cleanup those snapshots https://blog.workinghardinit.work/2015/ ... l-machine/

It's not a fix but as long as you cleanup the snapshots yourself quickly you won't run out of space.