Increasing VMCX files in VMs that are Veeam replicated

[GregaB]

Hi all. I was planning on opening a support case for this, but first I'm going to check if this is something that somebody else might have come across and I just can can't find it with my searches..

I have a 2 node Hyper-V cluster, both nodes are Server 2016, patched up to 2018-07 patches (didn't yet do the latest 2018-08..). I have a few VMs that I replicate to a standalone Hyper-V host very regularly (hourly) and a few that I replicate only twice a day, and I'm noticing a problem on these VMs, the configuration files for them (VMCX files) are increasing by about 100kB per day per hourly replicated VMs and like 10kB per day on twice daily replicated ones. VMs that are not replicated at all, don't show this. This has become a problem, as some VMCX are now over 20MB and these VMs now can't be successfully Live Migrated to another node, they get timed out when moving, while VMs with normal size VMCX files can be moved fine.

The weird thing is, I have another client with a similar setup, and I'm not observing the same there, the replicated VMs VMCXes there are not increasing in size..

This Veeam was at version 9.5 Update 2 at the time that I noticed this problem, since then I updated to Update 3a and it's still happening.

Anybody else noticing this and do we have a fix for this already or should I open a support case for this?

[DGrinev]

Hi Grega and welcome to the community!

I've discussed the said issue with the QA team, they've seen similar behavior, but it's related to the Hyper-V process, since Veeam doesn't write any information into VMCX.
You can reach the support team and ask for assistance, they should have additional information about the issue. Thanks!

[GregaB]

Thank you for this, I will open a support case for this issue and hopefully we'll get this resolved. I'll make sure to post results here for anybody else experiencing the same in the future.

[DGrinev]

Please share the case id here once it will be raised, so we can track investigation internally. Thanks!

[GregaB]

Sure, the case ID is #03168787.

[stevemciow]

I know this is an old thread, but we're experiencing exactly the same issue - we have a number of VMs in our live environment that are replicated every 4 hours to our DR site. They have now got .VMCX files that are around 20MB and will no longer live-migrate between our Hyper-V 2016 hosts.

Our test VMs, which were copies of the live ones, have .VMCX files of around 80KB, and live migrate without issues.

Was there a fix for this found?

I'll raise a case with support as well.

[GregaB]

I replied to you in the PM already, but I'll write this here for some other poor souls having these issues..

Unfortunately, no, we still haven't found the final solution to this problem.. We also managed to rule out Veeam as a cause of this problem, as it also happens if you do lots of checkpoints manually within Hyper-V itself. We have a case open with Microsoft for this. I moved on from this project, so I might not have the latest info at hand right now, but I'll check with the person who took over from me and see where they're at now.. If there's a fix from Microsoft, I'll certainly come back with the solution here!

In the meantime, I did find a workaround for this issue, not very elegant, as it needs VMs shutting down, but it is a workaround nonetheless. You can get the VMCX files back down to normal sizes if you shut down the VMs, export them, delete the originals and import them back into Hyper-V.. Then you have full functionality back, until of course the VMCXes get too large again and you have to do the workaround again, but still, better than nothing!

[benthomas]

I too am experiencing the same issue. However I wasn't able to reproduce it with normal checkpoints, only with the recovery checkpoints that backup software like Veeam uses.
I've got a Veeam support case #03414680 open as well as a Microsoft Premier support case open for this.

[Mike Resseler]

Hi Ben,

What type of checkpoints are you running if you do it through Hyper-V? If you do this with production checkpoints, it should have the same issue.

[stevemciow]

Apparently there is a tool that Microsoft can provide that will compact the .VMCX file - although at present they only supplied it under an NDA, so I'm trying to get them to give me a copy. If I do get one, and they're ok with it being made available, I'll update this post.

[benthomas]

Seems it might be the same issue as mentioned here
microsoft-hyper-v-f25/guest-os-starting ... ml#p302379

[benthomas]

Apparently there is a tool that Microsoft can provide that will compact the .VMCX file - although at present they only supplied it under an NDA, so I'm trying to get them to give me a copy. If I do get one, and they're ok with it being made available, I'll update this post.

It shouldn't need a NDA tool, there are WMI methods that can be called to remove the leftover reference points that Veeam isn't cleaning up.

[benthomas]

Hi Ben,

What type of checkpoints are you running if you do it through Hyper-V? If you do this with production checkpoints, it should have the same issue.

Hi Mike,

This seems to be a fault with the implementation of Hyper-V RCT by Veeam, where by Veeam isn't cleaning up the Reference points created as part of the backup and recover process.
This needs to be addressed by the Veeam Developers, not Microsoft

[Mike Resseler]

Hey Ben,

I got informed this should be fixed in update 4. Are you running the latest update already?

[benthomas]

Hey Ben,

I got informed this should be fixed in update 4. Are you running the latest update already?

I'm not running Update 4 yet. It would be nice if this was actually documented somewhere, as this is a massive improvement for Hyper-V environments if it's actually fixed in Update 4.

[WorkingHardInIt]

With VBR 9.5 U4 is had to add the reg key or the ref points would still accumulate.

[tomtim]

Hello, can you please tell me, which reg key I have to add, in order to prevent, that the ref points accumulate.

Thanks for your answer.

Best regards,

Thomas

[Mike Resseler]

Hi Thomas,
First: Welcome to the forums!
Second: This should be the reg key:

Path: HKEY_LOCAL_MACHINE\SOFTWARE\Veeam\Veeam Backup and Replication\
Name:HyperVReferencePointCleanup

Type: REG_DWORD

Value: 1

[tomtim]

Hi Mike,

thanks a lot for your answer.It works !

Great !

I have another question ? Will the old reference points be deleted automatically sometime?

Best regards,
Thomas

[hasoft]

I think yes, they are deleted after backup or replication of virtual machine.

[tomtim]

I found the following. No new reference points are created. The VMCX file does not grow anymore, but it keeps its size and does not shrink automatically. After I moved the VM to another host via live migration, the old reference points are taken from the VMCX file and the file has its original small output size again.

[chuck614]

We have tried this registry key and it does clean up reference points, but that ends up breaking any other product/service that uses reference points. For example we use Veeam backups but also Hyper-V replica. Hyper-V replica is ruined if all reference points are removed vs. just what Veeam created.

[foggy]

Hi Kurt, I suggest asking support to take a closer look at what happens. Thanks!

[chuck614]

We got the usual response of "this is a Microsoft problem" Which is funny, since all Microsoft products are able to clean up their reference points without impacting Veeam. Not sure why Veeam can't do the same. For VMs to become unusable in live migration after Veeam backs them up enough times is a bug. Hopefully this registry key is not the permanent fix.

[benthomas]

I too have seen reports of this registry key fix causing issues when the VMs are also protected with Hyper-V Replica.
Is Veeam's implementation of the cleanup just removing all reference points? Rather than 'Known' veeam related reference points?

[nmdange]

You can use this powershell script to cleanup Veeam reference points without breaking Hyper-V Replica
post338672.html#p338672