Host-based backup of Microsoft Hyper-V VMs.
Post Reply
GregaB
Novice
Posts: 4
Liked: 1 time
Joined: Aug 27, 2018 7:03 am
Full Name: Grega Blatnik
Contact:

Increasing VMCX files in VMs that are Veeam replicated

Post by GregaB »

Hi all. I was planning on opening a support case for this, but first I'm going to check if this is something that somebody else might have come across and I just can can't find it with my searches..

I have a 2 node Hyper-V cluster, both nodes are Server 2016, patched up to 2018-07 patches (didn't yet do the latest 2018-08..). I have a few VMs that I replicate to a standalone Hyper-V host very regularly (hourly) and a few that I replicate only twice a day, and I'm noticing a problem on these VMs, the configuration files for them (VMCX files) are increasing by about 100kB per day per hourly replicated VMs and like 10kB per day on twice daily replicated ones. VMs that are not replicated at all, don't show this. This has become a problem, as some VMCX are now over 20MB and these VMs now can't be successfully Live Migrated to another node, they get timed out when moving, while VMs with normal size VMCX files can be moved fine.

The weird thing is, I have another client with a similar setup, and I'm not observing the same there, the replicated VMs VMCXes there are not increasing in size..

This Veeam was at version 9.5 Update 2 at the time that I noticed this problem, since then I updated to Update 3a and it's still happening.

Anybody else noticing this and do we have a fix for this already or should I open a support case for this?
DGrinev
Veteran
Posts: 1943
Liked: 247 times
Joined: Dec 01, 2016 3:49 pm
Full Name: Dmitry Grinev
Location: St.Petersburg
Contact:

Re: Increasing VMCX files in VMs that are Veeam replicated

Post by DGrinev »

Hi Grega and welcome to the community!

I've discussed the said issue with the QA team, they've seen similar behavior, but it's related to the Hyper-V process, since Veeam doesn't write any information into VMCX.
You can reach the support team and ask for assistance, they should have additional information about the issue. Thanks!
GregaB
Novice
Posts: 4
Liked: 1 time
Joined: Aug 27, 2018 7:03 am
Full Name: Grega Blatnik
Contact:

Re: Increasing VMCX files in VMs that are Veeam replicated

Post by GregaB »

Thank you for this, I will open a support case for this issue and hopefully we'll get this resolved. I'll make sure to post results here for anybody else experiencing the same in the future.
DGrinev
Veteran
Posts: 1943
Liked: 247 times
Joined: Dec 01, 2016 3:49 pm
Full Name: Dmitry Grinev
Location: St.Petersburg
Contact:

Re: Increasing VMCX files in VMs that are Veeam replicated

Post by DGrinev »

Please share the case id here once it will be raised, so we can track investigation internally. Thanks!
GregaB
Novice
Posts: 4
Liked: 1 time
Joined: Aug 27, 2018 7:03 am
Full Name: Grega Blatnik
Contact:

Re: Increasing VMCX files in VMs that are Veeam replicated

Post by GregaB » 1 person likes this post

Sure, the case ID is #03168787.
stevemciow
Novice
Posts: 7
Liked: 2 times
Joined: Jun 13, 2018 9:59 am
Full Name: Stephen Courtney
Contact:

Re: Increasing VMCX files in VMs that are Veeam replicated

Post by stevemciow »

I know this is an old thread, but we're experiencing exactly the same issue - we have a number of VMs in our live environment that are replicated every 4 hours to our DR site. They have now got .VMCX files that are around 20MB and will no longer live-migrate between our Hyper-V 2016 hosts.

Our test VMs, which were copies of the live ones, have .VMCX files of around 80KB, and live migrate without issues.

Was there a fix for this found?

I'll raise a case with support as well.
GregaB
Novice
Posts: 4
Liked: 1 time
Joined: Aug 27, 2018 7:03 am
Full Name: Grega Blatnik
Contact:

Re: Increasing VMCX files in VMs that are Veeam replicated

Post by GregaB »

I replied to you in the PM already, but I'll write this here for some other poor souls having these issues..

Unfortunately, no, we still haven't found the final solution to this problem.. We also managed to rule out Veeam as a cause of this problem, as it also happens if you do lots of checkpoints manually within Hyper-V itself. We have a case open with Microsoft for this. I moved on from this project, so I might not have the latest info at hand right now, but I'll check with the person who took over from me and see where they're at now.. If there's a fix from Microsoft, I'll certainly come back with the solution here!

In the meantime, I did find a workaround for this issue, not very elegant, as it needs VMs shutting down, but it is a workaround nonetheless. You can get the VMCX files back down to normal sizes if you shut down the VMs, export them, delete the originals and import them back into Hyper-V.. Then you have full functionality back, until of course the VMCXes get too large again and you have to do the workaround again, but still, better than nothing!
benthomas
Veeam Vanguard
Posts: 39
Liked: 11 times
Joined: Apr 22, 2013 2:29 am
Full Name: Ben Thomas
Location: New Zealand
Contact:

Re: Increasing VMCX files in VMs that are Veeam replicated

Post by benthomas »

I too am experiencing the same issue. However I wasn't able to reproduce it with normal checkpoints, only with the recovery checkpoints that backup software like Veeam uses.
I've got a Veeam support case #03414680 open as well as a Microsoft Premier support case open for this.
Ben Thomas | Solutions Advisor | Veeam Vanguard 2023 | VMCE2022 | Microsoft MVP 2018-2023 | BCThomas.com
Mike Resseler
Product Manager
Posts: 8044
Liked: 1263 times
Joined: Feb 08, 2013 3:08 pm
Full Name: Mike Resseler
Location: Belgium
Contact:

Re: Increasing VMCX files in VMs that are Veeam replicated

Post by Mike Resseler »

Hi Ben,

What type of checkpoints are you running if you do it through Hyper-V? If you do this with production checkpoints, it should have the same issue.
stevemciow
Novice
Posts: 7
Liked: 2 times
Joined: Jun 13, 2018 9:59 am
Full Name: Stephen Courtney
Contact:

Re: Increasing VMCX files in VMs that are Veeam replicated

Post by stevemciow » 1 person likes this post

Apparently there is a tool that Microsoft can provide that will compact the .VMCX file - although at present they only supplied it under an NDA, so I'm trying to get them to give me a copy. If I do get one, and they're ok with it being made available, I'll update this post.
benthomas
Veeam Vanguard
Posts: 39
Liked: 11 times
Joined: Apr 22, 2013 2:29 am
Full Name: Ben Thomas
Location: New Zealand
Contact:

Re: Increasing VMCX files in VMs that are Veeam replicated

Post by benthomas »

Seems it might be the same issue as mentioned here
microsoft-hyper-v-f25/guest-os-starting ... ml#p302379
Ben Thomas | Solutions Advisor | Veeam Vanguard 2023 | VMCE2022 | Microsoft MVP 2018-2023 | BCThomas.com
benthomas
Veeam Vanguard
Posts: 39
Liked: 11 times
Joined: Apr 22, 2013 2:29 am
Full Name: Ben Thomas
Location: New Zealand
Contact:

Re: Increasing VMCX files in VMs that are Veeam replicated

Post by benthomas »

stevemciow wrote: Feb 15, 2019 1:39 pm Apparently there is a tool that Microsoft can provide that will compact the .VMCX file - although at present they only supplied it under an NDA, so I'm trying to get them to give me a copy. If I do get one, and they're ok with it being made available, I'll update this post.
It shouldn't need a NDA tool, there are WMI methods that can be called to remove the leftover reference points that Veeam isn't cleaning up.
Ben Thomas | Solutions Advisor | Veeam Vanguard 2023 | VMCE2022 | Microsoft MVP 2018-2023 | BCThomas.com
benthomas
Veeam Vanguard
Posts: 39
Liked: 11 times
Joined: Apr 22, 2013 2:29 am
Full Name: Ben Thomas
Location: New Zealand
Contact:

Re: Increasing VMCX files in VMs that are Veeam replicated

Post by benthomas »

Mike Resseler wrote: Feb 15, 2019 6:15 am Hi Ben,

What type of checkpoints are you running if you do it through Hyper-V? If you do this with production checkpoints, it should have the same issue.
Hi Mike,

This seems to be a fault with the implementation of Hyper-V RCT by Veeam, where by Veeam isn't cleaning up the Reference points created as part of the backup and recover process.
This needs to be addressed by the Veeam Developers, not Microsoft :)
Ben Thomas | Solutions Advisor | Veeam Vanguard 2023 | VMCE2022 | Microsoft MVP 2018-2023 | BCThomas.com
Mike Resseler
Product Manager
Posts: 8044
Liked: 1263 times
Joined: Feb 08, 2013 3:08 pm
Full Name: Mike Resseler
Location: Belgium
Contact:

Re: Increasing VMCX files in VMs that are Veeam replicated

Post by Mike Resseler »

Hey Ben,

I got informed this should be fixed in update 4. Are you running the latest update already?
benthomas
Veeam Vanguard
Posts: 39
Liked: 11 times
Joined: Apr 22, 2013 2:29 am
Full Name: Ben Thomas
Location: New Zealand
Contact:

Re: Increasing VMCX files in VMs that are Veeam replicated

Post by benthomas »

Mike Resseler wrote: Feb 19, 2019 6:15 am Hey Ben,

I got informed this should be fixed in update 4. Are you running the latest update already?
I'm not running Update 4 yet. It would be nice if this was actually documented somewhere, as this is a massive improvement for Hyper-V environments if it's actually fixed in Update 4.
Ben Thomas | Solutions Advisor | Veeam Vanguard 2023 | VMCE2022 | Microsoft MVP 2018-2023 | BCThomas.com
WorkingHardInIt
Veeam Vanguard
Posts: 39
Liked: 11 times
Joined: Feb 14, 2014 1:27 pm
Full Name: Didier Van Hoye
Contact:

Re: Increasing VMCX files in VMs that are Veeam replicated

Post by WorkingHardInIt »

With VBR 9.5 U4 is had to add the reg key or the ref points would still accumulate.
tomtim
Novice
Posts: 6
Liked: never
Joined: Mar 07, 2019 8:47 pm
Full Name: Thomas Timmermann
Contact:

Re: Increasing VMCX files in VMs that are Veeam replicated

Post by tomtim »

Hello, can you please tell me, which reg key I have to add, in order to prevent, that the ref points accumulate.

Thanks for your answer.

Best regards,

Thomas
Mike Resseler
Product Manager
Posts: 8044
Liked: 1263 times
Joined: Feb 08, 2013 3:08 pm
Full Name: Mike Resseler
Location: Belgium
Contact:

Re: Increasing VMCX files in VMs that are Veeam replicated

Post by Mike Resseler » 1 person likes this post

Hi Thomas,
First: Welcome to the forums!
Second: This should be the reg key:

Path: HKEY_LOCAL_MACHINE\SOFTWARE\Veeam\Veeam Backup and Replication\
Name:HyperVReferencePointCleanup

Type: REG_DWORD

Value: 1
tomtim
Novice
Posts: 6
Liked: never
Joined: Mar 07, 2019 8:47 pm
Full Name: Thomas Timmermann
Contact:

Re: Increasing VMCX files in VMs that are Veeam replicated

Post by tomtim »

Hi Mike,

thanks a lot for your answer.It works !

Great !

I have another question ? Will the old reference points be deleted automatically sometime?

Best regards,
Thomas
hasoft
Novice
Posts: 8
Liked: 1 time
Joined: Sep 29, 2018 10:19 am
Full Name: Zdenek Vasku
Contact:

Re: Increasing VMCX files in VMs that are Veeam replicated

Post by hasoft »

I think yes, they are deleted after backup or replication of virtual machine.
tomtim
Novice
Posts: 6
Liked: never
Joined: Mar 07, 2019 8:47 pm
Full Name: Thomas Timmermann
Contact:

Re: Increasing VMCX files in VMs that are Veeam replicated

Post by tomtim »

I found the following. No new reference points are created. The VMCX file does not grow anymore, but it keeps its size and does not shrink automatically. After I moved the VM to another host via live migration, the old reference points are taken from the VMCX file and the file has its original small output size again.
chuck614
Service Provider
Posts: 5
Liked: 1 time
Joined: Jun 20, 2019 9:39 pm
Full Name: Kurt
Contact:

Re: Increasing VMCX files in VMs that are Veeam replicated

Post by chuck614 »

We have tried this registry key and it does clean up reference points, but that ends up breaking any other product/service that uses reference points. For example we use Veeam backups but also Hyper-V replica. Hyper-V replica is ruined if all reference points are removed vs. just what Veeam created.
foggy
Veeam Software
Posts: 21069
Liked: 2115 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Increasing VMCX files in VMs that are Veeam replicated

Post by foggy »

Hi Kurt, I suggest asking support to take a closer look at what happens. Thanks!
chuck614
Service Provider
Posts: 5
Liked: 1 time
Joined: Jun 20, 2019 9:39 pm
Full Name: Kurt
Contact:

Re: Increasing VMCX files in VMs that are Veeam replicated

Post by chuck614 » 1 person likes this post

We got the usual response of "this is a Microsoft problem" Which is funny, since all Microsoft products are able to clean up their reference points without impacting Veeam. Not sure why Veeam can't do the same. For VMs to become unusable in live migration after Veeam backs them up enough times is a bug. Hopefully this registry key is not the permanent fix.
benthomas
Veeam Vanguard
Posts: 39
Liked: 11 times
Joined: Apr 22, 2013 2:29 am
Full Name: Ben Thomas
Location: New Zealand
Contact:

Re: Increasing VMCX files in VMs that are Veeam replicated

Post by benthomas »

I too have seen reports of this registry key fix causing issues when the VMs are also protected with Hyper-V Replica.
Is Veeam's implementation of the cleanup just removing all reference points? Rather than 'Known' veeam related reference points?
Ben Thomas | Solutions Advisor | Veeam Vanguard 2023 | VMCE2022 | Microsoft MVP 2018-2023 | BCThomas.com
nmdange
Veteran
Posts: 527
Liked: 142 times
Joined: Aug 20, 2015 9:30 pm
Contact:

Re: Increasing VMCX files in VMs that are Veeam replicated

Post by nmdange »

You can use this powershell script to cleanup Veeam reference points without breaking Hyper-V Replica
post338672.html#p338672
Post Reply

Who is online

Users browsing this forum: No registered users and 23 guests