Discussions specific to the Microsoft Hyper-V hypervisor
GregaB
Novice
Posts: 4
Liked: 1 time
Joined: Aug 27, 2018 7:03 am
Full Name: Grega Blatnik
Contact:

Increasing VMCX files in VMs that are Veeam replicated

Post by GregaB » Aug 27, 2018 7:23 am

Hi all. I was planning on opening a support case for this, but first I'm going to check if this is something that somebody else might have come across and I just can can't find it with my searches..

I have a 2 node Hyper-V cluster, both nodes are Server 2016, patched up to 2018-07 patches (didn't yet do the latest 2018-08..). I have a few VMs that I replicate to a standalone Hyper-V host very regularly (hourly) and a few that I replicate only twice a day, and I'm noticing a problem on these VMs, the configuration files for them (VMCX files) are increasing by about 100kB per day per hourly replicated VMs and like 10kB per day on twice daily replicated ones. VMs that are not replicated at all, don't show this. This has become a problem, as some VMCX are now over 20MB and these VMs now can't be successfully Live Migrated to another node, they get timed out when moving, while VMs with normal size VMCX files can be moved fine.

The weird thing is, I have another client with a similar setup, and I'm not observing the same there, the replicated VMs VMCXes there are not increasing in size..

This Veeam was at version 9.5 Update 2 at the time that I noticed this problem, since then I updated to Update 3a and it's still happening.

Anybody else noticing this and do we have a fix for this already or should I open a support case for this?

DGrinev
Veeam Software
Posts: 1529
Liked: 180 times
Joined: Dec 01, 2016 3:49 pm
Full Name: Dmitry Grinev
Location: St.Petersburg
Contact:

Re: Increasing VMCX files in VMs that are Veeam replicated

Post by DGrinev » Aug 28, 2018 1:58 pm

Hi Grega and welcome to the community!

I've discussed the said issue with the QA team, they've seen similar behavior, but it's related to the Hyper-V process, since Veeam doesn't write any information into VMCX.
You can reach the support team and ask for assistance, they should have additional information about the issue. Thanks!

GregaB
Novice
Posts: 4
Liked: 1 time
Joined: Aug 27, 2018 7:03 am
Full Name: Grega Blatnik
Contact:

Re: Increasing VMCX files in VMs that are Veeam replicated

Post by GregaB » Aug 29, 2018 7:27 am

Thank you for this, I will open a support case for this issue and hopefully we'll get this resolved. I'll make sure to post results here for anybody else experiencing the same in the future.

DGrinev
Veeam Software
Posts: 1529
Liked: 180 times
Joined: Dec 01, 2016 3:49 pm
Full Name: Dmitry Grinev
Location: St.Petersburg
Contact:

Re: Increasing VMCX files in VMs that are Veeam replicated

Post by DGrinev » Aug 29, 2018 10:19 am

Please share the case id here once it will be raised, so we can track investigation internally. Thanks!

GregaB
Novice
Posts: 4
Liked: 1 time
Joined: Aug 27, 2018 7:03 am
Full Name: Grega Blatnik
Contact:

Re: Increasing VMCX files in VMs that are Veeam replicated

Post by GregaB » Aug 29, 2018 12:50 pm 1 person likes this post

Sure, the case ID is #03168787.

stevemciow
Novice
Posts: 4
Liked: 1 time
Joined: Jun 13, 2018 9:59 am
Full Name: Stephen Courtney
Contact:

Re: Increasing VMCX files in VMs that are Veeam replicated

Post by stevemciow » Feb 14, 2019 3:47 pm

I know this is an old thread, but we're experiencing exactly the same issue - we have a number of VMs in our live environment that are replicated every 4 hours to our DR site. They have now got .VMCX files that are around 20MB and will no longer live-migrate between our Hyper-V 2016 hosts.

Our test VMs, which were copies of the live ones, have .VMCX files of around 80KB, and live migrate without issues.

Was there a fix for this found?

I'll raise a case with support as well.

GregaB
Novice
Posts: 4
Liked: 1 time
Joined: Aug 27, 2018 7:03 am
Full Name: Grega Blatnik
Contact:

Re: Increasing VMCX files in VMs that are Veeam replicated

Post by GregaB » Feb 14, 2019 6:05 pm

I replied to you in the PM already, but I'll write this here for some other poor souls having these issues..

Unfortunately, no, we still haven't found the final solution to this problem.. We also managed to rule out Veeam as a cause of this problem, as it also happens if you do lots of checkpoints manually within Hyper-V itself. We have a case open with Microsoft for this. I moved on from this project, so I might not have the latest info at hand right now, but I'll check with the person who took over from me and see where they're at now.. If there's a fix from Microsoft, I'll certainly come back with the solution here!

In the meantime, I did find a workaround for this issue, not very elegant, as it needs VMs shutting down, but it is a workaround nonetheless. You can get the VMCX files back down to normal sizes if you shut down the VMs, export them, delete the originals and import them back into Hyper-V.. Then you have full functionality back, until of course the VMCXes get too large again and you have to do the workaround again, but still, better than nothing!

benthomas
Service Provider
Posts: 15
Liked: never
Joined: Apr 22, 2013 2:29 am
Full Name: Ben Thomas
Location: New Zealand
Contact:

Re: Increasing VMCX files in VMs that are Veeam replicated

Post by benthomas » Feb 14, 2019 9:51 pm

I too am experiencing the same issue. However I wasn't able to reproduce it with normal checkpoints, only with the recovery checkpoints that backup software like Veeam uses.
I've got a Veeam support case #03414680 open as well as a Microsoft Premier support case open for this.

Mike Resseler
Product Manager
Posts: 5343
Liked: 560 times
Joined: Feb 08, 2013 3:08 pm
Full Name: Mike Resseler
Location: Belgium
Contact:

Re: Increasing VMCX files in VMs that are Veeam replicated

Post by Mike Resseler » Feb 15, 2019 6:15 am

Hi Ben,

What type of checkpoints are you running if you do it through Hyper-V? If you do this with production checkpoints, it should have the same issue.

stevemciow
Novice
Posts: 4
Liked: 1 time
Joined: Jun 13, 2018 9:59 am
Full Name: Stephen Courtney
Contact:

Re: Increasing VMCX files in VMs that are Veeam replicated

Post by stevemciow » Feb 15, 2019 1:39 pm 1 person likes this post

Apparently there is a tool that Microsoft can provide that will compact the .VMCX file - although at present they only supplied it under an NDA, so I'm trying to get them to give me a copy. If I do get one, and they're ok with it being made available, I'll update this post.

benthomas
Service Provider
Posts: 15
Liked: never
Joined: Apr 22, 2013 2:29 am
Full Name: Ben Thomas
Location: New Zealand
Contact:

Re: Increasing VMCX files in VMs that are Veeam replicated

Post by benthomas » Feb 19, 2019 2:49 am

Seems it might be the same issue as mentioned here
microsoft-hyper-v-f25/guest-os-starting ... ml#p302379

benthomas
Service Provider
Posts: 15
Liked: never
Joined: Apr 22, 2013 2:29 am
Full Name: Ben Thomas
Location: New Zealand
Contact:

Re: Increasing VMCX files in VMs that are Veeam replicated

Post by benthomas » Feb 19, 2019 2:51 am

stevemciow wrote:
Feb 15, 2019 1:39 pm
Apparently there is a tool that Microsoft can provide that will compact the .VMCX file - although at present they only supplied it under an NDA, so I'm trying to get them to give me a copy. If I do get one, and they're ok with it being made available, I'll update this post.
It shouldn't need a NDA tool, there are WMI methods that can be called to remove the leftover reference points that Veeam isn't cleaning up.

benthomas
Service Provider
Posts: 15
Liked: never
Joined: Apr 22, 2013 2:29 am
Full Name: Ben Thomas
Location: New Zealand
Contact:

Re: Increasing VMCX files in VMs that are Veeam replicated

Post by benthomas » Feb 19, 2019 2:52 am

Mike Resseler wrote:
Feb 15, 2019 6:15 am
Hi Ben,

What type of checkpoints are you running if you do it through Hyper-V? If you do this with production checkpoints, it should have the same issue.
Hi Mike,

This seems to be a fault with the implementation of Hyper-V RCT by Veeam, where by Veeam isn't cleaning up the Reference points created as part of the backup and recover process.
This needs to be addressed by the Veeam Developers, not Microsoft :)

Mike Resseler
Product Manager
Posts: 5343
Liked: 560 times
Joined: Feb 08, 2013 3:08 pm
Full Name: Mike Resseler
Location: Belgium
Contact:

Re: Increasing VMCX files in VMs that are Veeam replicated

Post by Mike Resseler » Feb 19, 2019 6:15 am

Hey Ben,

I got informed this should be fixed in update 4. Are you running the latest update already?

benthomas
Service Provider
Posts: 15
Liked: never
Joined: Apr 22, 2013 2:29 am
Full Name: Ben Thomas
Location: New Zealand
Contact:

Re: Increasing VMCX files in VMs that are Veeam replicated

Post by benthomas » Feb 19, 2019 7:04 pm

Mike Resseler wrote:
Feb 19, 2019 6:15 am
Hey Ben,

I got informed this should be fixed in update 4. Are you running the latest update already?
I'm not running Update 4 yet. It would be nice if this was actually documented somewhere, as this is a massive improvement for Hyper-V environments if it's actually fixed in Update 4.

Post Reply

Who is online

Users browsing this forum: Bing [Bot] and 10 guests