VM Stun on Active Full (CBT Reset)

[chrislove]

Hi has anyone else seen this issue.

On a powered on VM with existing snapshots if you do an Active Full which resets CBT it causes the VM to stun.

Veeam is successful in resetting CBT but vSphere (because of the existing snapshot/s) is still running a process on the VM which basically stuns/locks it.In some cases we have seen VM's unresponsive for for nearly an hour.

VMware support are basically saying...

"VMware does not recommend toggling CBT for a VM with pre-existing snapshots. VMware has not articulated the exclusive list of unsupported/unintended behavior in it's documentation for not recommended/unsupported configurations. Toggling CBT on a VM with pre-existing snapshots will take longer as it needs to iterate through each of the delta files."

While we are aware pre-existing snapshots could cause CBT corruption this is new to us and as it appears to be undocumented (according to VMware) we are wondering if anyone else has seen this as it might actually be something to do with our environment. Veeam support are also unaware of this problem.

Thanks.

[PetrM]

Hi Chris,

At first sight, it does not look like something that can be fixed on our side as Veeam just sends a request to reset CBT and the process itself is managed at the vSphere level. However, I leave the opportunity to confirm it with our engineers because they can have access to all necessary logs. I can say that I encountered maybe a few times the similar problems but if you want to reveal the root cause, you should ask VMware support to analyze your environment deeper. On the other hand, I'm wondering why you keep snapshots permanently on the VM?

Thanks!

[Andreas Neufert]

To add some background information.
Veeam has a massive amount of customers that use our software with vSphere and based on support statistics we try to stabilize the product as much as we can for our customers. Part of this is in reducing risks.
Overall it is a best practice to not keep snapshots of VMs as it has a very negative performance effect when you leave them open over a longer time. Specifically when you remove them later. This is by design of VMware.
When Veeam performs an active full, then old CBT references are not needed anymore and it would make sense to reset the CBT as it does not "cost" any overhead but would ensures that any possible CBT issue is addressed (new CBT chains and references). I think there is a way to disable this by asking our support.

[einhirn]

We had this kind of issue a few days ago - in our case only one VM was hit. Deactivating (and the pursuing Snapshot+Consolidation) took about 20sec overall, but it was stunned about 20min while re-enabling CBT. It had a pre-existing Snapshot about 1.3TB in size (accumulated over 2 days, so not really long time). We checked storage and networking but the issue doesn't seem to lie there.

[lunor]

We've expierenced the same issue.
A VM with a prexisting Snaphot (4d old, ~20GB in size) was stunned for about 2Mins during the Snapshot creation.

[chrislove]

We didn't get much else from VMware at the time and it is clearly a problem that side. They did say they don't recommend doing a CBT reset on a VM with existing snapshots. This option in Veeam was a workaround I think to VMware CBT issues so would be handy if Veeam could add another to detect pre-existing snaps before a reset but sounds like this just hasn't been a widespread issue.

[Sturniolo]

Hey guys,

My suggestion.. if you haven't done this already.. would be to open up a support case to properly troubleshoot the issue. There are a lot of factors/variable in play when it comes to your environment. i.e. Storage types, etc...