Forum Digest: CBT bug - Snapshot revert issues

[Gostev]

Not sure I follow you here Michael. Replicas are not being backed up in the first place? So, how is this Active Full backup issue relevant at all?

[mcz]

Well, you'd replicate between source and target. If you do a failover and later a failback, it'd use CBT for transfering the deltas from the replica to the original vm (if still available). Now because a replication pass first of all reverts the current snapshot, CBT results coult be affected by the bug and now when you failback, it could mean that not all data is being transferred to the failed back vm. That's my point. Thanks!

[Gostev]

OK, got it. Perhaps I just didn't mention this here before - but according to the recent update on our DCPN support case, the issue is specific to QueryChangedDiskAreas API with changeID parameter set to *, which in simple terms limits the impact to Active Full backups (and initial replication) only.

What an irony indeed, considering that most people who do periodic Active Fulls, do them specifically because they don't trust CBT.

[backupquestions]

Is it any worse to use vmware snapshots and rollbacks right now than it is to use a "quick backup" and then "quick rollback restore" in Veeam instead? Both methods utilize built in vmware snapshot revert. This is regarding backup and restoring, not replication feature for me. I understand the current official safe side position, but I want to go by latest information you have on the case.

I don't want to change company policy unless the quick backup and quick rollback exempts you from the bug, which it doesn't seem like it should since it uses vmware's built in snapshot revert.

I do synthetic REFS fulls which is still incremental forever as far as vsphere snapshots are concerned correct? So I'm good and just won't ever take active full unless I reset CBT first, and people can be free to snap and revert and I don't need to worry about it. I think the auto CBT reset on active full is a must for next version as then I don't even have to remember to do it down the road when I need a new active full etc.

[Gostev]

a "quick backup" and then "quick rollback restore" in Veeam instead? Both methods utilize built in vmware snapshot revert

What makes you think they do?

[backupquestions]

Watching the task section in vcenter and getting the same alert from the same configured alarm for snapshot recert of vm. It looks like Veeam calls this to vcenter or esxi etc.

You are saying this is not the case and the revert is different? Or is it just solely that since the CBT is reset by Veeam in these cases, it protects you?

[Gostev]

Interesting! I guess I will need to check with devs if that is the case, as I personally don't understand why would quick rollback need to do it... seems totally unnecessary.

[Frosty]

Thanks all for the help.
I eventually got the new Alarm Definition sorted with VCSA v6.7 and have documented how here:
vmware-vsphere-f24/configure-new-alarm- ... 62086.html

[FrancWest]

Is there a way to determine for which VM's a snapshot revert has occurred in the past, so that we can check if the backup is still valid for those VM's?

[mcz]

@Anton: How much progress has been achived on vmware-side? What do you think, when could they provide an appropriate solution aka reseting CBT automatically when data is corrupted?

[Gostev]

Michael, they did not provide any timelines. The only significant update I saw was that they're not going to provide a patch for ESXi 6.0, because this version has just had its last planned patch released before it will go out of support in March 2020. So if anyone needs a solution for vSphere 6.0 before v10 is out with its auto CBT reset workaround, you will need to contact VMware support and push them for some private fix.

[Frosty]

Is there a way to determine for which VM's a snapshot revert has occurred in the past, so that we can check if the backup is still valid for those VM's?

I was able to use PowerCLI with the following command to get a list from the Event Logs in my environment:

Code: Select all

Get-VM | Sort | Get-VIEvent | Where {$_.EventTypeID -eq "com.vmware.vc.vm.VmStateRevertedToSnapshot"} | fl ObjectName, EventTypeID, CreatedTime

I doubt that this will produce a comprehensive list however, as it would be limited by how much data is kept in the logs.

[skrause]

So is the best way to do a CBT reset on running VMs still to run VMware's PowerCLI script in this KB article: https://kb.vmware.com/s/article/2139574 ?

[Frosty]

I'm a novice at this stuff, so you should probably wait for others to weigh in before deciding. But I am responsible for a pretty small environment (~60 VMs) and I didn't like the idea of resetting CBT for all the VMs all at once. I also didn't want to just Storage vMotion them to force a reset. So I rolled my own Powershell script, loosely based on others that I found online, so that I could reset CBT one VM at a time. I then worked through my backup jobs one at a time, resetting CBT for the VMs in thejob, taking a fresh Active Full, then on to the next job and so on. Worked okay for me, but as I said, small environment and I'm happy to follow one-step-at-a-time processes in order to limit my risks. I'd be happy to share my script, with all suitable disclaimers, if it was thought potentially helpful.

[jmc]

i would like to come back to the storage vmotion. when i migrate the vm to a new storage than the vmdk and the cbt is rebuilt. this means that the first veeam backup creates an active full of the vm - right? Specifically: would a storage vmotion solve the problem for the vm?

[Gostev]

Actually, no. Veeam supported vMotion since day 1, and it does not impact backups in any way - so no active fulls or anything like that. Thanks!

[mkretzer]

Its solved
https://docs.vmware.com/en/VMware-vSphe ... g-resolved

@Gostev: Is the new implementation working with Veeam? From what i understand it just deactivates CBT...

[Andreas Neufert]

That way we see an error with CBT and reset it. It is exactly how it was fixed back in vSphere 4 times and we work with this "workaround" for VMware without issues.

[Andreas Neufert]

Summary so far:
Patch for 5.x will be not available from VMware as product is "end of support" since over a year.
Patch for 6.0 available: https://docs.vmware.com/en/VMware-vSphe ... 11001.html
Patch for 6.5 outstanding with availability "soon" based on VMware support.
Patch for 6.7 available: https://docs.vmware.com/en/VMware-vSphe ... g-resolved

[msbin]

Hi,

That way we see an error with CBT and reset it. It is exactly how it was fixed back in vSphere 4 times and we work with this "workaround" for VMware without issues.

To be 100% sure so after this patch installation on ESXi host https://docs.vmware.com/en/VMware-vSphe ... g-resolved the Veeam 9.5 U4b is able to detect CBT error and will automatically reset CBT for particular VM? (no need of doing this manually as was needed before recent vSphere patch)

[Gostev]

No, there's no automatic CBT reset with the patch installed. However, CBT API will now return all blocks as changed following snapshot revert, which solves the data loss issue.

[darecki]

Does this "revert" CBT bug affect dumps made as VeeamZIP? It is never incremental, however, it says somewhere that is enabling CBT tracking whilst dumping.

[veremin]

Yep, it does as VeeamZIP relies on CBT to skip unused blocks. Thanks!

[darecki]

So for thin provision should be immune as all blocks are in use?

[Gostev]

It's not known or confirmed to be immune. A lot depends on how exactly CBT API malfunctions after VM has been reverted to a snapshot, but this information is not available to Veeam (only VMware developers know this). Thanks!

[Andreas Neufert]

Summary so far:
Patch for 5.x will be not available from VMware as product is "end of support" since over a year.
Patch for 6.0 available: https://docs.vmware.com/en/VMware-vSphe ... 11001.html
Patch for 6.5 available: https://docs.vmware.com/en/VMware-vSphe ... d-resolved
Patch for 6.7 available: https://docs.vmware.com/en/VMware-vSphe ... g-resolved

[shamel_mcc]

Do we know if this is resolved in esxi 7.x ?

[Andreas Neufert]

Newer builts that the listed one in the link contain the fix. You see the old version in there as Vmware has backported the fix to 6.5 and 6.0 to help these customers as well.

[shamel_mcc]

I didnt see 7.x referenced im assuming its fixed in 7.x since they fixed it in 6.7, i was just looking for a confirmation that this is no longer an issue to be concerned about.

[Andreas Neufert]

GA release date was 5 month after this patch release and I can not remember that we faced any issues with this in vSphere 7 at GA version.