Discussions specific to the VMware vSphere hypervisor
Gostev
SVP, Product Management
Posts: 26478
Liked: 4148 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Forum Digest: CBT bug - Snapshot revert issues

Post by Gostev »

Not sure I follow you here Michael. Replicas are not being backed up in the first place? So, how is this Active Full backup issue relevant at all?

mcz
Expert
Posts: 393
Liked: 66 times
Joined: Jul 19, 2016 8:39 am
Full Name: Michael
Contact:

Re: Forum Digest: CBT bug - Snapshot revert issues

Post by mcz »

Well, you'd replicate between source and target. If you do a failover and later a failback, it'd use CBT for transfering the deltas from the replica to the original vm (if still available). Now because a replication pass first of all reverts the current snapshot, CBT results coult be affected by the bug and now when you failback, it could mean that not all data is being transferred to the failed back vm. That's my point. Thanks!

Gostev
SVP, Product Management
Posts: 26478
Liked: 4148 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Forum Digest: CBT bug - Snapshot revert issues

Post by Gostev » 3 people like this post

OK, got it. Perhaps I just didn't mention this here before - but according to the recent update on our DCPN support case, the issue is specific to QueryChangedDiskAreas API with changeID parameter set to *, which in simple terms limits the impact to Active Full backups (and initial replication) only.

What an irony indeed, considering that most people who do periodic Active Fulls, do them specifically because they don't trust CBT.

backupquestions
Expert
Posts: 181
Liked: 20 times
Joined: Mar 13, 2019 2:30 pm
Full Name: Alabaster McJenkins
Contact:

Re: Forum Digest: CBT bug - Snapshot revert issues

Post by backupquestions »

Is it any worse to use vmware snapshots and rollbacks right now than it is to use a "quick backup" and then "quick rollback restore" in Veeam instead? Both methods utilize built in vmware snapshot revert. This is regarding backup and restoring, not replication feature for me. I understand the current official safe side position, but I want to go by latest information you have on the case.

I don't want to change company policy unless the quick backup and quick rollback exempts you from the bug, which it doesn't seem like it should since it uses vmware's built in snapshot revert.

I do synthetic REFS fulls which is still incremental forever as far as vsphere snapshots are concerned correct? So I'm good and just won't ever take active full unless I reset CBT first, and people can be free to snap and revert and I don't need to worry about it. I think the auto CBT reset on active full is a must for next version as then I don't even have to remember to do it down the road when I need a new active full etc.

Gostev
SVP, Product Management
Posts: 26478
Liked: 4148 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Forum Digest: CBT bug - Snapshot revert issues

Post by Gostev »

backupquestions wrote:
Sep 27, 2019 5:33 pm
a "quick backup" and then "quick rollback restore" in Veeam instead? Both methods utilize built in vmware snapshot revert
What makes you think they do?

backupquestions
Expert
Posts: 181
Liked: 20 times
Joined: Mar 13, 2019 2:30 pm
Full Name: Alabaster McJenkins
Contact:

Re: Forum Digest: CBT bug - Snapshot revert issues

Post by backupquestions » 1 person likes this post

Watching the task section in vcenter and getting the same alert from the same configured alarm for snapshot recert of vm. It looks like Veeam calls this to vcenter or esxi etc.

You are saying this is not the case and the revert is different? Or is it just solely that since the CBT is reset by Veeam in these cases, it protects you?

Gostev
SVP, Product Management
Posts: 26478
Liked: 4148 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Forum Digest: CBT bug - Snapshot revert issues

Post by Gostev »

Interesting! I guess I will need to check with devs if that is the case, as I personally don't understand why would quick rollback need to do it... seems totally unnecessary.

Frosty
Expert
Posts: 183
Liked: 42 times
Joined: Dec 22, 2009 9:00 pm
Full Name: Stephen Frost
Contact:

Re: Forum Digest: CBT bug - Snapshot revert issues

Post by Frosty »

Thanks all for the help.
I eventually got the new Alarm Definition sorted with VCSA v6.7 and have documented how here:
vmware-vsphere-f24/configure-new-alarm- ... 62086.html

FrancWest
Expert
Posts: 205
Liked: 20 times
Joined: Sep 17, 2017 3:20 am
Full Name: Franc
Contact:

Re: Forum Digest: CBT bug - Snapshot revert issues

Post by FrancWest »

Is there a way to determine for which VM's a snapshot revert has occurred in the past, so that we can check if the backup is still valid for those VM's?

mcz
Expert
Posts: 393
Liked: 66 times
Joined: Jul 19, 2016 8:39 am
Full Name: Michael
Contact:

Re: Forum Digest: CBT bug - Snapshot revert issues

Post by mcz »

@Anton: How much progress has been achived on vmware-side? What do you think, when could they provide an appropriate solution aka reseting CBT automatically when data is corrupted?

Gostev
SVP, Product Management
Posts: 26478
Liked: 4148 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Forum Digest: CBT bug - Snapshot revert issues

Post by Gostev »

Michael, they did not provide any timelines. The only significant update I saw was that they're not going to provide a patch for ESXi 6.0, because this version has just had its last planned patch released before it will go out of support in March 2020. So if anyone needs a solution for vSphere 6.0 before v10 is out with its auto CBT reset workaround, you will need to contact VMware support and push them for some private fix.

Frosty
Expert
Posts: 183
Liked: 42 times
Joined: Dec 22, 2009 9:00 pm
Full Name: Stephen Frost
Contact:

Re: Forum Digest: CBT bug - Snapshot revert issues

Post by Frosty »

FrancWest wrote:
Sep 30, 2019 9:56 am
Is there a way to determine for which VM's a snapshot revert has occurred in the past, so that we can check if the backup is still valid for those VM's?
I was able to use PowerCLI with the following command to get a list from the Event Logs in my environment:

Code: Select all

Get-VM | Sort | Get-VIEvent | Where {$_.EventTypeID -eq "com.vmware.vc.vm.VmStateRevertedToSnapshot"} | fl ObjectName, EventTypeID, CreatedTime
I doubt that this will produce a comprehensive list however, as it would be limited by how much data is kept in the logs.

skrause
Expert
Posts: 450
Liked: 93 times
Joined: Dec 08, 2014 2:58 pm
Full Name: Steve Krause
Contact:

Re: Forum Digest: CBT bug - Snapshot revert issues

Post by skrause »

So is the best way to do a CBT reset on running VMs still to run VMware's PowerCLI script in this KB article: https://kb.vmware.com/s/article/2139574 ?
Steve Krause
Veeam Certified Architect

Frosty
Expert
Posts: 183
Liked: 42 times
Joined: Dec 22, 2009 9:00 pm
Full Name: Stephen Frost
Contact:

Re: Forum Digest: CBT bug - Snapshot revert issues

Post by Frosty »

I'm a novice at this stuff, so you should probably wait for others to weigh in before deciding. But I am responsible for a pretty small environment (~60 VMs) and I didn't like the idea of resetting CBT for all the VMs all at once. I also didn't want to just Storage vMotion them to force a reset. So I rolled my own Powershell script, loosely based on others that I found online, so that I could reset CBT one VM at a time. I then worked through my backup jobs one at a time, resetting CBT for the VMs in thejob, taking a fresh Active Full, then on to the next job and so on. Worked okay for me, but as I said, small environment and I'm happy to follow one-step-at-a-time processes in order to limit my risks. I'd be happy to share my script, with all suitable disclaimers, if it was thought potentially helpful.

jmc
Service Provider
Posts: 41
Liked: 1 time
Joined: Sep 12, 2011 11:49 am
Full Name: jmc
Location: Duisburg - Germany
Contact:

Re: Forum Digest: CBT bug - Snapshot revert issues

Post by jmc »

i would like to come back to the storage vmotion. when i migrate the vm to a new storage than the vmdk and the cbt is rebuilt. this means that the first veeam backup creates an active full of the vm - right? Specifically: would a storage vmotion solve the problem for the vm?
Everybody ask why the dinosaurs are gone - nobody ask why they are lived so long

Gostev
SVP, Product Management
Posts: 26478
Liked: 4148 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Forum Digest: CBT bug - Snapshot revert issues

Post by Gostev »

Actually, no. Veeam supported vMotion since day 1, and it does not impact backups in any way - so no active fulls or anything like that. Thanks!

mkretzer
Expert
Posts: 667
Liked: 153 times
Joined: Dec 17, 2015 7:17 am
Contact:

Re: Forum Digest: CBT bug - Snapshot revert issues

Post by mkretzer »

Its solved :-)
https://docs.vmware.com/en/VMware-vSphe ... g-resolved

@Gostev: Is the new implementation working with Veeam? From what i understand it just deactivates CBT...

Andreas Neufert
VP, Product Management
Posts: 4352
Liked: 814 times
Joined: May 04, 2011 8:36 am
Full Name: Andreas Neufert
Location: Germany
Contact:

Re: Forum Digest: CBT bug - Snapshot revert issues

Post by Andreas Neufert »

That way we see an error with CBT and reset it. It is exactly how it was fixed back in vSphere 4 times and we work with this "workaround" for VMware without issues.

Andreas Neufert
VP, Product Management
Posts: 4352
Liked: 814 times
Joined: May 04, 2011 8:36 am
Full Name: Andreas Neufert
Location: Germany
Contact:

Re: Forum Digest: CBT bug - Snapshot revert issues

Post by Andreas Neufert »

Summary so far:
Patch for 5.x will be not available from VMware as product is "end of support" since over a year.
Patch for 6.0 available: https://docs.vmware.com/en/VMware-vSphe ... 11001.html
Patch for 6.5 outstanding with availability "soon" based on VMware support.
Patch for 6.7 available: https://docs.vmware.com/en/VMware-vSphe ... g-resolved

msbin
Lurker
Posts: 2
Liked: never
Joined: Dec 12, 2019 6:59 pm
Contact:

Re: Forum Digest: CBT bug - Snapshot revert issues

Post by msbin »

Hi,
Andreas Neufert wrote:
Dec 06, 2019 8:52 am
That way we see an error with CBT and reset it. It is exactly how it was fixed back in vSphere 4 times and we work with this "workaround" for VMware without issues.
To be 100% sure ;) so after this patch installation on ESXi host https://docs.vmware.com/en/VMware-vSphe ... g-resolved the Veeam 9.5 U4b is able to detect CBT error and will automatically reset CBT for particular VM? (no need of doing this manually as was needed before recent vSphere patch)

Gostev
SVP, Product Management
Posts: 26478
Liked: 4148 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Forum Digest: CBT bug - Snapshot revert issues

Post by Gostev » 2 people like this post

No, there's no automatic CBT reset with the patch installed. However, CBT API will now return all blocks as changed following snapshot revert, which solves the data loss issue.

darecki
Influencer
Posts: 10
Liked: never
Joined: Jul 20, 2018 5:15 am
Full Name: Darek Margas
Contact:

Re: Forum Digest: CBT bug - Snapshot revert issues

Post by darecki »

Does this "revert" CBT bug affect dumps made as VeeamZIP? It is never incremental, however, it says somewhere that is enabling CBT tracking whilst dumping.

veremin
Product Manager
Posts: 17777
Liked: 1626 times
Joined: Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin
Contact:

Re: Forum Digest: CBT bug - Snapshot revert issues

Post by veremin »

Yep, it does as VeeamZIP relies on CBT to skip unused blocks. Thanks!

darecki
Influencer
Posts: 10
Liked: never
Joined: Jul 20, 2018 5:15 am
Full Name: Darek Margas
Contact:

Re: Forum Digest: CBT bug - Snapshot revert issues

Post by darecki »

So for thin provision should be immune as all blocks are in use?

Gostev
SVP, Product Management
Posts: 26478
Liked: 4148 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Forum Digest: CBT bug - Snapshot revert issues

Post by Gostev »

It's not known or confirmed to be immune. A lot depends on how exactly CBT API malfunctions after VM has been reverted to a snapshot, but this information is not available to Veeam (only VMware developers know this). Thanks!

Andreas Neufert
VP, Product Management
Posts: 4352
Liked: 814 times
Joined: May 04, 2011 8:36 am
Full Name: Andreas Neufert
Location: Germany
Contact:

Re: Forum Digest: CBT bug - Snapshot revert issues

Post by Andreas Neufert » 2 people like this post

Summary so far:
Patch for 5.x will be not available from VMware as product is "end of support" since over a year.
Patch for 6.0 available: https://docs.vmware.com/en/VMware-vSphe ... 11001.html
Patch for 6.5 available: https://docs.vmware.com/en/VMware-vSphe ... d-resolved
Patch for 6.7 available: https://docs.vmware.com/en/VMware-vSphe ... g-resolved

Post Reply

Who is online

Users browsing this forum: NerfHerder and 34 guests