Discussions specific to the VMware vSphere hypervisor
Gostev
SVP, Product Management
Posts: 24787
Liked: 3514 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Forum Digest: CBT bug - Snapshot revert issues

Post by Gostev » Sep 11, 2019 10:52 pm

mkretzer wrote:
Sep 11, 2019 5:00 pm
@Gostev: To be honest that would be simple an genious. Please implement this and make it possible to turn it off :-)
No worries. In Veeam, there's a registry key for everything!

matthewingram
Influencer
Posts: 18
Liked: never
Joined: Oct 03, 2018 10:13 pm
Contact:

Re: Forum Digest: CBT bug - Snapshot revert issues

Post by matthewingram » Sep 12, 2019 5:01 pm

another helpful change maybe to allow us to run a active full of an individual VM within a job.

Gostev
SVP, Product Management
Posts: 24787
Liked: 3514 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Forum Digest: CBT bug - Snapshot revert issues

Post by Gostev » Sep 12, 2019 10:23 pm

This will require some quite major job architecture, which is planned - but this requires a major release runway, so not something we can add last minute to v10.

doum
Influencer
Posts: 15
Liked: 2 times
Joined: Feb 15, 2018 10:45 pm
Full Name: Benoit Machiavello
Contact:

Re: Forum Digest: CBT bug - Snapshot revert issues

Post by doum » Sep 13, 2019 7:31 am

Gostev wrote:
Sep 10, 2019 4:37 pm
By the way, based on these events we're now looking at adding one last minute feature to v10: automatic CBT reset on each VM when Active Full backup is performed. Looking back at the past few CBT bugs, having this would have helped everyone to make sure they have a good backup without having to mess with CBT reset scripts.
better than nothing but doesn't solve the problem if you don't use active full, and for all the backup between a revert to snapshot (monday for example) and an active full (sunday)

doum
Influencer
Posts: 15
Liked: 2 times
Joined: Feb 15, 2018 10:45 pm
Full Name: Benoit Machiavello
Contact:

Re: Forum Digest: CBT bug - Snapshot revert issues

Post by doum » Sep 13, 2019 7:34 am

matthewingram wrote:
Sep 12, 2019 5:01 pm
another helpful change maybe to allow us to run a active full of an individual VM within a job.
and reset the CBT before the active full :D

the best thing though would be for vmware to fix this huge bug "by design" and automatically reset the CBT when you revert a snapshot

Gostev
SVP, Product Management
Posts: 24787
Liked: 3514 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Forum Digest: CBT bug - Snapshot revert issues

Post by Gostev » Sep 13, 2019 2:15 pm

Currently, their intention is to disable CBT following snapshot revert.

Regnor
Service Provider
Posts: 318
Liked: 62 times
Joined: Jan 31, 2011 11:17 am
Full Name: Max
Contact:

Re: Forum Digest: CBT bug - Snapshot revert issues

Post by Regnor » Sep 16, 2019 6:16 am

Gostev wrote:
Sep 10, 2019 4:37 pm
By the way, based on these events we're now looking at adding one last minute feature to v10: automatic CBT reset on each VM when Active Full backup is performed. Looking back at the past few CBT bugs, having this would have helped everyone to make sure they have a good backup without having to mess with CBT reset scripts.
This sounds like a great idea; +1 ;)
I already thought of regulary disabling CBT for every customer because of all the bugs.

omerlin98000
Veeam ProPartner
Posts: 33
Liked: never
Joined: Aug 27, 2014 9:29 am
Full Name: Olivier Merlin
Location: Monte-Carlo, Monaco
Contact:

Re: Forum Digest: CBT bug - Snapshot revert issues

Post by omerlin98000 » Sep 16, 2019 6:50 am

Hello all :)

One question is still in my mind:
- what an "Active Full backup" has to do with CBT (correct or corrupted)? I thought an Active full will ignore CBT by design and has to read ALL blocks from scratch?
- if Active full is sufficient, why do I have to reset CBT by script?

Of course, I have some jobs with incremental forever, those are now suspicious to me.
And yes, if I do a revert snapshot, it will be my responsibility to force Active full for the next backup job (event it is an incremental that very day).

Do I understand well?

mkh
Service Provider
Posts: 5
Liked: never
Joined: Apr 20, 2018 6:17 am
Full Name: Michael Høyer
Contact:

Re: Forum Digest: CBT bug - Snapshot revert issues

Post by mkh » Sep 16, 2019 7:02 am

please keep the "automatic CBT reset on active full" even if vmware arrived with patches before v10. It will give so much more peace of mind.

jb238
Novice
Posts: 8
Liked: 1 time
Joined: Mar 11, 2015 11:19 am
Full Name: Joerg Baur
Contact:

Re: Forum Digest: CBT bug - Snapshot revert issues

Post by jb238 » Sep 16, 2019 8:19 am

Gostev wrote:
Sep 10, 2019 4:37 pm
By the way, based on these events we're now looking at adding one last minute feature to v10: automatic CBT reset on each VM when Active Full backup is performed. Looking back at the past few CBT bugs, having this would have helped everyone to make sure they have a good backup without having to mess with CBT reset scripts.
I would like that. Data integrity is by far the most important, speed a distant second. Maybe implement a switch to turn it off if in some environtments the performance impact is significant.

mma
Service Provider
Posts: 98
Liked: 18 times
Joined: Dec 22, 2011 9:12 am
Full Name: Marcel
Location: Lucerne, Switzerland
Contact:

Re: Forum Digest: CBT bug - Snapshot revert issues

Post by mma » Sep 16, 2019 1:50 pm

Keep in mind that some VMs might be in more than one job. CBT reset at active full is not optimal in this case.

omerlin98000
Veeam ProPartner
Posts: 33
Liked: never
Joined: Aug 27, 2014 9:29 am
Full Name: Olivier Merlin
Location: Monte-Carlo, Monaco
Contact:

Re: Forum Digest: CBT bug - Snapshot revert issues

Post by omerlin98000 » Sep 23, 2019 5:58 am

omerlin98000 wrote:
Sep 16, 2019 6:50 am
Hello all :)
One question is still in my mind:
- what an "Active Full backup" has to do with CBT (correct or corrupted)? I thought an Active full will ignore CBT by design and has to read ALL blocks from scratch?
- if Active full is sufficient, why do I have to reset CBT by script?
Of course, I have some jobs with incremental forever, those are now suspicious to me.
And yes, if I do a revert snapshot, it will be my responsibility to force Active full for the next backup job (event it is an incremental that very day).
Do I understand well?
Hello again, I have not seen any clear answer to my question, perhaps someone can explain to me if I'm wrong?

Gostev
SVP, Product Management
Posts: 24787
Liked: 3514 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Forum Digest: CBT bug - Snapshot revert issues

Post by Gostev » Sep 23, 2019 12:57 pm

The issue discussed here is with QueryChangedDiskAreas API, which is indeed used with Active Full backups to avoid having to physically read through empty VMDK blocks. Thanks!

backupquestions
Expert
Posts: 113
Liked: 13 times
Joined: Mar 13, 2019 2:30 pm
Full Name: Alabaster McJenkins
Contact:

Re: Forum Digest: CBT bug - Snapshot revert issues

Post by backupquestions » Sep 23, 2019 3:28 pm 1 person likes this post

I set up a vcenter alarm to alert me via email when any vm has been reverted. Now, my question...

If a revert happens and then an incremental Veeam backup is taken as normal, and THEN I become aware the next day and I reset CBT for the VM in vsphere.... The next incremental backup would be OK but is the chain broken now as the previous increment had garbage data possibly? Or am I OK so long as it gets reset eventually and I just have maybe 1 or 2 bad increments that happened before I could reset CBT?

Also, it sounded like as long as I never take a new active full on these vms, you could theoretically continue on as normal and the problem does not actually appear? You have to delete your backup and/or take new active full for the full thing to trigger. That and storage vmotion maybe...

Gostev
SVP, Product Management
Posts: 24787
Liked: 3514 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Forum Digest: CBT bug - Snapshot revert issues

Post by Gostev » Sep 23, 2019 5:34 pm

No one except VMware developers can tell you for sure, because only they have access to the source code and know exactly what happens with CBT data after VM has been reverted to a snapshot.

Based on the official publicly available information (VMware support KB article linked in my previous post), CBT does not support VM snapshot revert operation at all. Thus, to be on the safe side, you should assume your incremental chain is now broken, reset CBT on a VM, and do an Active Full backup.

backupquestions
Expert
Posts: 113
Liked: 13 times
Joined: Mar 13, 2019 2:30 pm
Full Name: Alabaster McJenkins
Contact:

Re: Forum Digest: CBT bug - Snapshot revert issues

Post by backupquestions » Sep 24, 2019 11:12 pm

So then how can I do an active full after cbt reset for just one vm in a job with a lot of vms in it? We are using per vm backup files but there is still no choice I can find for this. I can't just delete the old chain as I would lose a bunch of history.

Gostev
SVP, Product Management
Posts: 24787
Liked: 3514 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Forum Digest: CBT bug - Snapshot revert issues

Post by Gostev » Sep 24, 2019 11:28 pm 1 person likes this post

Unfortunately, it is not possible to do an active full on the given VM in a job currently. So I think the best solution will be to move that VM in the dedicated job, while letting "old backups" expire as a part of the original job.

backupquestions
Expert
Posts: 113
Liked: 13 times
Joined: Mar 13, 2019 2:30 pm
Full Name: Alabaster McJenkins
Contact:

Re: Forum Digest: CBT bug - Snapshot revert issues

Post by backupquestions » Sep 24, 2019 11:45 pm

What about backup copy jobs and offloaded data to object storage. Those are both incremental forever and would now contain corrupt data as well right?

I cant delete months worth of retention on those object gfs points.

The regular backup copy jobs only have a few points so I guess I can delete the vm on that if that's what I have to do and re seed it.

Gostev
SVP, Product Management
Posts: 24787
Liked: 3514 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Forum Digest: CBT bug - Snapshot revert issues

Post by Gostev » Sep 25, 2019 11:28 am

All backups made prior to reverting VM snapshot are fine, so you definitely don't have to "delete months worth of retention on those object gfs points". You can keep them, and let them expire.

stewsie
Expert
Posts: 128
Liked: 7 times
Joined: May 22, 2015 7:16 am
Full Name: Paul
Contact:

Re: Forum Digest: CBT bug - Snapshot revert issues

Post by stewsie » Sep 25, 2019 1:23 pm 1 person likes this post

backupquestions wrote:
Sep 23, 2019 3:28 pm
I set up a vcenter alarm to alert me via email when any vm has been reverted. Now, my question...

If a revert happens and then an incremental Veeam backup is taken as normal, and THEN I become aware the next day and I reset CBT for the VM in vsphere.... The next incremental backup would be OK but is the chain broken now as the previous increment had garbage data possibly? Or am I OK so long as it gets reset eventually and I just have maybe 1 or 2 bad increments that happened before I could reset CBT?

Also, it sounded like as long as I never take a new active full on these vms, you could theoretically continue on as normal and the problem does not actually appear? You have to delete your backup and/or take new active full for the full thing to trigger. That and storage vmotion maybe...
What alert did you configure in vCenter? I cannot find anything to configure.

Thanks

backupquestions
Expert
Posts: 113
Liked: 13 times
Joined: Mar 13, 2019 2:30 pm
Full Name: Alabaster McJenkins
Contact:

Re: Forum Digest: CBT bug - Snapshot revert issues

Post by backupquestions » Sep 25, 2019 2:08 pm

It was not my idea so I will post the link where I learned how. Pretty easy to do. Make sure to set the alarm at the top level (vcenter object). https://www.virtual-allan.com/cbt-bug-o ... -snapshot/

But yeah, this is really unfortunate because I thought of moving my chain elsewhere but then it would expand all the space since REFS block clone is lost. Also I think you pretty much need to delete your chain from your VCC backup copy job unless you are not using incremental forever, (and most probably are since that is default). (as the bad increment that gets copied out there will eventually be merged right into the vbk since chain transform) The only good thing for me is I am using per-vm backup files so I can just delete that chain alone on the VCC backup copy job.

Frosty
Expert
Posts: 174
Liked: 36 times
Joined: Dec 22, 2009 9:00 pm
Full Name: Stephen Frost
Contact:

Re: Forum Digest: CBT bug - Snapshot revert issues

Post by Frosty » Sep 26, 2019 3:51 am

I would like to configure an Alarm for exactly this situation. We're running vCenter/vSphere 6.7 here, however in the web interface I cannot find how to create an alarm that targets the com.vmware.vc.vm.VmStateRevertedToSnapshot event. Can anyone shed any light on this? Maybe I am just not looking in the right place.

stewsie
Expert
Posts: 128
Liked: 7 times
Joined: May 22, 2015 7:16 am
Full Name: Paul
Contact:

Re: Forum Digest: CBT bug - Snapshot revert issues

Post by stewsie » Sep 26, 2019 7:49 am

It is not easy how this alarm needs to be created.

I did the following in vCenter 6

Identify the object where the alarm is to be configured
Click Manage
Click +
Choose VM and Specific event
Click Next
Click +
There is no option in the drop down so I cleared the text and pasted com.vmware.vc.vm.VmStateRevertedToSnapshot
It took a couple of tries to add this
Complete the creation
I can now see the alarm

Next step is to test it

*** Update *** I have just tested this and the alarm has worked

stewsie
Expert
Posts: 128
Liked: 7 times
Joined: May 22, 2015 7:16 am
Full Name: Paul
Contact:

Re: Forum Digest: CBT bug - Snapshot revert issues

Post by stewsie » Sep 26, 2019 8:02 am

Frosty wrote:
Sep 26, 2019 3:51 am
I would like to configure an Alarm for exactly this situation. We're running vCenter/vSphere 6.7 here, however in the web interface I cannot find how to create an alarm that targets the com.vmware.vc.vm.VmStateRevertedToSnapshot event. Can anyone shed any light on this? Maybe I am just not looking in the right place.
Hopefully you have seen my latest update

mkretzer
Expert
Posts: 553
Liked: 124 times
Joined: Dec 17, 2015 7:17 am
Contact:

Re: Forum Digest: CBT bug - Snapshot revert issues

Post by mkretzer » Sep 26, 2019 9:37 am 1 person likes this post

@stewsie its working, thank you very much!!

mkretzer
Expert
Posts: 553
Liked: 124 times
Joined: Dec 17, 2015 7:17 am
Contact:

Re: Forum Digest: CBT bug - Snapshot revert issues

Post by mkretzer » Sep 26, 2019 10:06 am

Someone asked it before but i think there has been no answer: What about quick rollback? Will it cause corrupted backups on the next backup take?

backupquestions
Expert
Posts: 113
Liked: 13 times
Joined: Mar 13, 2019 2:30 pm
Full Name: Alabaster McJenkins
Contact:

Re: Forum Digest: CBT bug - Snapshot revert issues

Post by backupquestions » Sep 26, 2019 4:10 pm

Gostev or someone at Veeam can confirm, but if you read here https://helpcenter.veeam.com/docs/backu ... l?ver=95u4 in the notes at the bottom, it says that CBT is reset after a quick rollback is used. (that is also why you can only perform a quick rollback once, until you have taken another backup after doing so) Here is another link which is older but likely still applies and talks about how you can only do it once and CBT is reset. https://www.veeam.com/blog/8-gems-in-ve ... lback.html

So it seems like this should be totally safe option and just never use snapshots anymore.

mkretzer
Expert
Posts: 553
Liked: 124 times
Joined: Dec 17, 2015 7:17 am
Contact:

Re: Forum Digest: CBT bug - Snapshot revert issues

Post by mkretzer » Sep 26, 2019 5:16 pm

Nice.
@Gostev: Can you confirm?

stewsie
Expert
Posts: 128
Liked: 7 times
Joined: May 22, 2015 7:16 am
Full Name: Paul
Contact:

Re: Forum Digest: CBT bug - Snapshot revert issues

Post by stewsie » Sep 27, 2019 6:25 am

Now I have the alarm configured, overnight I have received a few emails regarding Veeam replicas. I hadn't looked in detail before but it looks like as part of the process the snapshot is reverted a number of times. Interesting behaviour.

If I need to use the replicas, one of the first things I would need to do is reset CBT.

mcz
Expert
Posts: 295
Liked: 53 times
Joined: Jul 19, 2016 8:39 am
Full Name: Michael
Contact:

Re: Forum Digest: CBT bug - Snapshot revert issues

Post by mcz » Sep 27, 2019 7:04 am

@Anton Gostev: Can you please clarify the situation with replicas? I mean veeam reverts snapshots by design (and I think it's the right thing to do). So what happens in case of a failback (talking about possible corruption), should we be considered or not?

Thanks!

Post Reply

Who is online

Users browsing this forum: Bing [Bot], Google [Bot] and 29 guests