Host-based backup of VMware vSphere VMs.
Gostev
Chief Product Officer
Posts: 31457
Liked: 6648 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Forum Digest: CBT bug - Snapshot revert issues

Post by Gostev »

mkretzer wrote: Sep 11, 2019 5:00 pm@Gostev: To be honest that would be simple an genious. Please implement this and make it possible to turn it off :-)
No worries. In Veeam, there's a registry key for everything!
matthewingram
Influencer
Posts: 20
Liked: never
Joined: Oct 03, 2018 10:13 pm
Contact:

Re: Forum Digest: CBT bug - Snapshot revert issues

Post by matthewingram »

another helpful change maybe to allow us to run a active full of an individual VM within a job.
Gostev
Chief Product Officer
Posts: 31457
Liked: 6648 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Forum Digest: CBT bug - Snapshot revert issues

Post by Gostev »

This will require some quite major job architecture, which is planned - but this requires a major release runway, so not something we can add last minute to v10.
doum
Enthusiast
Posts: 30
Liked: 6 times
Joined: Feb 15, 2018 10:45 pm
Full Name: Benoit Machiavello
Contact:

Re: Forum Digest: CBT bug - Snapshot revert issues

Post by doum »

Gostev wrote: Sep 10, 2019 4:37 pm By the way, based on these events we're now looking at adding one last minute feature to v10: automatic CBT reset on each VM when Active Full backup is performed. Looking back at the past few CBT bugs, having this would have helped everyone to make sure they have a good backup without having to mess with CBT reset scripts.
better than nothing but doesn't solve the problem if you don't use active full, and for all the backup between a revert to snapshot (monday for example) and an active full (sunday)
doum
Enthusiast
Posts: 30
Liked: 6 times
Joined: Feb 15, 2018 10:45 pm
Full Name: Benoit Machiavello
Contact:

Re: Forum Digest: CBT bug - Snapshot revert issues

Post by doum »

matthewingram wrote: Sep 12, 2019 5:01 pm another helpful change maybe to allow us to run a active full of an individual VM within a job.
and reset the CBT before the active full :D

the best thing though would be for vmware to fix this huge bug "by design" and automatically reset the CBT when you revert a snapshot
Gostev
Chief Product Officer
Posts: 31457
Liked: 6648 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Forum Digest: CBT bug - Snapshot revert issues

Post by Gostev »

Currently, their intention is to disable CBT following snapshot revert.
Regnor
Veeam Software
Posts: 934
Liked: 287 times
Joined: Jan 31, 2011 11:17 am
Full Name: Max
Contact:

Re: Forum Digest: CBT bug - Snapshot revert issues

Post by Regnor »

Gostev wrote: Sep 10, 2019 4:37 pm By the way, based on these events we're now looking at adding one last minute feature to v10: automatic CBT reset on each VM when Active Full backup is performed. Looking back at the past few CBT bugs, having this would have helped everyone to make sure they have a good backup without having to mess with CBT reset scripts.
This sounds like a great idea; +1 ;)
I already thought of regulary disabling CBT for every customer because of all the bugs.
omerlin98000
Veeam ProPartner
Posts: 33
Liked: never
Joined: Aug 27, 2014 9:29 am
Full Name: Olivier Merlin
Location: Monte-Carlo, Monaco
Contact:

Re: Forum Digest: CBT bug - Snapshot revert issues

Post by omerlin98000 »

Hello all :)

One question is still in my mind:
- what an "Active Full backup" has to do with CBT (correct or corrupted)? I thought an Active full will ignore CBT by design and has to read ALL blocks from scratch?
- if Active full is sufficient, why do I have to reset CBT by script?

Of course, I have some jobs with incremental forever, those are now suspicious to me.
And yes, if I do a revert snapshot, it will be my responsibility to force Active full for the next backup job (event it is an incremental that very day).

Do I understand well?
mkh
Service Provider
Posts: 64
Liked: 18 times
Joined: Apr 20, 2018 6:17 am
Full Name: Michael Høyer
Contact:

Re: Forum Digest: CBT bug - Snapshot revert issues

Post by mkh »

please keep the "automatic CBT reset on active full" even if vmware arrived with patches before v10. It will give so much more peace of mind.
jb238
Influencer
Posts: 10
Liked: 1 time
Joined: Mar 11, 2015 11:19 am
Full Name: Joerg Baur
Contact:

Re: Forum Digest: CBT bug - Snapshot revert issues

Post by jb238 »

Gostev wrote: Sep 10, 2019 4:37 pm By the way, based on these events we're now looking at adding one last minute feature to v10: automatic CBT reset on each VM when Active Full backup is performed. Looking back at the past few CBT bugs, having this would have helped everyone to make sure they have a good backup without having to mess with CBT reset scripts.
I would like that. Data integrity is by far the most important, speed a distant second. Maybe implement a switch to turn it off if in some environtments the performance impact is significant.
mma
Service Provider
Posts: 111
Liked: 21 times
Joined: Dec 22, 2011 9:12 am
Full Name: Marcel
Location: Lucerne, Switzerland
Contact:

Re: Forum Digest: CBT bug - Snapshot revert issues

Post by mma »

Keep in mind that some VMs might be in more than one job. CBT reset at active full is not optimal in this case.
omerlin98000
Veeam ProPartner
Posts: 33
Liked: never
Joined: Aug 27, 2014 9:29 am
Full Name: Olivier Merlin
Location: Monte-Carlo, Monaco
Contact:

Re: Forum Digest: CBT bug - Snapshot revert issues

Post by omerlin98000 »

omerlin98000 wrote: Sep 16, 2019 6:50 am Hello all :)
One question is still in my mind:
- what an "Active Full backup" has to do with CBT (correct or corrupted)? I thought an Active full will ignore CBT by design and has to read ALL blocks from scratch?
- if Active full is sufficient, why do I have to reset CBT by script?
Of course, I have some jobs with incremental forever, those are now suspicious to me.
And yes, if I do a revert snapshot, it will be my responsibility to force Active full for the next backup job (event it is an incremental that very day).
Do I understand well?
Hello again, I have not seen any clear answer to my question, perhaps someone can explain to me if I'm wrong?
Gostev
Chief Product Officer
Posts: 31457
Liked: 6648 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Forum Digest: CBT bug - Snapshot revert issues

Post by Gostev »

The issue discussed here is with QueryChangedDiskAreas API, which is indeed used with Active Full backups to avoid having to physically read through empty VMDK blocks. Thanks!
backupquestions
Expert
Posts: 186
Liked: 21 times
Joined: Mar 13, 2019 2:30 pm
Full Name: Alabaster McJenkins
Contact:

Re: Forum Digest: CBT bug - Snapshot revert issues

Post by backupquestions » 1 person likes this post

I set up a vcenter alarm to alert me via email when any vm has been reverted. Now, my question...

If a revert happens and then an incremental Veeam backup is taken as normal, and THEN I become aware the next day and I reset CBT for the VM in vsphere.... The next incremental backup would be OK but is the chain broken now as the previous increment had garbage data possibly? Or am I OK so long as it gets reset eventually and I just have maybe 1 or 2 bad increments that happened before I could reset CBT?

Also, it sounded like as long as I never take a new active full on these vms, you could theoretically continue on as normal and the problem does not actually appear? You have to delete your backup and/or take new active full for the full thing to trigger. That and storage vmotion maybe...
Gostev
Chief Product Officer
Posts: 31457
Liked: 6648 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Forum Digest: CBT bug - Snapshot revert issues

Post by Gostev »

No one except VMware developers can tell you for sure, because only they have access to the source code and know exactly what happens with CBT data after VM has been reverted to a snapshot.

Based on the official publicly available information (VMware support KB article linked in my previous post), CBT does not support VM snapshot revert operation at all. Thus, to be on the safe side, you should assume your incremental chain is now broken, reset CBT on a VM, and do an Active Full backup.
backupquestions
Expert
Posts: 186
Liked: 21 times
Joined: Mar 13, 2019 2:30 pm
Full Name: Alabaster McJenkins
Contact:

Re: Forum Digest: CBT bug - Snapshot revert issues

Post by backupquestions »

So then how can I do an active full after cbt reset for just one vm in a job with a lot of vms in it? We are using per vm backup files but there is still no choice I can find for this. I can't just delete the old chain as I would lose a bunch of history.
Gostev
Chief Product Officer
Posts: 31457
Liked: 6648 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Forum Digest: CBT bug - Snapshot revert issues

Post by Gostev » 1 person likes this post

Unfortunately, it is not possible to do an active full on the given VM in a job currently. So I think the best solution will be to move that VM in the dedicated job, while letting "old backups" expire as a part of the original job.
backupquestions
Expert
Posts: 186
Liked: 21 times
Joined: Mar 13, 2019 2:30 pm
Full Name: Alabaster McJenkins
Contact:

Re: Forum Digest: CBT bug - Snapshot revert issues

Post by backupquestions »

What about backup copy jobs and offloaded data to object storage. Those are both incremental forever and would now contain corrupt data as well right?

I cant delete months worth of retention on those object gfs points.

The regular backup copy jobs only have a few points so I guess I can delete the vm on that if that's what I have to do and re seed it.
Gostev
Chief Product Officer
Posts: 31457
Liked: 6648 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Forum Digest: CBT bug - Snapshot revert issues

Post by Gostev »

All backups made prior to reverting VM snapshot are fine, so you definitely don't have to "delete months worth of retention on those object gfs points". You can keep them, and let them expire.
stewsie
Expert
Posts: 247
Liked: 20 times
Joined: May 22, 2015 7:16 am
Full Name: Paul
Contact:

Re: Forum Digest: CBT bug - Snapshot revert issues

Post by stewsie » 1 person likes this post

backupquestions wrote: Sep 23, 2019 3:28 pm I set up a vcenter alarm to alert me via email when any vm has been reverted. Now, my question...

If a revert happens and then an incremental Veeam backup is taken as normal, and THEN I become aware the next day and I reset CBT for the VM in vsphere.... The next incremental backup would be OK but is the chain broken now as the previous increment had garbage data possibly? Or am I OK so long as it gets reset eventually and I just have maybe 1 or 2 bad increments that happened before I could reset CBT?

Also, it sounded like as long as I never take a new active full on these vms, you could theoretically continue on as normal and the problem does not actually appear? You have to delete your backup and/or take new active full for the full thing to trigger. That and storage vmotion maybe...
What alert did you configure in vCenter? I cannot find anything to configure.

Thanks
backupquestions
Expert
Posts: 186
Liked: 21 times
Joined: Mar 13, 2019 2:30 pm
Full Name: Alabaster McJenkins
Contact:

Re: Forum Digest: CBT bug - Snapshot revert issues

Post by backupquestions »

It was not my idea so I will post the link where I learned how. Pretty easy to do. Make sure to set the alarm at the top level (vcenter object). https://www.virtual-allan.com/cbt-bug-o ... -snapshot/

But yeah, this is really unfortunate because I thought of moving my chain elsewhere but then it would expand all the space since REFS block clone is lost. Also I think you pretty much need to delete your chain from your VCC backup copy job unless you are not using incremental forever, (and most probably are since that is default). (as the bad increment that gets copied out there will eventually be merged right into the vbk since chain transform) The only good thing for me is I am using per-vm backup files so I can just delete that chain alone on the VCC backup copy job.
Frosty
Expert
Posts: 200
Liked: 43 times
Joined: Dec 22, 2009 9:00 pm
Full Name: Stephen Frost
Contact:

Re: Forum Digest: CBT bug - Snapshot revert issues

Post by Frosty »

I would like to configure an Alarm for exactly this situation. We're running vCenter/vSphere 6.7 here, however in the web interface I cannot find how to create an alarm that targets the com.vmware.vc.vm.VmStateRevertedToSnapshot event. Can anyone shed any light on this? Maybe I am just not looking in the right place.
stewsie
Expert
Posts: 247
Liked: 20 times
Joined: May 22, 2015 7:16 am
Full Name: Paul
Contact:

Re: Forum Digest: CBT bug - Snapshot revert issues

Post by stewsie »

It is not easy how this alarm needs to be created.

I did the following in vCenter 6

Identify the object where the alarm is to be configured
Click Manage
Click +
Choose VM and Specific event
Click Next
Click +
There is no option in the drop down so I cleared the text and pasted com.vmware.vc.vm.VmStateRevertedToSnapshot
It took a couple of tries to add this
Complete the creation
I can now see the alarm

Next step is to test it

*** Update *** I have just tested this and the alarm has worked
stewsie
Expert
Posts: 247
Liked: 20 times
Joined: May 22, 2015 7:16 am
Full Name: Paul
Contact:

Re: Forum Digest: CBT bug - Snapshot revert issues

Post by stewsie »

Frosty wrote: Sep 26, 2019 3:51 am I would like to configure an Alarm for exactly this situation. We're running vCenter/vSphere 6.7 here, however in the web interface I cannot find how to create an alarm that targets the com.vmware.vc.vm.VmStateRevertedToSnapshot event. Can anyone shed any light on this? Maybe I am just not looking in the right place.
Hopefully you have seen my latest update
mkretzer
Veeam Legend
Posts: 1140
Liked: 387 times
Joined: Dec 17, 2015 7:17 am
Contact:

Re: Forum Digest: CBT bug - Snapshot revert issues

Post by mkretzer » 1 person likes this post

@stewsie its working, thank you very much!!
mkretzer
Veeam Legend
Posts: 1140
Liked: 387 times
Joined: Dec 17, 2015 7:17 am
Contact:

Re: Forum Digest: CBT bug - Snapshot revert issues

Post by mkretzer »

Someone asked it before but i think there has been no answer: What about quick rollback? Will it cause corrupted backups on the next backup take?
backupquestions
Expert
Posts: 186
Liked: 21 times
Joined: Mar 13, 2019 2:30 pm
Full Name: Alabaster McJenkins
Contact:

Re: Forum Digest: CBT bug - Snapshot revert issues

Post by backupquestions »

Gostev or someone at Veeam can confirm, but if you read here https://helpcenter.veeam.com/docs/backu ... l?ver=95u4 in the notes at the bottom, it says that CBT is reset after a quick rollback is used. (that is also why you can only perform a quick rollback once, until you have taken another backup after doing so) Here is another link which is older but likely still applies and talks about how you can only do it once and CBT is reset. https://www.veeam.com/blog/8-gems-in-ve ... lback.html

So it seems like this should be totally safe option and just never use snapshots anymore.
mkretzer
Veeam Legend
Posts: 1140
Liked: 387 times
Joined: Dec 17, 2015 7:17 am
Contact:

Re: Forum Digest: CBT bug - Snapshot revert issues

Post by mkretzer »

Nice.
@Gostev: Can you confirm?
stewsie
Expert
Posts: 247
Liked: 20 times
Joined: May 22, 2015 7:16 am
Full Name: Paul
Contact:

Re: Forum Digest: CBT bug - Snapshot revert issues

Post by stewsie »

Now I have the alarm configured, overnight I have received a few emails regarding Veeam replicas. I hadn't looked in detail before but it looks like as part of the process the snapshot is reverted a number of times. Interesting behaviour.

If I need to use the replicas, one of the first things I would need to do is reset CBT.
mcz
Veeam Legend
Posts: 835
Liked: 172 times
Joined: Jul 19, 2016 8:39 am
Full Name: Michael
Location: Rheintal, Austria
Contact:

Re: Forum Digest: CBT bug - Snapshot revert issues

Post by mcz »

@Anton Gostev: Can you please clarify the situation with replicas? I mean veeam reverts snapshots by design (and I think it's the right thing to do). So what happens in case of a failback (talking about possible corruption), should we be considered or not?

Thanks!
Post Reply

Who is online

Users browsing this forum: Amazon [Bot] and 98 guests