Forum Digest: CBT bug - Snapshot revert issues

[Gostev]

@Gostev: To be honest that would be simple an genious. Please implement this and make it possible to turn it off

No worries. In Veeam, there's a registry key for everything!

[matthewingram]

another helpful change maybe to allow us to run a active full of an individual VM within a job.

[Gostev]

This will require some quite major job architecture, which is planned - but this requires a major release runway, so not something we can add last minute to v10.

[doum]

By the way, based on these events we're now looking at adding one last minute feature to v10: automatic CBT reset on each VM when Active Full backup is performed. Looking back at the past few CBT bugs, having this would have helped everyone to make sure they have a good backup without having to mess with CBT reset scripts.

better than nothing but doesn't solve the problem if you don't use active full, and for all the backup between a revert to snapshot (monday for example) and an active full (sunday)

[doum]

another helpful change maybe to allow us to run a active full of an individual VM within a job.

and reset the CBT before the active full

the best thing though would be for vmware to fix this huge bug "by design" and automatically reset the CBT when you revert a snapshot

[Gostev]

Currently, their intention is to disable CBT following snapshot revert.

[Regnor]

By the way, based on these events we're now looking at adding one last minute feature to v10: automatic CBT reset on each VM when Active Full backup is performed. Looking back at the past few CBT bugs, having this would have helped everyone to make sure they have a good backup without having to mess with CBT reset scripts.

This sounds like a great idea; +1
I already thought of regulary disabling CBT for every customer because of all the bugs.

[omerlin98000]

Hello all

One question is still in my mind:
- what an "Active Full backup" has to do with CBT (correct or corrupted)? I thought an Active full will ignore CBT by design and has to read ALL blocks from scratch?
- if Active full is sufficient, why do I have to reset CBT by script?

Of course, I have some jobs with incremental forever, those are now suspicious to me.
And yes, if I do a revert snapshot, it will be my responsibility to force Active full for the next backup job (event it is an incremental that very day).

Do I understand well?

[mkh]

please keep the "automatic CBT reset on active full" even if vmware arrived with patches before v10. It will give so much more peace of mind.

[jb238]

By the way, based on these events we're now looking at adding one last minute feature to v10: automatic CBT reset on each VM when Active Full backup is performed. Looking back at the past few CBT bugs, having this would have helped everyone to make sure they have a good backup without having to mess with CBT reset scripts.

I would like that. Data integrity is by far the most important, speed a distant second. Maybe implement a switch to turn it off if in some environtments the performance impact is significant.

[mma]

Keep in mind that some VMs might be in more than one job. CBT reset at active full is not optimal in this case.

[omerlin98000]

Hello all
One question is still in my mind:
- what an "Active Full backup" has to do with CBT (correct or corrupted)? I thought an Active full will ignore CBT by design and has to read ALL blocks from scratch?
- if Active full is sufficient, why do I have to reset CBT by script?
Of course, I have some jobs with incremental forever, those are now suspicious to me.
And yes, if I do a revert snapshot, it will be my responsibility to force Active full for the next backup job (event it is an incremental that very day).
Do I understand well?

Hello again, I have not seen any clear answer to my question, perhaps someone can explain to me if I'm wrong?

[Gostev]

The issue discussed here is with QueryChangedDiskAreas API, which is indeed used with Active Full backups to avoid having to physically read through empty VMDK blocks. Thanks!

[backupquestions]

I set up a vcenter alarm to alert me via email when any vm has been reverted. Now, my question...

If a revert happens and then an incremental Veeam backup is taken as normal, and THEN I become aware the next day and I reset CBT for the VM in vsphere.... The next incremental backup would be OK but is the chain broken now as the previous increment had garbage data possibly? Or am I OK so long as it gets reset eventually and I just have maybe 1 or 2 bad increments that happened before I could reset CBT?

Also, it sounded like as long as I never take a new active full on these vms, you could theoretically continue on as normal and the problem does not actually appear? You have to delete your backup and/or take new active full for the full thing to trigger. That and storage vmotion maybe...

[Gostev]

No one except VMware developers can tell you for sure, because only they have access to the source code and know exactly what happens with CBT data after VM has been reverted to a snapshot.

Based on the official publicly available information (VMware support KB article linked in my previous post), CBT does not support VM snapshot revert operation at all. Thus, to be on the safe side, you should assume your incremental chain is now broken, reset CBT on a VM, and do an Active Full backup.

[backupquestions]

So then how can I do an active full after cbt reset for just one vm in a job with a lot of vms in it? We are using per vm backup files but there is still no choice I can find for this. I can't just delete the old chain as I would lose a bunch of history.

[Gostev]

Unfortunately, it is not possible to do an active full on the given VM in a job currently. So I think the best solution will be to move that VM in the dedicated job, while letting "old backups" expire as a part of the original job.

[backupquestions]

What about backup copy jobs and offloaded data to object storage. Those are both incremental forever and would now contain corrupt data as well right?

I cant delete months worth of retention on those object gfs points.

The regular backup copy jobs only have a few points so I guess I can delete the vm on that if that's what I have to do and re seed it.

[Gostev]

All backups made prior to reverting VM snapshot are fine, so you definitely don't have to "delete months worth of retention on those object gfs points". You can keep them, and let them expire.

[stewsie]

I set up a vcenter alarm to alert me via email when any vm has been reverted. Now, my question...

If a revert happens and then an incremental Veeam backup is taken as normal, and THEN I become aware the next day and I reset CBT for the VM in vsphere.... The next incremental backup would be OK but is the chain broken now as the previous increment had garbage data possibly? Or am I OK so long as it gets reset eventually and I just have maybe 1 or 2 bad increments that happened before I could reset CBT?

Also, it sounded like as long as I never take a new active full on these vms, you could theoretically continue on as normal and the problem does not actually appear? You have to delete your backup and/or take new active full for the full thing to trigger. That and storage vmotion maybe...

What alert did you configure in vCenter? I cannot find anything to configure.

Thanks

[backupquestions]

It was not my idea so I will post the link where I learned how. Pretty easy to do. Make sure to set the alarm at the top level (vcenter object). https://www.virtual-allan.com/cbt-bug-o ... -snapshot/

But yeah, this is really unfortunate because I thought of moving my chain elsewhere but then it would expand all the space since REFS block clone is lost. Also I think you pretty much need to delete your chain from your VCC backup copy job unless you are not using incremental forever, (and most probably are since that is default). (as the bad increment that gets copied out there will eventually be merged right into the vbk since chain transform) The only good thing for me is I am using per-vm backup files so I can just delete that chain alone on the VCC backup copy job.

[Frosty]

I would like to configure an Alarm for exactly this situation. We're running vCenter/vSphere 6.7 here, however in the web interface I cannot find how to create an alarm that targets the com.vmware.vc.vm.VmStateRevertedToSnapshot event. Can anyone shed any light on this? Maybe I am just not looking in the right place.

[stewsie]

It is not easy how this alarm needs to be created.

I did the following in vCenter 6

Identify the object where the alarm is to be configured
Click Manage
Click +
Choose VM and Specific event
Click Next
Click +
There is no option in the drop down so I cleared the text and pasted com.vmware.vc.vm.VmStateRevertedToSnapshot
It took a couple of tries to add this
Complete the creation
I can now see the alarm

Next step is to test it

*** Update *** I have just tested this and the alarm has worked

[stewsie]

I would like to configure an Alarm for exactly this situation. We're running vCenter/vSphere 6.7 here, however in the web interface I cannot find how to create an alarm that targets the com.vmware.vc.vm.VmStateRevertedToSnapshot event. Can anyone shed any light on this? Maybe I am just not looking in the right place.

Hopefully you have seen my latest update

[mkretzer]

@stewsie its working, thank you very much!!

[mkretzer]

Someone asked it before but i think there has been no answer: What about quick rollback? Will it cause corrupted backups on the next backup take?

[backupquestions]

Gostev or someone at Veeam can confirm, but if you read here https://helpcenter.veeam.com/docs/backu ... l?ver=95u4 in the notes at the bottom, it says that CBT is reset after a quick rollback is used. (that is also why you can only perform a quick rollback once, until you have taken another backup after doing so) Here is another link which is older but likely still applies and talks about how you can only do it once and CBT is reset. https://www.veeam.com/blog/8-gems-in-ve ... lback.html

So it seems like this should be totally safe option and just never use snapshots anymore.

[mkretzer]

Nice.
@Gostev: Can you confirm?

[stewsie]

Now I have the alarm configured, overnight I have received a few emails regarding Veeam replicas. I hadn't looked in detail before but it looks like as part of the process the snapshot is reverted a number of times. Interesting behaviour.

If I need to use the replicas, one of the first things I would need to do is reset CBT.

[mcz]

@Anton Gostev: Can you please clarify the situation with replicas? I mean veeam reverts snapshots by design (and I think it's the right thing to do). So what happens in case of a failback (talking about possible corruption), should we be considered or not?

Thanks!