Forum Digest: CBT bug - Snapshot revert issues

[Gostev]

Marcel, try to perform a Storage VMotion on your test VM between steps 3 and 4.

[Gostev]

"Note: Ensure that there are no snapshots on the virtual machine before enabling change tracking. If you create snapshots before enabling CBT, the QueryChangedDiskAreas API might not return any error or the data returned by QueryChangedDiskAreas might be incorrect."

No, it was there from the start (and was even already discussed earlier in this thread).

[crackocain]

https://kb.vmware.com/s/article/2139574

In vmware kb affected versions are 5.1, 5.5, 6.0

6.5 and 6.7 are unaffected i guess right?

[mma]

Marcel, try to perform a Storage VMotion on your test VM between steps 3 and 4.

Tasks:

1. create active full backup and check if cbt will be used
2. create incremental backup and check if cbt will be used
3. create snapshot CBT1 and revert
4. migrate VM to different Datastore
5. create incremental backup, check if cbt will be used

Result:

• CBT used
• CBT used
• done
• done
• CBT data is invalid, failing over to legacy incremental backup.

Regards
Marcel

[Gostev]

Sorry, missed the fact that you keep doing incrementals. If you now delete the backup completely, and create the new one (Active Full), it should be invalid due to QueryChangedDiskAreas with changeID * now returning invalid information on occupied VMDK blocks.

[mma]

Thanks Gostev.
I just try to figure out if we should change something in our snapshoshot handling procedure. If I get it correctly, it's not that CBT isn't aware of snapshots reverts at all. There is just a very specific case, in which a backup gets corrupted?

[Gostev]

Well, VMware's current official position is that CBT does not support reverting VM snapshots in principle - and as noted earlier, I cannot really "argue" it publicly without potentially getting Veeam in trouble

We've opened a different type of support case (DCPN) with VMware earlier this week, supposedly this one enables more direct developer to developer conversations. We shared everything we know about the issue there - let's see what response are we going to get.

We also raised this topic at a few meetings we had with VMware at VMworld U.S., so I think there's enough awareness within VMware of this issue at the moment. Let's see what they come back with after taking another look.

[masonit]

Hi

What happens if I would use Quick rollback in Veeam? Would I potentially end upp with same cbt issues as with reverting snapshot?

\Masonit

[SRMETAS]

I'm a little bit confused, maybe somone here can help me.
When talking about CBT i allways thought it was a VEEAM only feature. Now when i read this i found out that one also can enable CBT on the VM itself with (ctkEnabled true).
Non of our VM's have this enabled but still when backing up with VEEAM in the Action Summery i see CBT entrys. Are we talking about two different CBT's?

Anyway i did a test with one VM and enabled CBT (ctkEnabled true). Did some backups, snapshots and revert snapshots etc. it seeam that i cannot reproduce this problem. BTW vSphere 6.7 is used.

[mcz]

CBT is a vmware feature and tracks the changed blocks between one and another snapshot. If CBT is not available, veeam reads the whole disk and compares the blocks to the ones in the latest restore point to only store the increments. So veeam has built in functionalities to get the same result. If you say you cannot reproduce the problem, what exactly do you mean? If you've got a data corruption due to wrong CBT result's you probably wouldn't recognize it immediately.

[SRMETAS]

Thanks mcz. I somewhow misunderstood the article from gostev. i thought there will be an error message if i revert from snapshot when cbt is enabled. So our VM's do not have CBT enabled anyway (except the one i tested) and i will leave it that way until the problem is solved.

[darecki]

Is it possible to disable CBT on Veeam side while making winzip?

[stewsie]

Hi

A recent email from GOSTEV mentioned the following

This just in. We've been troubleshooting one backup corruption issue seen internally in one of our labs, where all signs pointed to a possible VMware changed block tracking (CBT) bug. Eventually, this was tracked down to a revert snapshot operation on the protected VM, following which CBT API started to return invalid data.

So we've opened a support case with VMware Support, and after 2 months their conclusion was that this corruption is "by design" and is due to the fact that CBT API does not support reverting snapshot on a VM. They even published the official support KB article about this. I'm still trying to wrap my head around their response, but my first reaction is that it makes little sense? I would argue ESXi should then simply reset CBT on a VM following snapshot revert operation, or even just start returning an error – instead of providing invalid CBT information, as if nothing happened? So this week, we'll be escalating this issue through our VMware Alliance channel as the next step. Normally, I would wait until we get another opinion there, however I had to share what we know so far immediately - since this issue leads to backup corruption.

I have just had to revert some VMs and need to know if this now means any future backups will be corrupt and if so do I just need to take a new full backup to avoid any corruption?

Thanks

[stewsie]

I have just seen vmware-vsphere-f24/forum-digest-snapsho ... 61404.html

Should have looked in more detail before posting this

[stewsie]

Is there a way to identify if backups are corrupt following a snapshot revert operation?

[Gostev]

Sure... with SureBackup!

[stewsie]

I ran a backup overnight where all VMs in that job had a snapshot reverted. All VMs in the job said CBT Data is Invalid. As Gostev suggested I am running a Surebackup job. Confidence is low at the moment now this issue has been highlighted

Is the backup file integrity scan sufficient to identify any issues?

[mcz]

Sure... with SureBackup!

...but I guess not always? Suppose that CBT wouldn't return one single block and suppose that this block belongs to a file which isn't part of any OS or other process which is being checked by SureBackup, veeam wouldn't recognize that there is something wrong, right? Please correct me if I'm wrong.

[stewsie]

So is the safest thing to do following a snapshot revert to do the following for each VM?

https://kb.vmware.com/s/article/1031873 ... CaseLike=1


$vm="VM_Name"

$vmtest = Get-vm $vm| get-view
$vmConfigSpec = New-Object VMware.Vim.VirtualMachineConfigSpec

#disable ctk
$vmConfigSpec.changeTrackingEnabled = $false
$vmtest.reconfigVM($vmConfigSpec)
$snap=New-Snapshot $vm -Name "Disable CBT"
$snap | Remove-Snapshot -confirm:$false

# enable ctk
$vmConfigSpec.changeTrackingEnabled = $true
$vmtest.reconfigVM($vmConfigSpec)
$snap=New-Snapshot $vm -Name "Enable CBT"
$snap | Remove-Snapshot -confirm:$false

Is there a way to report failed CBT. The backup job completes with no issues so not easy to see if any VMs had issues

[Gostev]

...but I guess not always? Suppose that CBT wouldn't return one single block and suppose that this block belongs to a file which isn't part of any OS or other process which is being checked by SureBackup, veeam wouldn't recognize that there is something wrong, right? Please correct me if I'm wrong.

It is important to realize that Veeam by itself does not recognize wrong"stuff at all - this is the job of SureBackup test scripts. For which we recommend you use application-specific integrity testing tools that most enterprise apps provide (Isinteg.exe for Exchange, DBCC CHECKDB for SQL Server to name a few). These tools will always recognize an issue with an application, even if only a single block of a mission-critical file is wrong.

Also, don't think about SureBackup as about some king of magic. It merely makes possible to fully automate DR testing, and do it at no extra cost every day and for every backup vs. only testing a few selected backups once a quarter (which was best most IT shops could do before we invented SureBackup - simply because legacy DR tests take too long and are too costly from human resources perspective). But beyond that, I'm sure you realize that everything you said above applies to any DR test at all, no matter how you perform it. For example, if you simply do a full restore and start the server, but don't scan critical application files with some content-aware integrity checker - that makes it a bad manual DR test as well.

[mcz]

Thanks for your response, Anton. Yes of course, if you are able to test the integrity of files then you can be sure for that part. But what I wannted to point out was: If you "just" have a drive with some files on it and you can't check the integrity (suppose a excel-sheet) then you wouldn't recognize it even if you run the sureBackup job. Of course, that's not veeam's fault and only due to the fact that you can't verify every file of a vm.

[stewsie]

So is the safest thing to do following a snapshot revert to do the following for each VM?

https://kb.vmware.com/s/article/1031873 ... CaseLike=1

Should I reset CBT for the affected VMs? I have tested this and the who disk appears to be read although a full backup is not taken. Would it be best practice to reset CBT and then take a full backup?

[mma]

"I have tested this and the who disk appears to be read although a full backup is not taken."

That's the trick behind CBT .
CBT is a feature of ESXi; it tells you which block have changed since your last CBT "reference". Your backup software just has to query the changed block. If you don't use CBT, Veeam has to read every block, hash it and check if the hash is allready known. It's not just veeam, it's the same for any other* backup software which uses CBT.

And yes, if you dont trust your CBT information you have to do an active full after the CBT reset.

(*Disclaimer - don't use other backup software )


Offtopic - Is it me or is there no way to quote the latest post?

Regards
Marcel

[mcz]

Marcel, the last post cannot be quoted - that's what I know from the forum quys. And one question: Why would you have to do an active full after CBT reset? If veeam reads the whole disk and compares every block, there's no need to do an active full.

[mma]

I only do a CBT reset if something happend an I no longer trust my CBT information. The backup taken before the reset might be corrupted or something else.
We talk about backup, it's allways good to be on the safe side.

Do an active full, get a new backup chain with working CBT an sleep well

[stewsie]

"I have tested this and the who disk appears to be read although a full backup is not taken."

That's the trick behind CBT .
CBT is a feature of ESXi; it tells you which block have changed since your last CBT "reference". Your backup software just has to query the changed block. If you don't use CBT, Veeam has to read every block, hash it and check if the hash is allready known. It's not just veeam, it's the same for any other* backup software which uses CBT.

And yes, if you dont trust your CBT information you have to do an active full after the CBT reset.

Thanks for the info. I will reset CBT on the VMs I reverted the snapshot on to be safe.

[Gostev]

By the way, based on these events we're now looking at adding one last minute feature to v10: automatic CBT reset on each VM when Active Full backup is performed. Looking back at the past few CBT bugs, having this would have helped everyone to make sure they have a good backup without having to mess with CBT reset scripts.

[Gostev]

Offtopic - Is it me or is there no way to quote the latest post?

No, it's not just you :) this was implemented a few months ago, and it was the best thing since sliced bread. Most users tend to use "Quote" as their default "Reply" button, and they quote the post in its entirety, which is really unnecessary when responding to the immediate post. We had some threads where literally every single post duplicated the previous one, making the whole discussion extremely hard to read.

I do struggle myself sometimes due to this change (being unable to quote certain part of the immediate post), but I take it over the mess we had before any day!

[stewsie]

By the way, based on these events we're now looking at adding one last minute feature to v10: automatic CBT reset on each VM when Active Full backup is performed. Looking back at the past few CBT bugs, having this would have helped everyone to make sure they have a good backup without having to mess with CBT reset scripts.

Excellent. Thank you

[mkretzer]

@Gostev: To be honest that would be simple an genious. Please implement this and make it possible to turn it off