Discussions specific to the VMware vSphere hypervisor
Gostev
SVP, Product Management
Posts: 24638
Liked: 3467 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Forum Digest: CBT bug - Snapshot revert issues

Post by Gostev » Aug 29, 2019 2:28 pm

Marcel, try to perform a Storage VMotion on your test VM between steps 3 and 4.

Gostev
SVP, Product Management
Posts: 24638
Liked: 3467 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Forum Digest: CBT bug - Snapshot revert issues

Post by Gostev » Aug 29, 2019 2:30 pm

tinto1970 wrote:
Aug 29, 2019 12:45 pm
"Note: Ensure that there are no snapshots on the virtual machine before enabling change tracking. If you create snapshots before enabling CBT, the QueryChangedDiskAreas API might not return any error or the data returned by QueryChangedDiskAreas might be incorrect."
No, it was there from the start (and was even already discussed earlier in this thread).

crackocain
Service Provider
Posts: 127
Liked: 9 times
Joined: Dec 14, 2015 8:20 pm
Full Name: Mehmet Istanbullu
Location: Turkey
Contact:

Re: Forum Digest: CBT bug - Snapshot revert issues

Post by crackocain » Aug 29, 2019 2:54 pm

https://kb.vmware.com/s/article/2139574

In vmware kb affected versions are 5.1, 5.5, 6.0

6.5 and 6.7 are unaffected i guess right?

mma
Service Provider
Posts: 98
Liked: 18 times
Joined: Dec 22, 2011 9:12 am
Full Name: Marcel
Location: Lucerne, Switzerland
Contact:

Re: Forum Digest: CBT bug - Snapshot revert issues

Post by mma » Aug 30, 2019 7:15 am

Gostev wrote:
Aug 29, 2019 2:28 pm
Marcel, try to perform a Storage VMotion on your test VM between steps 3 and 4.
Tasks:
  1. create active full backup and check if cbt will be used
  2. create incremental backup and check if cbt will be used
  3. create snapshot CBT1 and revert
  4. migrate VM to different Datastore
  5. create incremental backup, check if cbt will be used
Result:
  1. CBT used
  2. CBT used
  3. done
  4. done
  5. CBT data is invalid, failing over to legacy incremental backup.
Regards
Marcel

Gostev
SVP, Product Management
Posts: 24638
Liked: 3467 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Forum Digest: CBT bug - Snapshot revert issues

Post by Gostev » Aug 30, 2019 12:52 pm

Sorry, missed the fact that you keep doing incrementals. If you now delete the backup completely, and create the new one (Active Full), it should be invalid due to QueryChangedDiskAreas with changeID * now returning invalid information on occupied VMDK blocks.

mma
Service Provider
Posts: 98
Liked: 18 times
Joined: Dec 22, 2011 9:12 am
Full Name: Marcel
Location: Lucerne, Switzerland
Contact:

Re: Forum Digest: CBT bug - Snapshot revert issues

Post by mma » Aug 30, 2019 1:18 pm

Thanks Gostev.
I just try to figure out if we should change something in our snapshoshot handling procedure. If I get it correctly, it's not that CBT isn't aware of snapshots reverts at all. There is just a very specific case, in which a backup gets corrupted?

Gostev
SVP, Product Management
Posts: 24638
Liked: 3467 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Forum Digest: CBT bug - Snapshot revert issues

Post by Gostev » Aug 30, 2019 9:01 pm

Well, VMware's current official position is that CBT does not support reverting VM snapshots in principle - and as noted earlier, I cannot really "argue" it publicly without potentially getting Veeam in trouble :D

We've opened a different type of support case (DCPN) with VMware earlier this week, supposedly this one enables more direct developer to developer conversations. We shared everything we know about the issue there - let's see what response are we going to get.

We also raised this topic at a few meetings we had with VMware at VMworld U.S., so I think there's enough awareness within VMware of this issue at the moment. Let's see what they come back with after taking another look.

masonit
Service Provider
Posts: 215
Liked: 14 times
Joined: Oct 09, 2012 2:30 pm
Full Name: Magnus
Contact:

Re: Forum Digest: CBT bug - Snapshot revert issues

Post by masonit » Sep 03, 2019 8:28 am

Hi

What happens if I would use Quick rollback in Veeam? Would I potentially end upp with same cbt issues as with reverting snapshot?

\Masonit

SRMETAS
Influencer
Posts: 22
Liked: 2 times
Joined: Apr 10, 2015 4:59 am
Full Name: Simon Roggli
Location: Bern, Switzerland
Contact:

Re: Forum Digest: CBT bug - Snapshot revert issues

Post by SRMETAS » Sep 03, 2019 8:59 am

I'm a little bit confused, maybe somone here can help me.
When talking about CBT i allways thought it was a VEEAM only feature. Now when i read this i found out that one also can enable CBT on the VM itself with (ctkEnabled true).
Non of our VM's have this enabled but still when backing up with VEEAM in the Action Summery i see CBT entrys. Are we talking about two different CBT's?

Anyway i did a test with one VM and enabled CBT (ctkEnabled true). Did some backups, snapshots and revert snapshots etc. it seeam that i cannot reproduce this problem. BTW vSphere 6.7 is used.

mcz
Expert
Posts: 291
Liked: 53 times
Joined: Jul 19, 2016 8:39 am
Full Name: Michael
Contact:

Re: Forum Digest: CBT bug - Snapshot revert issues

Post by mcz » Sep 03, 2019 11:52 am 1 person likes this post

CBT is a vmware feature and tracks the changed blocks between one and another snapshot. If CBT is not available, veeam reads the whole disk and compares the blocks to the ones in the latest restore point to only store the increments. So veeam has built in functionalities to get the same result. If you say you cannot reproduce the problem, what exactly do you mean? If you've got a data corruption due to wrong CBT result's you probably wouldn't recognize it immediately.

SRMETAS
Influencer
Posts: 22
Liked: 2 times
Joined: Apr 10, 2015 4:59 am
Full Name: Simon Roggli
Location: Bern, Switzerland
Contact:

Re: Forum Digest: CBT bug - Snapshot revert issues

Post by SRMETAS » Sep 04, 2019 12:18 pm

Thanks mcz. I somewhow misunderstood the article from gostev. i thought there will be an error message if i revert from snapshot when cbt is enabled. So our VM's do not have CBT enabled anyway (except the one i tested) and i will leave it that way until the problem is solved.

darecki
Novice
Posts: 8
Liked: never
Joined: Jul 20, 2018 5:15 am
Full Name: Darek Margas
Contact:

Re: Forum Digest: CBT bug - Snapshot revert issues

Post by darecki » Sep 05, 2019 6:55 am

Is it possible to disable CBT on Veeam side while making winzip?

stewsie
Expert
Posts: 118
Liked: 6 times
Joined: May 22, 2015 7:16 am
Full Name: Paul
Contact:

[MERGED] Backup corruption issue

Post by stewsie » Sep 05, 2019 9:21 am

Hi

A recent email from GOSTEV mentioned the following

This just in. We've been troubleshooting one backup corruption issue seen internally in one of our labs, where all signs pointed to a possible VMware changed block tracking (CBT) bug. Eventually, this was tracked down to a revert snapshot operation on the protected VM, following which CBT API started to return invalid data.

So we've opened a support case with VMware Support, and after 2 months their conclusion was that this corruption is "by design" and is due to the fact that CBT API does not support reverting snapshot on a VM. They even published the official support KB article about this. I'm still trying to wrap my head around their response, but my first reaction is that it makes little sense? I would argue ESXi should then simply reset CBT on a VM following snapshot revert operation, or even just start returning an error – instead of providing invalid CBT information, as if nothing happened? So this week, we'll be escalating this issue through our VMware Alliance channel as the next step. Normally, I would wait until we get another opinion there, however I had to share what we know so far immediately - since this issue leads to backup corruption.

I have just had to revert some VMs and need to know if this now means any future backups will be corrupt and if so do I just need to take a new full backup to avoid any corruption?

Thanks

stewsie
Expert
Posts: 118
Liked: 6 times
Joined: May 22, 2015 7:16 am
Full Name: Paul
Contact:

Re: Backup corruption issue - weekly email GOSTEV

Post by stewsie » Sep 05, 2019 9:29 am

I have just seen vmware-vsphere-f24/forum-digest-snapsho ... 61404.html

Should have looked in more detail before posting this

stewsie
Expert
Posts: 118
Liked: 6 times
Joined: May 22, 2015 7:16 am
Full Name: Paul
Contact:

Re: Forum Digest: CBT bug - Snapshot revert issues

Post by stewsie » Sep 05, 2019 9:44 am 1 person likes this post

Is there a way to identify if backups are corrupt following a snapshot revert operation?

Gostev
SVP, Product Management
Posts: 24638
Liked: 3467 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Forum Digest: CBT bug - Snapshot revert issues

Post by Gostev » Sep 05, 2019 8:35 pm 1 person likes this post

Sure... with SureBackup!

stewsie
Expert
Posts: 118
Liked: 6 times
Joined: May 22, 2015 7:16 am
Full Name: Paul
Contact:

Re: Forum Digest: CBT bug - Snapshot revert issues

Post by stewsie » Sep 06, 2019 6:16 am 1 person likes this post

I ran a backup overnight where all VMs in that job had a snapshot reverted. All VMs in the job said CBT Data is Invalid. As Gostev suggested I am running a Surebackup job. Confidence is low at the moment now this issue has been highlighted

Is the backup file integrity scan sufficient to identify any issues?

mcz
Expert
Posts: 291
Liked: 53 times
Joined: Jul 19, 2016 8:39 am
Full Name: Michael
Contact:

Re: Forum Digest: CBT bug - Snapshot revert issues

Post by mcz » Sep 06, 2019 8:34 am 1 person likes this post

Gostev wrote:
Sep 05, 2019 8:35 pm
Sure... with SureBackup!
...but I guess not always? Suppose that CBT wouldn't return one single block and suppose that this block belongs to a file which isn't part of any OS or other process which is being checked by SureBackup, veeam wouldn't recognize that there is something wrong, right? Please correct me if I'm wrong.

stewsie
Expert
Posts: 118
Liked: 6 times
Joined: May 22, 2015 7:16 am
Full Name: Paul
Contact:

Re: Forum Digest: CBT bug - Snapshot revert issues

Post by stewsie » Sep 06, 2019 9:03 am

So is the safest thing to do following a snapshot revert to do the following for each VM?

https://kb.vmware.com/s/article/1031873 ... CaseLike=1


$vm="VM_Name"

$vmtest = Get-vm $vm| get-view
$vmConfigSpec = New-Object VMware.Vim.VirtualMachineConfigSpec

#disable ctk
$vmConfigSpec.changeTrackingEnabled = $false
$vmtest.reconfigVM($vmConfigSpec)
$snap=New-Snapshot $vm -Name "Disable CBT"
$snap | Remove-Snapshot -confirm:$false

# enable ctk
$vmConfigSpec.changeTrackingEnabled = $true
$vmtest.reconfigVM($vmConfigSpec)
$snap=New-Snapshot $vm -Name "Enable CBT"
$snap | Remove-Snapshot -confirm:$false

Is there a way to report failed CBT. The backup job completes with no issues so not easy to see if any VMs had issues

Gostev
SVP, Product Management
Posts: 24638
Liked: 3467 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Forum Digest: CBT bug - Snapshot revert issues

Post by Gostev » Sep 06, 2019 12:22 pm 2 people like this post

mcz wrote:
Sep 06, 2019 8:34 am
...but I guess not always? Suppose that CBT wouldn't return one single block and suppose that this block belongs to a file which isn't part of any OS or other process which is being checked by SureBackup, veeam wouldn't recognize that there is something wrong, right? Please correct me if I'm wrong.
It is important to realize that Veeam by itself does not recognize wrong"stuff at all - this is the job of SureBackup test scripts. For which we recommend you use application-specific integrity testing tools that most enterprise apps provide (Isinteg.exe for Exchange, DBCC CHECKDB for SQL Server to name a few). These tools will always recognize an issue with an application, even if only a single block of a mission-critical file is wrong.

Also, don't think about SureBackup as about some king of magic. It merely makes possible to fully automate DR testing, and do it at no extra cost every day and for every backup vs. only testing a few selected backups once a quarter (which was best most IT shops could do before we invented SureBackup - simply because legacy DR tests take too long and are too costly from human resources perspective). But beyond that, I'm sure you realize that everything you said above applies to any DR test at all, no matter how you perform it. For example, if you simply do a full restore and start the server, but don't scan critical application files with some content-aware integrity checker - that makes it a bad manual DR test as well.

mcz
Expert
Posts: 291
Liked: 53 times
Joined: Jul 19, 2016 8:39 am
Full Name: Michael
Contact:

Re: Forum Digest: CBT bug - Snapshot revert issues

Post by mcz » Sep 09, 2019 9:31 am

Thanks for your response, Anton. Yes of course, if you are able to test the integrity of files then you can be sure for that part. But what I wannted to point out was: If you "just" have a drive with some files on it and you can't check the integrity (suppose a excel-sheet) then you wouldn't recognize it even if you run the sureBackup job. Of course, that's not veeam's fault and only due to the fact that you can't verify every file of a vm.

stewsie
Expert
Posts: 118
Liked: 6 times
Joined: May 22, 2015 7:16 am
Full Name: Paul
Contact:

Re: Forum Digest: CBT bug - Snapshot revert issues

Post by stewsie » Sep 10, 2019 6:17 am

stewsie wrote:
Sep 06, 2019 9:03 am
So is the safest thing to do following a snapshot revert to do the following for each VM?

https://kb.vmware.com/s/article/1031873 ... CaseLike=1
Should I reset CBT for the affected VMs? I have tested this and the who disk appears to be read although a full backup is not taken. Would it be best practice to reset CBT and then take a full backup?

mma
Service Provider
Posts: 98
Liked: 18 times
Joined: Dec 22, 2011 9:12 am
Full Name: Marcel
Location: Lucerne, Switzerland
Contact:

Re: Forum Digest: CBT bug - Snapshot revert issues

Post by mma » Sep 10, 2019 6:33 am 1 person likes this post

"I have tested this and the who disk appears to be read although a full backup is not taken."

That's the trick behind CBT ;-).
CBT is a feature of ESXi; it tells you which block have changed since your last CBT "reference". Your backup software just has to query the changed block. If you don't use CBT, Veeam has to read every block, hash it and check if the hash is allready known. It's not just veeam, it's the same for any other* backup software which uses CBT.

And yes, if you dont trust your CBT information you have to do an active full after the CBT reset.

(*Disclaimer - don't use other backup software :P )


Offtopic - Is it me or is there no way to quote the latest post?

Regards
Marcel

mcz
Expert
Posts: 291
Liked: 53 times
Joined: Jul 19, 2016 8:39 am
Full Name: Michael
Contact:

Re: Forum Digest: CBT bug - Snapshot revert issues

Post by mcz » Sep 10, 2019 7:02 am 1 person likes this post

Marcel, the last post cannot be quoted - that's what I know from the forum quys. And one question: Why would you have to do an active full after CBT reset? If veeam reads the whole disk and compares every block, there's no need to do an active full.

mma
Service Provider
Posts: 98
Liked: 18 times
Joined: Dec 22, 2011 9:12 am
Full Name: Marcel
Location: Lucerne, Switzerland
Contact:

Re: Forum Digest: CBT bug - Snapshot revert issues

Post by mma » Sep 10, 2019 7:14 am

I only do a CBT reset if something happend an I no longer trust my CBT information. The backup taken before the reset might be corrupted or something else.
We talk about backup, it's allways good to be on the safe side.

Do an active full, get a new backup chain with working CBT an sleep well :-)

stewsie
Expert
Posts: 118
Liked: 6 times
Joined: May 22, 2015 7:16 am
Full Name: Paul
Contact:

Re: Forum Digest: CBT bug - Snapshot revert issues

Post by stewsie » Sep 10, 2019 8:00 am

mma wrote:
Sep 10, 2019 6:33 am
"I have tested this and the who disk appears to be read although a full backup is not taken."

That's the trick behind CBT ;-).
CBT is a feature of ESXi; it tells you which block have changed since your last CBT "reference". Your backup software just has to query the changed block. If you don't use CBT, Veeam has to read every block, hash it and check if the hash is allready known. It's not just veeam, it's the same for any other* backup software which uses CBT.

And yes, if you dont trust your CBT information you have to do an active full after the CBT reset.
Thanks for the info. I will reset CBT on the VMs I reverted the snapshot on to be safe.

Gostev
SVP, Product Management
Posts: 24638
Liked: 3467 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Forum Digest: CBT bug - Snapshot revert issues

Post by Gostev » Sep 10, 2019 4:37 pm 6 people like this post

By the way, based on these events we're now looking at adding one last minute feature to v10: automatic CBT reset on each VM when Active Full backup is performed. Looking back at the past few CBT bugs, having this would have helped everyone to make sure they have a good backup without having to mess with CBT reset scripts.

Gostev
SVP, Product Management
Posts: 24638
Liked: 3467 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Forum Digest: CBT bug - Snapshot revert issues

Post by Gostev » Sep 10, 2019 4:44 pm

mma wrote:
Sep 10, 2019 6:33 am
Offtopic - Is it me or is there no way to quote the latest post?
No, it's not just you :) this was implemented a few months ago, and it was the best thing since sliced bread. Most users tend to use "Quote" as their default "Reply" button, and they quote the post in its entirety, which is really unnecessary when responding to the immediate post. We had some threads where literally every single post duplicated the previous one, making the whole discussion extremely hard to read.

I do struggle myself sometimes due to this change (being unable to quote certain part of the immediate post), but I take it over the mess we had before any day!

stewsie
Expert
Posts: 118
Liked: 6 times
Joined: May 22, 2015 7:16 am
Full Name: Paul
Contact:

Re: Forum Digest: CBT bug - Snapshot revert issues

Post by stewsie » Sep 11, 2019 6:33 am 1 person likes this post

Gostev wrote:
Sep 10, 2019 4:37 pm
By the way, based on these events we're now looking at adding one last minute feature to v10: automatic CBT reset on each VM when Active Full backup is performed. Looking back at the past few CBT bugs, having this would have helped everyone to make sure they have a good backup without having to mess with CBT reset scripts.
Excellent. Thank you :D

mkretzer
Expert
Posts: 546
Liked: 123 times
Joined: Dec 17, 2015 7:17 am
Contact:

Re: Forum Digest: CBT bug - Snapshot revert issues

Post by mkretzer » Sep 11, 2019 5:00 pm

@Gostev: To be honest that would be simple an genious. Please implement this and make it possible to turn it off :-)

Post Reply

Who is online

Users browsing this forum: Bing [Bot], restore-helper and 37 guests