VMware CBT bug KB 2090639

[Vitaliy S.]

Hi Kevin,

Try the instructions posted in this blog post > http://pipitone.co/2014/02/09/reset-cbt-script/

Thanks!

[kwells]

Thank you for the link. I have looked at it, and am still confused. The script listed seems to have the same lines listed in it in several places. Is the powershell ISE environment the standard windows powershell and not the powercli environment (sorry, but I am a total novice when it comes to powercli as I have never had to use it).

[orangep7]

We have reset CBT against a number of VMs and have one quick question.

Firstly, I understand that after disabling/enabling CBT the next backup will scan the VMDKs entirely but only copy the missing deltas to the backup repository. The advice is to then run a full backup for the next job to start afresh.

However, if for whatever reason a full job wasn't possible and the next job after the CBT reset was only a differential, would the newly created CBT data still be accurate for future jobs?

I'm guessing CBT would still be accurate for future jobs, but the drawback with only running a differential following a CBT reset would be that only the latest differential could be trusted, and the rest of the chain may be inconsistent. Is this correct or does it mean that in order for CBT to be trusted you must run a Full (NOT a Diff) immediately after the CBT reset?

[brandon.patton]

Subscribing to thread

[jai64]

Subscribing to thread

[sedrum]

We have been experiencing CBT errors for quite a while during replications and backups. Sometimes these are just warnings, sometimes the job fails and the retry is successful. I have had several tickets with Veeam on this issue and we also had VMware involved at one point. We went through all the reset CBT processes. This was happening to only one of our vms and even cloning the vm would not fix the issue. I am now worried seeing this thread that our replications and/or backups might be corrupted due to this issue although we have on several occasions done failovers of replication and tested full backup restores that seem ok, but how would I really know for sure?
We are running under Redhat 6x and also are using a D3 multi-value database and these CBT errors only occur on the DB partitions which is strange although they are the largest. We are now having this issue on other vms and it's random not always the same vm.

Is there a way to determine our backups/replicas are not corrupt? if this was posted here and I missed it, sorry for the redundant question.

[rei.andrea]

We have schedule "active full backups" to run weekly..

"active full backups" reset CBT is right ?

[lightsout]

We have schedule "active full backups" to run weekly..

"active full backups" reset CBT is right ?

No, the CBT must be reset independently.

[rei.andrea]

No, the CBT must be reset independently.

OK I am planning a reset of CBT when possible...

My question is:
the "active full backups" (NOT Synthetic Full Backup) [.vbk] are not affected to this Bug and are consinstant ?
the "active full backups" it should not use CBT It's perform a full read on whole VM right ?

[andymcknight]

For those of you who don't like the idea of fully scripting the take snapshot, remove snapshot method of resetting CBT, I've modified the script a little so that it gives you a list of VMs you have that have CBT enabled and a hard disk over 128GB in size but doesn't take any further action. That way you can do the snapshot part manually if you feel better about that. This doesn't tell you if you have extended the drives at any point but this may be useful to narrow down your search a little.

Code: Select all

$problemvms = @()
$cbtenabledvms = Get-VM | Where {$_.Extensiondata.Config.ChangeTrackingEnabled -eq 'True'}
Foreach ($vm in $cbtenabledvms)
	{
		Foreach ($disk in $vm.Guest.Disks)
			{
				If ($disk.CapacityGB -gt 128)
					{
						$problemvms += $vm
					}
			}
	}
$uniquevms = $problemvms | Group-Object Name | Measure-Object
$problemvms
Write-Host("Count of problem VMs: " + $uniquevms.Count)

[Gostev]

My question is:
the "active full backups" (NOT Synthetic Full Backup) [.vbk] are not affected to this Bug and are consinstant ?
the "active full backups" it should not use CBT It's perform a full read on whole VM right ?

Active Full backup does uses CBT information, so it is affected.
Resetting CBT is a must no matter of backup mode or schedule.

[Gostev]

Is there a way to determine our backups/replicas are not corrupt? if this was posted here and I missed it, sorry for the redundant question.

In our testing, at least in case of Windows VMs, Instant VM Recovery from an incremental restore point taken after virtual disk extension that breaks CBT has been a good simple test. In case of expansion of the OS disk, OS simply fails to boot. In case of expansion of the data disk, OS looses volumes located on the disk (and disk shows up as offline in the Disk Management snap-in). So, simple Instant VM Recovery from backup with network disconnected (or test replica failover) followed by manual inspection of disks inside a VM should help to identify the issue in many cases in our experience. This also means that anyone using SureBackup and getting successes is unlikely to have issues with the current backups and replicas, as either issue mentioned above will cause a failed SureBackup test (either due to no heartbeat, or failed application test for apps using data disks).

But again, due to not having access to VMware source code, we have no insight of CBT internals and how this bug breaks CBT logic. As such, we cannot guarantee that the test above is sufficient in every scenario CBT corruption scenario and for every OS. And because this is impossible to test in a reasonable time frame, we highly recommend to reset CBT for all VMs with disks over 128GB in size, as a matter of precaution, and run the jobs at least once to create the new restore point that will be guaranteed to be consistent.

[Gostev]

To subscribe to this topic, please use the Subscribe topic link at the bottom of this page. Posting a comment will NOT subscribe you to the post with this board (this will do nothing except including the topic into your personalized weekly digest, under personal contents).

[REuser]

How about just unchecking the CBT option within the veeam job? Then running the job without cbt enabled. And then checking cbt in the job after a successful backup without cbt? Or do we still have to disable cbt on the vmware side of things?

[jlester]

Thanks to Veeam and andymcnight for the PowerCLI scripts. I combined the two to only make the changes for VM's that seemed to be affected:

Code: Select all

    $problemvms = @()
    $cbtenabledvms = Get-VM | Where {$_.Extensiondata.Config.ChangeTrackingEnabled -eq 'True'}
    Foreach ($vm in $cbtenabledvms)       {Foreach ($disk in $vm.Guest.Disks){If ($disk.CapacityGB -gt 128){$problemvms += $vm}}}

    $uniquevms = $problemvms | Group-Object Name | Measure-Object
    $problemvms
    Write-Host("Count of problem VMs: " + $uniquevms.Count)
	
    foreach($vm in $problemvms){
    $vm.ExtensionData.ReconfigVM($spec)
    $snap=$vm | New-Snapshot -Name 'Disable CBT'
    $snap | Remove-Snapshot -confirm:$false }

In case you are a newbie to PowerCLI like me, the foreach loops will not run until you hit enter on a blank space after the trailing }. Took me a few minutes to figure that out. YMMV, so use caution before using in your environment.

Jason

[lightsout]

Thanks to Veeam and andymcnight for the PowerCLI scripts. I combined the two to only make the changes for VM's that seemed to be affected:

Code: Select all

    $problemvms = @()
    $cbtenabledvms = Get-VM | Where {$_.Extensiondata.Config.ChangeTrackingEnabled -eq 'True'}
    Foreach ($vm in $cbtenabledvms)       {Foreach ($disk in $vm.Guest.Disks){If ($disk.CapacityGB -gt 128){$problemvms += $vm}}}

    $uniquevms = $problemvms | Group-Object Name | Measure-Object
    $problemvms
    Write-Host("Count of problem VMs: " + $uniquevms.Count)
	
    foreach($vm in $problemvms){
    $vm.ExtensionData.ReconfigVM($spec)
    $snap=$vm | New-Snapshot -Name 'Disable CBT'
    $snap | Remove-Snapshot -confirm:$false }

In case you are a newbie to PowerCLI like me, the foreach loops will not run until you hit enter on a blank space after the trailing }. Took me a few minutes to figure that out. YMMV, so use caution before using in your environment.

Jason

Thats a nice script. Just don't forget to define $spec.

Code: Select all

$spec = New-Object VMware.Vim.VirtualMachineConfigSpec 
$spec.ChangeTrackingEnabled = $false

[jlester]

Whoops, you're right:

Code: Select all

    $problemvms = @()
    $cbtenabledvms = Get-VM | Where {$_.Extensiondata.Config.ChangeTrackingEnabled -eq 'True'}
    Foreach ($vm in $cbtenabledvms)       {Foreach ($disk in $vm.Guest.Disks){If ($disk.CapacityGB -gt 128){$problemvms += $vm}}}

    $uniquevms = $problemvms | Group-Object Name | Measure-Object
    $problemvms
    Write-Host("Count of problem VMs: " + $uniquevms.Count)

    $spec = New-Object VMware.Vim.VirtualMachineConfigSpec 
    $spec.ChangeTrackingEnabled = $false

    foreach($vm in $problemvms){
    $vm.ExtensionData.ReconfigVM($spec)
    $snap=$vm | New-Snapshot -Name 'Disable CBT'
    $snap | Remove-Snapshot -confirm:$false }

I think if the VM has multiple volumes that match, this will run multiple times for that VM. I believe you could do that last foreach like this:

Code: Select all

foreach($vm in $uniquevms){

And it would only run once per VM. I'm not entirely sure what Andy's

Code: Select all

$uniquevms = $problemvms | Group-Object Name | Measure-Object

line does though. I didn't have many affected, so I didn't test it that way. In a large environment, you might want to only run it once per VM.

[Gostev]

How about just unchecking the CBT option within the veeam job? Then running the job without cbt enabled. And then checking cbt in the job after a successful backup without cbt? Or do we still have to disable cbt on the vmware side of things?

Yes, CBT must be reset on VMware side.

[GabesVirtualWorld]

Looking at the script from JLESTER. Is confirmed that the following procedure resets the CBT and will cause NO downtime to VMs? And if confirmed, is it also "official" by VMware?

- Powershell -> set trackingenabled to false
- Create snapshot
- Remove snapshot
- Perform a backup which will again enabled CBT and do a full scan.

[jlester]

Gabe,

Did you see the Veeam KB article that was linked earlier:

http://www.veeam.com/kb1940

Their example script resets it for every VM with CBT enabled. So, I think that is the procedure and it worked with no downtime in my environment. My nightly backups kickoff in about an hour, so I'll know tomorrow if the affected VM's did a full scan.

Jason

[GabesVirtualWorld]

Gabe,

Did you see the Veeam KB article that was linked earlier:

http://www.veeam.com/kb1940

Their example script resets it for every VM with CBT enabled. So, I think that is the procedure and it worked with no downtime in my environment. My nightly backups kickoff in about an hour, so I'll know tomorrow if the affected VM's did a full scan.

Jason

Thanks Jason !!!

[jlester]

Is there an easy way to tell if CBT was reset from the backup statistics? Some of the affected VM's have backed up alreadt tonight and still show [CBT] beside the disks in the statistics. I don't see anything indicating that CBT was reset, but I'm not sure if it tells in the statistics window?

[kwells]

I tried the manual reset method of shutting down vm changing settings to false ....etc, and when the backup jub started I got the following message in the job summary:-

Cannot use CBT: Soap fault. Error caused by file /vmfs/volumes/0853ac7d-d1c8ad7a/vm-name/vm-name.vmdkDetail: '', endpoint: ''

Does this mean it has not restarted CBT or has something gone wrong?

[MrSpock]

I tried the manual reset method of shutting down vm changing settings to false ....etc, and when the backup jub started I got the following message in the job summary:-

Cannot use CBT: Soap fault. Error caused by file /vmfs/volumes/0853ac7d-d1c8ad7a/vm-name/vm-name.vmdkDetail: '', endpoint: ''

Does this mean it has not restarted CBT or has something gone wrong?

I got the same error on one host. Solution: http://www.veeam.com/kb1113

Best regards,

Johan

[andymcknight]

I think if the VM has multiple volumes that match, this will run multiple times for that VM. I believe you could do that last foreach like this:

Code: Select all

foreach($vm in $uniquevms){

And it would only run once per VM. I'm not entirely sure what Andy's

Code: Select all

$uniquevms = $problemvms | Group-Object Name | Measure-Object

line does though. I didn't have many affected, so I didn't test it that way. In a large environment, you might want to only run it once per VM.

That's not going to work. The $uniquevms variable is a measurement object, it's not an array of vms you can work with. The script I posted is a quick and dirty thing to try to understand the potential scope of the problem in your own environment, it doesn't really suit being automated into a fully scripted solution. You're better using the script posted by Veeam at http://www.veeam.com/kb1940 if you want to automate it.

[ditro]

HI @ll,
I've read trough the forum quite a while.
If I use the script posted by Veeam at http://www.veeam.com/kb1940 is there a way to be sure that the CBT was resetted successfully?
I tried this with one VM and there is no indication that the CBT was resetted.
I see only the that CBT is used.
The harddisks are very small and the backup target is really slow - so there is no real indication that the script has resetted the cbt and veeam is building it from scratch.

Can anyone proof that this script is working 100% and that after using the script and do a backup again I will not run into the CBT Bug from VMware?

[joergr]

ditro: You will experience the same time level you´d see when using an active full. now i understand you want 100% proof. Fine This is how: After you applied the script and the snapshot was taken and removed take a look in the vm´s folder using vsphere client. the ctk files must be gone.

they are gone? good - here is your proof. no ctk files - no cbt.

after the next veeam backup where cbt is automatically re-enanbled the ctk files in the vm folder will be there again.

[DrColonel]

I've been in communication with both Veeam support and VMware since Gostev sent out the email about the issue. After several calls and emails I ended up resetting CBT on our affected VM's however just yesterday I received the following from VMware support:

"Hello,

Yes, I am suggesting that a new full backup will essentially mitigate the effects of the issue. By clearing the CBT file, which is essentially a tracking file of what blocks to backup for an incremental backup, it will clear the issue that would have occurred if you grew the datastore past the 128 GB boundary between full backups. During this grow operation, the CBT tracking file is not updated properly. This currently is the best guidance I have from Engineering.

Right now, the best KB that supports how CBT functions is in KB 1020128 (Changed Block Tracking (CBT) on virtual machines)
http://kb.vmware.com/kb/1020128

Regards,

Technical Support Engineer
Global Support Services, VMware Inc.
1-877-4-VMWARE (1-877-486-9273)"

That seems to tell me that this was really a non-issue for those of us that run standard incremntals / frequent full backups. Has anyone else received similar info?

[cliffm]

I'm confused about this CBT issue.
If I run a full backup this is a non issue?
If I check my backups with SureBackup and they succeed this is a non issue?
What about replicas, there is no Sure Replica though I would dearly like one. I depend on replicas for failover. How do I know my replicas going back several months are any good? I'm not going to sit there and try to restore various items from them all one by one.
Can you give a firm date when a patch will deal with this?
Thanks

[chrisdearden]

there is no Sure Replica

Sure Replica is in v7.