-
- VP, Product Management
- Posts: 27342
- Liked: 2785 times
- Joined: Mar 30, 2009 9:13 am
- Full Name: Vitaliy Safarov
- Contact:
Re: VMware CBT bug KB 2090639
Hi Kevin,
Try the instructions posted in this blog post > http://pipitone.co/2014/02/09/reset-cbt-script/
Thanks!
Try the instructions posted in this blog post > http://pipitone.co/2014/02/09/reset-cbt-script/
Thanks!
-
- Novice
- Posts: 8
- Liked: never
- Joined: Jan 07, 2014 5:20 pm
- Full Name: Kevin Wells
- Contact:
Re: VMware CBT bug KB 2090639
Thank you for the link. I have looked at it, and am still confused. The script listed seems to have the same lines listed in it in several places. Is the powershell ISE environment the standard windows powershell and not the powercli environment (sorry, but I am a total novice when it comes to powercli as I have never had to use it).
-
- Influencer
- Posts: 18
- Liked: 1 time
- Joined: Aug 26, 2014 10:10 am
- Full Name: James Serjeant
- Contact:
Re: VMware CBT bug KB 2090639
We have reset CBT against a number of VMs and have one quick question.
Firstly, I understand that after disabling/enabling CBT the next backup will scan the VMDKs entirely but only copy the missing deltas to the backup repository. The advice is to then run a full backup for the next job to start afresh.
However, if for whatever reason a full job wasn't possible and the next job after the CBT reset was only a differential, would the newly created CBT data still be accurate for future jobs?
I'm guessing CBT would still be accurate for future jobs, but the drawback with only running a differential following a CBT reset would be that only the latest differential could be trusted, and the rest of the chain may be inconsistent. Is this correct or does it mean that in order for CBT to be trusted you must run a Full (NOT a Diff) immediately after the CBT reset?
Firstly, I understand that after disabling/enabling CBT the next backup will scan the VMDKs entirely but only copy the missing deltas to the backup repository. The advice is to then run a full backup for the next job to start afresh.
However, if for whatever reason a full job wasn't possible and the next job after the CBT reset was only a differential, would the newly created CBT data still be accurate for future jobs?
I'm guessing CBT would still be accurate for future jobs, but the drawback with only running a differential following a CBT reset would be that only the latest differential could be trusted, and the rest of the chain may be inconsistent. Is this correct or does it mean that in order for CBT to be trusted you must run a Full (NOT a Diff) immediately after the CBT reset?
-
- Novice
- Posts: 7
- Liked: 2 times
- Joined: May 21, 2013 5:09 pm
- Full Name: Brandon Patton
- Location: Indiana
- Contact:
Re: VMware CBT bug KB 2090639
Subscribing to thread
-
- Influencer
- Posts: 23
- Liked: 2 times
- Joined: Jun 20, 2009 10:43 am
- Full Name: Joe Iovinelli
- Contact:
Re: VMware CBT bug KB 2090639
Subscribing to thread
-
- Lurker
- Posts: 1
- Liked: never
- Joined: Jun 26, 2013 8:18 pm
- Full Name: scott d
- Contact:
Re: VMware CBT bug KB 2090639
We have been experiencing CBT errors for quite a while during replications and backups. Sometimes these are just warnings, sometimes the job fails and the retry is successful. I have had several tickets with Veeam on this issue and we also had VMware involved at one point. We went through all the reset CBT processes. This was happening to only one of our vms and even cloning the vm would not fix the issue. I am now worried seeing this thread that our replications and/or backups might be corrupted due to this issue although we have on several occasions done failovers of replication and tested full backup restores that seem ok, but how would I really know for sure?
We are running under Redhat 6x and also are using a D3 multi-value database and these CBT errors only occur on the DB partitions which is strange although they are the largest. We are now having this issue on other vms and it's random not always the same vm.
Is there a way to determine our backups/replicas are not corrupt? if this was posted here and I missed it, sorry for the redundant question.
We are running under Redhat 6x and also are using a D3 multi-value database and these CBT errors only occur on the DB partitions which is strange although they are the largest. We are now having this issue on other vms and it's random not always the same vm.
Is there a way to determine our backups/replicas are not corrupt? if this was posted here and I missed it, sorry for the redundant question.
-
- Novice
- Posts: 5
- Liked: 1 time
- Joined: Dec 04, 2013 4:23 pm
- Contact:
Re: VMware CBT bug KB 2090639
We have schedule "active full backups" to run weekly..
"active full backups" reset CBT is right ?
"active full backups" reset CBT is right ?
-
- Expert
- Posts: 227
- Liked: 62 times
- Joined: Apr 10, 2014 4:13 pm
- Contact:
Re: VMware CBT bug KB 2090639
No, the CBT must be reset independently.rei.andrea wrote:We have schedule "active full backups" to run weekly..
"active full backups" reset CBT is right ?
-
- Novice
- Posts: 5
- Liked: 1 time
- Joined: Dec 04, 2013 4:23 pm
- Contact:
Re: VMware CBT bug KB 2090639
OK I am planning a reset of CBT when possible...lightsout wrote: No, the CBT must be reset independently.
My question is:
the "active full backups" (NOT Synthetic Full Backup) [.vbk] are not affected to this Bug and are consinstant ?
the "active full backups" it should not use CBT It's perform a full read on whole VM right ?
-
- Novice
- Posts: 3
- Liked: never
- Joined: Apr 26, 2013 7:37 am
- Full Name: Andrew McKnight
- Contact:
Re: VMware CBT bug KB 2090639
For those of you who don't like the idea of fully scripting the take snapshot, remove snapshot method of resetting CBT, I've modified the script a little so that it gives you a list of VMs you have that have CBT enabled and a hard disk over 128GB in size but doesn't take any further action. That way you can do the snapshot part manually if you feel better about that. This doesn't tell you if you have extended the drives at any point but this may be useful to narrow down your search a little.
Code: Select all
$problemvms = @()
$cbtenabledvms = Get-VM | Where {$_.Extensiondata.Config.ChangeTrackingEnabled -eq 'True'}
Foreach ($vm in $cbtenabledvms)
{
Foreach ($disk in $vm.Guest.Disks)
{
If ($disk.CapacityGB -gt 128)
{
$problemvms += $vm
}
}
}
$uniquevms = $problemvms | Group-Object Name | Measure-Object
$problemvms
Write-Host("Count of problem VMs: " + $uniquevms.Count)
-
- Chief Product Officer
- Posts: 31748
- Liked: 7251 times
- Joined: Jan 01, 2006 1:01 am
- Location: Baar, Switzerland
- Contact:
Re: VMware CBT bug KB 2090639
Active Full backup does uses CBT information, so it is affected.rei.andrea wrote:My question is:
the "active full backups" (NOT Synthetic Full Backup) [.vbk] are not affected to this Bug and are consinstant ?
the "active full backups" it should not use CBT It's perform a full read on whole VM right ?
Resetting CBT is a must no matter of backup mode or schedule.
-
- Chief Product Officer
- Posts: 31748
- Liked: 7251 times
- Joined: Jan 01, 2006 1:01 am
- Location: Baar, Switzerland
- Contact:
Re: VMware CBT bug KB 2090639
In our testing, at least in case of Windows VMs, Instant VM Recovery from an incremental restore point taken after virtual disk extension that breaks CBT has been a good simple test. In case of expansion of the OS disk, OS simply fails to boot. In case of expansion of the data disk, OS looses volumes located on the disk (and disk shows up as offline in the Disk Management snap-in). So, simple Instant VM Recovery from backup with network disconnected (or test replica failover) followed by manual inspection of disks inside a VM should help to identify the issue in many cases in our experience. This also means that anyone using SureBackup and getting successes is unlikely to have issues with the current backups and replicas, as either issue mentioned above will cause a failed SureBackup test (either due to no heartbeat, or failed application test for apps using data disks).sedrum wrote:Is there a way to determine our backups/replicas are not corrupt? if this was posted here and I missed it, sorry for the redundant question.
But again, due to not having access to VMware source code, we have no insight of CBT internals and how this bug breaks CBT logic. As such, we cannot guarantee that the test above is sufficient in every scenario CBT corruption scenario and for every OS. And because this is impossible to test in a reasonable time frame, we highly recommend to reset CBT for all VMs with disks over 128GB in size, as a matter of precaution, and run the jobs at least once to create the new restore point that will be guaranteed to be consistent.
-
- Chief Product Officer
- Posts: 31748
- Liked: 7251 times
- Joined: Jan 01, 2006 1:01 am
- Location: Baar, Switzerland
- Contact:
Re: VMware CBT bug KB 2090639
To subscribe to this topic, please use the Subscribe topic link at the bottom of this page. Posting a comment will NOT subscribe you to the post with this board (this will do nothing except including the topic into your personalized weekly digest, under personal contents).
-
- Lurker
- Posts: 2
- Liked: never
- Joined: Sep 03, 2014 5:55 pm
- Full Name: Parker Jardine
- Contact:
Re: VMware CBT bug KB 2090639
How about just unchecking the CBT option within the veeam job? Then running the job without cbt enabled. And then checking cbt in the job after a successful backup without cbt? Or do we still have to disable cbt on the vmware side of things?
-
- Enthusiast
- Posts: 56
- Liked: 5 times
- Joined: Mar 23, 2010 1:59 pm
- Full Name: Jason Lester
- Contact:
Re: VMware CBT bug KB 2090639
Thanks to Veeam and andymcnight for the PowerCLI scripts. I combined the two to only make the changes for VM's that seemed to be affected:
In case you are a newbie to PowerCLI like me, the foreach loops will not run until you hit enter on a blank space after the trailing }. Took me a few minutes to figure that out. YMMV, so use caution before using in your environment.
Jason
Code: Select all
$problemvms = @()
$cbtenabledvms = Get-VM | Where {$_.Extensiondata.Config.ChangeTrackingEnabled -eq 'True'}
Foreach ($vm in $cbtenabledvms) {Foreach ($disk in $vm.Guest.Disks){If ($disk.CapacityGB -gt 128){$problemvms += $vm}}}
$uniquevms = $problemvms | Group-Object Name | Measure-Object
$problemvms
Write-Host("Count of problem VMs: " + $uniquevms.Count)
foreach($vm in $problemvms){
$vm.ExtensionData.ReconfigVM($spec)
$snap=$vm | New-Snapshot -Name 'Disable CBT'
$snap | Remove-Snapshot -confirm:$false }
Jason
-
- Expert
- Posts: 227
- Liked: 62 times
- Joined: Apr 10, 2014 4:13 pm
- Contact:
Re: VMware CBT bug KB 2090639
Thats a nice script. Just don't forget to define $spec.jlester wrote:Thanks to Veeam and andymcnight for the PowerCLI scripts. I combined the two to only make the changes for VM's that seemed to be affected:
In case you are a newbie to PowerCLI like me, the foreach loops will not run until you hit enter on a blank space after the trailing }. Took me a few minutes to figure that out. YMMV, so use caution before using in your environment.Code: Select all
$problemvms = @() $cbtenabledvms = Get-VM | Where {$_.Extensiondata.Config.ChangeTrackingEnabled -eq 'True'} Foreach ($vm in $cbtenabledvms) {Foreach ($disk in $vm.Guest.Disks){If ($disk.CapacityGB -gt 128){$problemvms += $vm}}} $uniquevms = $problemvms | Group-Object Name | Measure-Object $problemvms Write-Host("Count of problem VMs: " + $uniquevms.Count) foreach($vm in $problemvms){ $vm.ExtensionData.ReconfigVM($spec) $snap=$vm | New-Snapshot -Name 'Disable CBT' $snap | Remove-Snapshot -confirm:$false }
Jason
Code: Select all
$spec = New-Object VMware.Vim.VirtualMachineConfigSpec
$spec.ChangeTrackingEnabled = $false
-
- Enthusiast
- Posts: 56
- Liked: 5 times
- Joined: Mar 23, 2010 1:59 pm
- Full Name: Jason Lester
- Contact:
Re: VMware CBT bug KB 2090639
Whoops, you're right:
I think if the VM has multiple volumes that match, this will run multiple times for that VM. I believe you could do that last foreach like this:
And it would only run once per VM. I'm not entirely sure what Andy's line does though. I didn't have many affected, so I didn't test it that way. In a large environment, you might want to only run it once per VM.
Code: Select all
$problemvms = @()
$cbtenabledvms = Get-VM | Where {$_.Extensiondata.Config.ChangeTrackingEnabled -eq 'True'}
Foreach ($vm in $cbtenabledvms) {Foreach ($disk in $vm.Guest.Disks){If ($disk.CapacityGB -gt 128){$problemvms += $vm}}}
$uniquevms = $problemvms | Group-Object Name | Measure-Object
$problemvms
Write-Host("Count of problem VMs: " + $uniquevms.Count)
$spec = New-Object VMware.Vim.VirtualMachineConfigSpec
$spec.ChangeTrackingEnabled = $false
foreach($vm in $problemvms){
$vm.ExtensionData.ReconfigVM($spec)
$snap=$vm | New-Snapshot -Name 'Disable CBT'
$snap | Remove-Snapshot -confirm:$false }
Code: Select all
foreach($vm in $uniquevms){
Code: Select all
$uniquevms = $problemvms | Group-Object Name | Measure-Object
-
- Chief Product Officer
- Posts: 31748
- Liked: 7251 times
- Joined: Jan 01, 2006 1:01 am
- Location: Baar, Switzerland
- Contact:
Re: VMware CBT bug KB 2090639
Yes, CBT must be reset on VMware side.REuser wrote:How about just unchecking the CBT option within the veeam job? Then running the job without cbt enabled. And then checking cbt in the job after a successful backup without cbt? Or do we still have to disable cbt on the vmware side of things?
-
- Expert
- Posts: 248
- Liked: 38 times
- Joined: Jun 15, 2009 10:49 am
- Full Name: Gabrie van Zanten
- Contact:
Re: VMware CBT bug KB 2090639
Looking at the script from JLESTER. Is confirmed that the following procedure resets the CBT and will cause NO downtime to VMs? And if confirmed, is it also "official" by VMware?
- Powershell -> set trackingenabled to false
- Create snapshot
- Remove snapshot
- Perform a backup which will again enabled CBT and do a full scan.
- Powershell -> set trackingenabled to false
- Create snapshot
- Remove snapshot
- Perform a backup which will again enabled CBT and do a full scan.
-
- Enthusiast
- Posts: 56
- Liked: 5 times
- Joined: Mar 23, 2010 1:59 pm
- Full Name: Jason Lester
- Contact:
Re: VMware CBT bug KB 2090639
Gabe,
Did you see the Veeam KB article that was linked earlier:
http://www.veeam.com/kb1940
Their example script resets it for every VM with CBT enabled. So, I think that is the procedure and it worked with no downtime in my environment. My nightly backups kickoff in about an hour, so I'll know tomorrow if the affected VM's did a full scan.
Jason
Did you see the Veeam KB article that was linked earlier:
http://www.veeam.com/kb1940
Their example script resets it for every VM with CBT enabled. So, I think that is the procedure and it worked with no downtime in my environment. My nightly backups kickoff in about an hour, so I'll know tomorrow if the affected VM's did a full scan.
Jason
-
- Expert
- Posts: 248
- Liked: 38 times
- Joined: Jun 15, 2009 10:49 am
- Full Name: Gabrie van Zanten
- Contact:
Re: VMware CBT bug KB 2090639
Thanks Jason !!!jlester wrote:Gabe,
Did you see the Veeam KB article that was linked earlier:
http://www.veeam.com/kb1940
Their example script resets it for every VM with CBT enabled. So, I think that is the procedure and it worked with no downtime in my environment. My nightly backups kickoff in about an hour, so I'll know tomorrow if the affected VM's did a full scan.
Jason
-
- Enthusiast
- Posts: 56
- Liked: 5 times
- Joined: Mar 23, 2010 1:59 pm
- Full Name: Jason Lester
- Contact:
Re: VMware CBT bug KB 2090639
Is there an easy way to tell if CBT was reset from the backup statistics? Some of the affected VM's have backed up alreadt tonight and still show [CBT] beside the disks in the statistics. I don't see anything indicating that CBT was reset, but I'm not sure if it tells in the statistics window?
-
- Novice
- Posts: 8
- Liked: never
- Joined: Jan 07, 2014 5:20 pm
- Full Name: Kevin Wells
- Contact:
Re: VMware CBT bug KB 2090639
I tried the manual reset method of shutting down vm changing settings to false ....etc, and when the backup jub started I got the following message in the job summary:-
Cannot use CBT: Soap fault. Error caused by file /vmfs/volumes/0853ac7d-d1c8ad7a/vm-name/vm-name.vmdkDetail: '', endpoint: ''
Does this mean it has not restarted CBT or has something gone wrong?
Cannot use CBT: Soap fault. Error caused by file /vmfs/volumes/0853ac7d-d1c8ad7a/vm-name/vm-name.vmdkDetail: '', endpoint: ''
Does this mean it has not restarted CBT or has something gone wrong?
-
- Service Provider
- Posts: 49
- Liked: 3 times
- Joined: Apr 24, 2009 10:16 pm
- Contact:
Re: VMware CBT bug KB 2090639
I got the same error on one host. Solution: http://www.veeam.com/kb1113kwells wrote:I tried the manual reset method of shutting down vm changing settings to false ....etc, and when the backup jub started I got the following message in the job summary:-
Cannot use CBT: Soap fault. Error caused by file /vmfs/volumes/0853ac7d-d1c8ad7a/vm-name/vm-name.vmdkDetail: '', endpoint: ''
Does this mean it has not restarted CBT or has something gone wrong?
Best regards,
Johan
-
- Novice
- Posts: 3
- Liked: never
- Joined: Apr 26, 2013 7:37 am
- Full Name: Andrew McKnight
- Contact:
Re: VMware CBT bug KB 2090639
That's not going to work. The $uniquevms variable is a measurement object, it's not an array of vms you can work with. The script I posted is a quick and dirty thing to try to understand the potential scope of the problem in your own environment, it doesn't really suit being automated into a fully scripted solution. You're better using the script posted by Veeam at http://www.veeam.com/kb1940 if you want to automate it.I think if the VM has multiple volumes that match, this will run multiple times for that VM. I believe you could do that last foreach like this:
And it would only run once per VM. I'm not entirely sure what Andy'sCode: Select all
foreach($vm in $uniquevms){
line does though. I didn't have many affected, so I didn't test it that way. In a large environment, you might want to only run it once per VM.Code: Select all
$uniquevms = $problemvms | Group-Object Name | Measure-Object
-
- Novice
- Posts: 3
- Liked: never
- Joined: Jan 05, 2011 10:27 am
- Contact:
Re: VMware CBT bug KB 2090639
HI @ll,
I've read trough the forum quite a while.
If I use the script posted by Veeam at http://www.veeam.com/kb1940 is there a way to be sure that the CBT was resetted successfully?
I tried this with one VM and there is no indication that the CBT was resetted.
I see only the that CBT is used.
The harddisks are very small and the backup target is really slow - so there is no real indication that the script has resetted the cbt and veeam is building it from scratch.
Can anyone proof that this script is working 100% and that after using the script and do a backup again I will not run into the CBT Bug from VMware?
I've read trough the forum quite a while.
If I use the script posted by Veeam at http://www.veeam.com/kb1940 is there a way to be sure that the CBT was resetted successfully?
I tried this with one VM and there is no indication that the CBT was resetted.
I see only the that CBT is used.
The harddisks are very small and the backup target is really slow - so there is no real indication that the script has resetted the cbt and veeam is building it from scratch.
Can anyone proof that this script is working 100% and that after using the script and do a backup again I will not run into the CBT Bug from VMware?
-
- Veteran
- Posts: 391
- Liked: 39 times
- Joined: Jun 08, 2010 2:01 pm
- Full Name: Joerg Riether
- Contact:
Re: VMware CBT bug KB 2090639
ditro: You will experience the same time level you´d see when using an active full. now i understand you want 100% proof. Fine This is how: After you applied the script and the snapshot was taken and removed take a look in the vm´s folder using vsphere client. the ctk files must be gone.
they are gone? good - here is your proof. no ctk files - no cbt.
after the next veeam backup where cbt is automatically re-enanbled the ctk files in the vm folder will be there again.
they are gone? good - here is your proof. no ctk files - no cbt.
after the next veeam backup where cbt is automatically re-enanbled the ctk files in the vm folder will be there again.
-
- Novice
- Posts: 5
- Liked: never
- Joined: Apr 29, 2014 8:00 pm
- Full Name: Kyle Morrow
- Contact:
Re: VMware CBT bug KB 2090639
I've been in communication with both Veeam support and VMware since Gostev sent out the email about the issue. After several calls and emails I ended up resetting CBT on our affected VM's however just yesterday I received the following from VMware support:
"Hello,
Yes, I am suggesting that a new full backup will essentially mitigate the effects of the issue. By clearing the CBT file, which is essentially a tracking file of what blocks to backup for an incremental backup, it will clear the issue that would have occurred if you grew the datastore past the 128 GB boundary between full backups. During this grow operation, the CBT tracking file is not updated properly. This currently is the best guidance I have from Engineering.
Right now, the best KB that supports how CBT functions is in KB 1020128 (Changed Block Tracking (CBT) on virtual machines)
http://kb.vmware.com/kb/1020128
Regards,
Technical Support Engineer
Global Support Services, VMware Inc.
1-877-4-VMWARE (1-877-486-9273)"
That seems to tell me that this was really a non-issue for those of us that run standard incremntals / frequent full backups. Has anyone else received similar info?
"Hello,
Yes, I am suggesting that a new full backup will essentially mitigate the effects of the issue. By clearing the CBT file, which is essentially a tracking file of what blocks to backup for an incremental backup, it will clear the issue that would have occurred if you grew the datastore past the 128 GB boundary between full backups. During this grow operation, the CBT tracking file is not updated properly. This currently is the best guidance I have from Engineering.
Right now, the best KB that supports how CBT functions is in KB 1020128 (Changed Block Tracking (CBT) on virtual machines)
http://kb.vmware.com/kb/1020128
Regards,
Technical Support Engineer
Global Support Services, VMware Inc.
1-877-4-VMWARE (1-877-486-9273)"
That seems to tell me that this was really a non-issue for those of us that run standard incremntals / frequent full backups. Has anyone else received similar info?
-
- Enthusiast
- Posts: 41
- Liked: 4 times
- Joined: Jun 03, 2011 12:41 am
- Full Name: Cliff Meakin
- Contact:
Re: VMware CBT bug KB 2090639
I'm confused about this CBT issue.
If I run a full backup this is a non issue?
If I check my backups with SureBackup and they succeed this is a non issue?
What about replicas, there is no Sure Replica though I would dearly like one. I depend on replicas for failover. How do I know my replicas going back several months are any good? I'm not going to sit there and try to restore various items from them all one by one.
Can you give a firm date when a patch will deal with this?
Thanks
If I run a full backup this is a non issue?
If I check my backups with SureBackup and they succeed this is a non issue?
What about replicas, there is no Sure Replica though I would dearly like one. I depend on replicas for failover. How do I know my replicas going back several months are any good? I'm not going to sit there and try to restore various items from them all one by one.
Can you give a firm date when a patch will deal with this?
Thanks
-
- Veteran
- Posts: 1531
- Liked: 226 times
- Joined: Jul 21, 2010 9:47 am
- Full Name: Chris Dearden
- Contact:
Re: VMware CBT bug KB 2090639
Sure Replica is in v7.there is no Sure Replica
Who is online
Users browsing this forum: No registered users and 77 guests