Host-based backup of VMware vSphere VMs.
Locked
Vitaliy S.
VP, Product Management
Posts: 27055
Liked: 2710 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: VMware CBT bug KB 2090639

Post by Vitaliy S. »

Hi Kevin,

Try the instructions posted in this blog post > http://pipitone.co/2014/02/09/reset-cbt-script/

Thanks!
kwells
Novice
Posts: 8
Liked: never
Joined: Jan 07, 2014 5:20 pm
Full Name: Kevin Wells
Contact:

Re: VMware CBT bug KB 2090639

Post by kwells »

Thank you for the link. I have looked at it, and am still confused. The script listed seems to have the same lines listed in it in several places. Is the powershell ISE environment the standard windows powershell and not the powercli environment (sorry, but I am a total novice when it comes to powercli as I have never had to use it).
orangep7
Influencer
Posts: 18
Liked: 1 time
Joined: Aug 26, 2014 10:10 am
Full Name: James Serjeant
Contact:

Re: VMware CBT bug KB 2090639

Post by orangep7 »

We have reset CBT against a number of VMs and have one quick question.

Firstly, I understand that after disabling/enabling CBT the next backup will scan the VMDKs entirely but only copy the missing deltas to the backup repository. The advice is to then run a full backup for the next job to start afresh.

However, if for whatever reason a full job wasn't possible and the next job after the CBT reset was only a differential, would the newly created CBT data still be accurate for future jobs?

I'm guessing CBT would still be accurate for future jobs, but the drawback with only running a differential following a CBT reset would be that only the latest differential could be trusted, and the rest of the chain may be inconsistent. Is this correct or does it mean that in order for CBT to be trusted you must run a Full (NOT a Diff) immediately after the CBT reset?
brandon.patton
Novice
Posts: 7
Liked: 2 times
Joined: May 21, 2013 5:09 pm
Full Name: Brandon Patton
Location: Indiana
Contact:

Re: VMware CBT bug KB 2090639

Post by brandon.patton »

Subscribing to thread
jai64
Influencer
Posts: 23
Liked: 2 times
Joined: Jun 20, 2009 10:43 am
Full Name: Joe Iovinelli
Contact:

Re: VMware CBT bug KB 2090639

Post by jai64 »

Subscribing to thread
sedrum
Lurker
Posts: 1
Liked: never
Joined: Jun 26, 2013 8:18 pm
Full Name: scott d
Contact:

Re: VMware CBT bug KB 2090639

Post by sedrum »

We have been experiencing CBT errors for quite a while during replications and backups. Sometimes these are just warnings, sometimes the job fails and the retry is successful. I have had several tickets with Veeam on this issue and we also had VMware involved at one point. We went through all the reset CBT processes. This was happening to only one of our vms and even cloning the vm would not fix the issue. I am now worried seeing this thread that our replications and/or backups might be corrupted due to this issue although we have on several occasions done failovers of replication and tested full backup restores that seem ok, but how would I really know for sure?
We are running under Redhat 6x and also are using a D3 multi-value database and these CBT errors only occur on the DB partitions which is strange although they are the largest. We are now having this issue on other vms and it's random not always the same vm.

Is there a way to determine our backups/replicas are not corrupt? if this was posted here and I missed it, sorry for the redundant question.
rei.andrea
Novice
Posts: 5
Liked: 1 time
Joined: Dec 04, 2013 4:23 pm
Contact:

Re: VMware CBT bug KB 2090639

Post by rei.andrea »

We have schedule "active full backups" to run weekly..

"active full backups" reset CBT is right ?
lightsout
Expert
Posts: 227
Liked: 62 times
Joined: Apr 10, 2014 4:13 pm
Contact:

Re: VMware CBT bug KB 2090639

Post by lightsout »

rei.andrea wrote:We have schedule "active full backups" to run weekly..

"active full backups" reset CBT is right ?
No, the CBT must be reset independently.
rei.andrea
Novice
Posts: 5
Liked: 1 time
Joined: Dec 04, 2013 4:23 pm
Contact:

Re: VMware CBT bug KB 2090639

Post by rei.andrea »

lightsout wrote: No, the CBT must be reset independently.
OK I am planning a reset of CBT when possible...

My question is:
the "active full backups" (NOT Synthetic Full Backup) [.vbk] are not affected to this Bug and are consinstant ?
the "active full backups" it should not use CBT It's perform a full read on whole VM right ?
andymcknight
Novice
Posts: 3
Liked: never
Joined: Apr 26, 2013 7:37 am
Full Name: Andrew McKnight
Contact:

Re: VMware CBT bug KB 2090639

Post by andymcknight »

For those of you who don't like the idea of fully scripting the take snapshot, remove snapshot method of resetting CBT, I've modified the script a little so that it gives you a list of VMs you have that have CBT enabled and a hard disk over 128GB in size but doesn't take any further action. That way you can do the snapshot part manually if you feel better about that. This doesn't tell you if you have extended the drives at any point but this may be useful to narrow down your search a little.

Code: Select all

$problemvms = @()
$cbtenabledvms = Get-VM | Where {$_.Extensiondata.Config.ChangeTrackingEnabled -eq 'True'}
Foreach ($vm in $cbtenabledvms)
	{
		Foreach ($disk in $vm.Guest.Disks)
			{
				If ($disk.CapacityGB -gt 128)
					{
						$problemvms += $vm
					}
			}
	}
$uniquevms = $problemvms | Group-Object Name | Measure-Object
$problemvms
Write-Host("Count of problem VMs: " + $uniquevms.Count)
Gostev
Chief Product Officer
Posts: 31457
Liked: 6647 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: VMware CBT bug KB 2090639

Post by Gostev »

rei.andrea wrote:My question is:
the "active full backups" (NOT Synthetic Full Backup) [.vbk] are not affected to this Bug and are consinstant ?
the "active full backups" it should not use CBT It's perform a full read on whole VM right ?
Active Full backup does uses CBT information, so it is affected.
Resetting CBT is a must no matter of backup mode or schedule.
Gostev
Chief Product Officer
Posts: 31457
Liked: 6647 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: VMware CBT bug KB 2090639

Post by Gostev »

sedrum wrote:Is there a way to determine our backups/replicas are not corrupt? if this was posted here and I missed it, sorry for the redundant question.
In our testing, at least in case of Windows VMs, Instant VM Recovery from an incremental restore point taken after virtual disk extension that breaks CBT has been a good simple test. In case of expansion of the OS disk, OS simply fails to boot. In case of expansion of the data disk, OS looses volumes located on the disk (and disk shows up as offline in the Disk Management snap-in). So, simple Instant VM Recovery from backup with network disconnected (or test replica failover) followed by manual inspection of disks inside a VM should help to identify the issue in many cases in our experience. This also means that anyone using SureBackup and getting successes is unlikely to have issues with the current backups and replicas, as either issue mentioned above will cause a failed SureBackup test (either due to no heartbeat, or failed application test for apps using data disks).

But again, due to not having access to VMware source code, we have no insight of CBT internals and how this bug breaks CBT logic. As such, we cannot guarantee that the test above is sufficient in every scenario CBT corruption scenario and for every OS. And because this is impossible to test in a reasonable time frame, we highly recommend to reset CBT for all VMs with disks over 128GB in size, as a matter of precaution, and run the jobs at least once to create the new restore point that will be guaranteed to be consistent.
Gostev
Chief Product Officer
Posts: 31457
Liked: 6647 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: VMware CBT bug KB 2090639

Post by Gostev »

To subscribe to this topic, please use the Subscribe topic link at the bottom of this page. Posting a comment will NOT subscribe you to the post with this board (this will do nothing except including the topic into your personalized weekly digest, under personal contents).
REuser
Lurker
Posts: 2
Liked: never
Joined: Sep 03, 2014 5:55 pm
Full Name: Parker Jardine
Contact:

Re: VMware CBT bug KB 2090639

Post by REuser »

How about just unchecking the CBT option within the veeam job? Then running the job without cbt enabled. And then checking cbt in the job after a successful backup without cbt? Or do we still have to disable cbt on the vmware side of things?
jlester
Enthusiast
Posts: 56
Liked: 5 times
Joined: Mar 23, 2010 1:59 pm
Full Name: Jason Lester
Contact:

Re: VMware CBT bug KB 2090639

Post by jlester »

Thanks to Veeam and andymcnight for the PowerCLI scripts. I combined the two to only make the changes for VM's that seemed to be affected:

Code: Select all

    $problemvms = @()
    $cbtenabledvms = Get-VM | Where {$_.Extensiondata.Config.ChangeTrackingEnabled -eq 'True'}
    Foreach ($vm in $cbtenabledvms)       {Foreach ($disk in $vm.Guest.Disks){If ($disk.CapacityGB -gt 128){$problemvms += $vm}}}

    $uniquevms = $problemvms | Group-Object Name | Measure-Object
    $problemvms
    Write-Host("Count of problem VMs: " + $uniquevms.Count)
	
    foreach($vm in $problemvms){
    $vm.ExtensionData.ReconfigVM($spec)
    $snap=$vm | New-Snapshot -Name 'Disable CBT'
    $snap | Remove-Snapshot -confirm:$false }

In case you are a newbie to PowerCLI like me, the foreach loops will not run until you hit enter on a blank space after the trailing }. Took me a few minutes to figure that out. YMMV, so use caution before using in your environment.

Jason
lightsout
Expert
Posts: 227
Liked: 62 times
Joined: Apr 10, 2014 4:13 pm
Contact:

Re: VMware CBT bug KB 2090639

Post by lightsout »

jlester wrote:Thanks to Veeam and andymcnight for the PowerCLI scripts. I combined the two to only make the changes for VM's that seemed to be affected:

Code: Select all

    $problemvms = @()
    $cbtenabledvms = Get-VM | Where {$_.Extensiondata.Config.ChangeTrackingEnabled -eq 'True'}
    Foreach ($vm in $cbtenabledvms)       {Foreach ($disk in $vm.Guest.Disks){If ($disk.CapacityGB -gt 128){$problemvms += $vm}}}

    $uniquevms = $problemvms | Group-Object Name | Measure-Object
    $problemvms
    Write-Host("Count of problem VMs: " + $uniquevms.Count)
	
    foreach($vm in $problemvms){
    $vm.ExtensionData.ReconfigVM($spec)
    $snap=$vm | New-Snapshot -Name 'Disable CBT'
    $snap | Remove-Snapshot -confirm:$false }

In case you are a newbie to PowerCLI like me, the foreach loops will not run until you hit enter on a blank space after the trailing }. Took me a few minutes to figure that out. YMMV, so use caution before using in your environment.

Jason
Thats a nice script. Just don't forget to define $spec.

Code: Select all

$spec = New-Object VMware.Vim.VirtualMachineConfigSpec 
$spec.ChangeTrackingEnabled = $false
jlester
Enthusiast
Posts: 56
Liked: 5 times
Joined: Mar 23, 2010 1:59 pm
Full Name: Jason Lester
Contact:

Re: VMware CBT bug KB 2090639

Post by jlester »

Whoops, you're right:

Code: Select all

    $problemvms = @()
    $cbtenabledvms = Get-VM | Where {$_.Extensiondata.Config.ChangeTrackingEnabled -eq 'True'}
    Foreach ($vm in $cbtenabledvms)       {Foreach ($disk in $vm.Guest.Disks){If ($disk.CapacityGB -gt 128){$problemvms += $vm}}}

    $uniquevms = $problemvms | Group-Object Name | Measure-Object
    $problemvms
    Write-Host("Count of problem VMs: " + $uniquevms.Count)

    $spec = New-Object VMware.Vim.VirtualMachineConfigSpec 
    $spec.ChangeTrackingEnabled = $false

    foreach($vm in $problemvms){
    $vm.ExtensionData.ReconfigVM($spec)
    $snap=$vm | New-Snapshot -Name 'Disable CBT'
    $snap | Remove-Snapshot -confirm:$false }
I think if the VM has multiple volumes that match, this will run multiple times for that VM. I believe you could do that last foreach like this:

Code: Select all

foreach($vm in $uniquevms){
And it would only run once per VM. I'm not entirely sure what Andy's

Code: Select all

$uniquevms = $problemvms | Group-Object Name | Measure-Object
line does though. I didn't have many affected, so I didn't test it that way. In a large environment, you might want to only run it once per VM.
Gostev
Chief Product Officer
Posts: 31457
Liked: 6647 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: VMware CBT bug KB 2090639

Post by Gostev »

REuser wrote:How about just unchecking the CBT option within the veeam job? Then running the job without cbt enabled. And then checking cbt in the job after a successful backup without cbt? Or do we still have to disable cbt on the vmware side of things?
Yes, CBT must be reset on VMware side.
GabesVirtualWorld
Expert
Posts: 244
Liked: 37 times
Joined: Jun 15, 2009 10:49 am
Full Name: Gabrie van Zanten
Contact:

Re: VMware CBT bug KB 2090639

Post by GabesVirtualWorld »

Looking at the script from JLESTER. Is confirmed that the following procedure resets the CBT and will cause NO downtime to VMs? And if confirmed, is it also "official" by VMware?

- Powershell -> set trackingenabled to false
- Create snapshot
- Remove snapshot
- Perform a backup which will again enabled CBT and do a full scan.
jlester
Enthusiast
Posts: 56
Liked: 5 times
Joined: Mar 23, 2010 1:59 pm
Full Name: Jason Lester
Contact:

Re: VMware CBT bug KB 2090639

Post by jlester »

Gabe,

Did you see the Veeam KB article that was linked earlier:

http://www.veeam.com/kb1940

Their example script resets it for every VM with CBT enabled. So, I think that is the procedure and it worked with no downtime in my environment. My nightly backups kickoff in about an hour, so I'll know tomorrow if the affected VM's did a full scan.

Jason
GabesVirtualWorld
Expert
Posts: 244
Liked: 37 times
Joined: Jun 15, 2009 10:49 am
Full Name: Gabrie van Zanten
Contact:

Re: VMware CBT bug KB 2090639

Post by GabesVirtualWorld »

jlester wrote:Gabe,

Did you see the Veeam KB article that was linked earlier:

http://www.veeam.com/kb1940

Their example script resets it for every VM with CBT enabled. So, I think that is the procedure and it worked with no downtime in my environment. My nightly backups kickoff in about an hour, so I'll know tomorrow if the affected VM's did a full scan.

Jason
Thanks Jason !!!
jlester
Enthusiast
Posts: 56
Liked: 5 times
Joined: Mar 23, 2010 1:59 pm
Full Name: Jason Lester
Contact:

Re: VMware CBT bug KB 2090639

Post by jlester »

Is there an easy way to tell if CBT was reset from the backup statistics? Some of the affected VM's have backed up alreadt tonight and still show [CBT] beside the disks in the statistics. I don't see anything indicating that CBT was reset, but I'm not sure if it tells in the statistics window?
kwells
Novice
Posts: 8
Liked: never
Joined: Jan 07, 2014 5:20 pm
Full Name: Kevin Wells
Contact:

Re: VMware CBT bug KB 2090639

Post by kwells »

I tried the manual reset method of shutting down vm changing settings to false ....etc, and when the backup jub started I got the following message in the job summary:-

Cannot use CBT: Soap fault. Error caused by file /vmfs/volumes/0853ac7d-d1c8ad7a/vm-name/vm-name.vmdkDetail: '', endpoint: ''

Does this mean it has not restarted CBT or has something gone wrong?
MrSpock
Service Provider
Posts: 46
Liked: 3 times
Joined: Apr 24, 2009 10:16 pm
Contact:

Re: VMware CBT bug KB 2090639

Post by MrSpock »

kwells wrote:I tried the manual reset method of shutting down vm changing settings to false ....etc, and when the backup jub started I got the following message in the job summary:-

Cannot use CBT: Soap fault. Error caused by file /vmfs/volumes/0853ac7d-d1c8ad7a/vm-name/vm-name.vmdkDetail: '', endpoint: ''

Does this mean it has not restarted CBT or has something gone wrong?
I got the same error on one host. Solution: http://www.veeam.com/kb1113

Best regards,

Johan
andymcknight
Novice
Posts: 3
Liked: never
Joined: Apr 26, 2013 7:37 am
Full Name: Andrew McKnight
Contact:

Re: VMware CBT bug KB 2090639

Post by andymcknight »

I think if the VM has multiple volumes that match, this will run multiple times for that VM. I believe you could do that last foreach like this:

Code: Select all

foreach($vm in $uniquevms){
And it would only run once per VM. I'm not entirely sure what Andy's

Code: Select all

$uniquevms = $problemvms | Group-Object Name | Measure-Object
line does though. I didn't have many affected, so I didn't test it that way. In a large environment, you might want to only run it once per VM.
That's not going to work. The $uniquevms variable is a measurement object, it's not an array of vms you can work with. The script I posted is a quick and dirty thing to try to understand the potential scope of the problem in your own environment, it doesn't really suit being automated into a fully scripted solution. You're better using the script posted by Veeam at http://www.veeam.com/kb1940 if you want to automate it.
ditro
Novice
Posts: 3
Liked: never
Joined: Jan 05, 2011 10:27 am
Contact:

Re: VMware CBT bug KB 2090639

Post by ditro »

HI @ll,
I've read trough the forum quite a while.
If I use the script posted by Veeam at http://www.veeam.com/kb1940 is there a way to be sure that the CBT was resetted successfully?
I tried this with one VM and there is no indication that the CBT was resetted.
I see only the that CBT is used.
The harddisks are very small and the backup target is really slow - so there is no real indication that the script has resetted the cbt and veeam is building it from scratch.

Can anyone proof that this script is working 100% and that after using the script and do a backup again I will not run into the CBT Bug from VMware?
joergr
Veteran
Posts: 391
Liked: 39 times
Joined: Jun 08, 2010 2:01 pm
Full Name: Joerg Riether
Contact:

Re: VMware CBT bug KB 2090639

Post by joergr » 1 person likes this post

ditro: You will experience the same time level you´d see when using an active full. now i understand you want 100% proof. Fine ;-) This is how: After you applied the script and the snapshot was taken and removed take a look in the vm´s folder using vsphere client. the ctk files must be gone.

they are gone? good - here is your proof. no ctk files - no cbt.

after the next veeam backup where cbt is automatically re-enanbled the ctk files in the vm folder will be there again.
DrColonel
Novice
Posts: 5
Liked: never
Joined: Apr 29, 2014 8:00 pm
Full Name: Kyle Morrow
Contact:

Re: VMware CBT bug KB 2090639

Post by DrColonel »

I've been in communication with both Veeam support and VMware since Gostev sent out the email about the issue. After several calls and emails I ended up resetting CBT on our affected VM's however just yesterday I received the following from VMware support:

"Hello,

Yes, I am suggesting that a new full backup will essentially mitigate the effects of the issue. By clearing the CBT file, which is essentially a tracking file of what blocks to backup for an incremental backup, it will clear the issue that would have occurred if you grew the datastore past the 128 GB boundary between full backups. During this grow operation, the CBT tracking file is not updated properly. This currently is the best guidance I have from Engineering.

Right now, the best KB that supports how CBT functions is in KB 1020128 (Changed Block Tracking (CBT) on virtual machines)
http://kb.vmware.com/kb/1020128

Regards,

Technical Support Engineer
Global Support Services, VMware Inc.
1-877-4-VMWARE (1-877-486-9273)"


That seems to tell me that this was really a non-issue for those of us that run standard incremntals / frequent full backups. Has anyone else received similar info?
cliffm
Enthusiast
Posts: 41
Liked: 4 times
Joined: Jun 03, 2011 12:41 am
Full Name: Cliff Meakin
Contact:

Re: VMware CBT bug KB 2090639

Post by cliffm »

I'm confused about this CBT issue.
If I run a full backup this is a non issue?
If I check my backups with SureBackup and they succeed this is a non issue?
What about replicas, there is no Sure Replica though I would dearly like one. I depend on replicas for failover. How do I know my replicas going back several months are any good? I'm not going to sit there and try to restore various items from them all one by one.
Can you give a firm date when a patch will deal with this?
Thanks
chrisdearden
Veteran
Posts: 1531
Liked: 226 times
Joined: Jul 21, 2010 9:47 am
Full Name: Chris Dearden
Contact:

Re: VMware CBT bug KB 2090639

Post by chrisdearden »

there is no Sure Replica
Sure Replica is in v7.
Locked

Who is online

Users browsing this forum: No registered users and 76 guests