VMware CBT bug KB 2090639

[Gostev]

@Tom yes, it was the actual test.

All, sorry I cannot comment on every post (I am on the road this and next week). I will be updating you as news appear, and will try to do it at least daily.

To answer a common question about CBT reset handling: along with the patch, we will provide a script that resets CBT on all VMs (in fact, its was already published on Monday > http://www.veeam.com/kb1940).

[chrisbarr35]

Hi
What is the implication of resetting CBT when using copy jobs? We already run "Active Full" backups weekly, but the intelligent copy still only copies the increments on the next copy run. If CBT is reset will the next copy job need to run a full copy of the VMs in the backup?
Chris

[xadamz23]

I would like clarification on one thing that I havent seen touched on yet. I wanted to test the solution of resetting CBT and running a backup job to get everything back in working order. I have a backup job with just one VM in it. I ran the PowerCLI code provided in http://www.veeam.com/kb1940 to reset CBT. I then kicked off my backup job as I normally would. The VM is 130 GB in size, but to my surprise the backup job created a .vib file that was only 300 MB in size. Obviously only 300 MB of data had changed since the last backup. At first I thought something was wrong because I expected a huge .vib file. But then I thought to myself it reset CBT so it only had to "scan" the entire vmdk for changed blocks, it didnt actually need to back the entire vmdk file up.

Is that correct?

Edit:
I guess I dont fully understand that either though. Lets say this particular job was experiencing the corrupt CBT issue. So lets say the issue first arose weeks ago. So doesnt that mean that possibly every backup since the issue arose is bad, and not all blocks that needed to be backed up were being backed up? Then I fix CBT by doing the above, but it simply scans for changed blocks since the "last" backup. Wouldnt it still be missing some blocks?

[BriFar]

Yes, it will be shown. However, backup size is not impacted with CBT reset (in case of our product at least), only duration will be longer because the job needs to read the entire VMDK.

Yes you are right. I ment to write time not size.

This is not a good idea to do, because often there could be other jobs processing the same VM (for example, backing up and replicating the same VM). The other job will be impacted by CBT reset, and will likely not meet its RTOs as the result.

I know what you mean but wouldn't replicas also be effected by this bug? If so it could be that you might have a replica which meets RTO but does not boot or the data is missing or corrupted.
Don't take me wrong, I know that there will not be a one for all fix and I appreciate the efforts being put here.

[Reimold]

I just saw that the VMware-KB was updated:

"This issue occurs on a virtual machine with Changed Block Tracking (CBT) enabled, when extending a virtual disk (vmdk) file to a size strictly above 128 GB, due to the block tracking information being recalculated incorrectly."
http://kb.vmware.com/selfservice/micros ... Id=2090639

So if I understand that right, every VM that was extended to a target size greater than 128 GB is at risk, no matter how big the extend was or from what size the extend was started.

Dirk

[foggy]

What is the implication of resetting CBT when using copy jobs? We already run "Active Full" backups weekly, but the intelligent copy still only copies the increments on the next copy run. If CBT is reset will the next copy job need to run a full copy of the VMs in the backup?

Backup copy jobs do not rely on VMware CBT and will still copy only changed blocks from the new full.

But then I thought to myself it reset CBT so it only had to "scan" the entire vmdk for changed blocks, it didnt actually need to back the entire vmdk file up.

Is that correct?

Correct.

I guess I dont fully understand that either though. Lets say this particular job was experiencing the corrupt CBT issue. So lets say the issue first arose weeks ago. So doesnt that mean that possibly every backup since the issue arose is bad, and not all blocks that needed to be backed up were being backed up? Then I fix CBT by doing the above, but it simply scans for changed blocks since the "last" backup. Wouldnt it still be missing some blocks?

No, all the missing blocks will be copied after scanning the entire VM image.

[GabesVirtualWorld]

I was wondering if there is a way to check which VMs are affected? Can we see "inside" the CBT file? Can we query the VM ?

With close to 1500 VMs I would hate to force full scans of just any VM.

The KB mentions:

When you run the command QueryChangedDiskAreas("*") to return a list of allocated disk sectors, you experience these symptoms:
Allocated portions of the virtual machine disk vmdk file are not returned.
The list of allocated virtual machine disk sectors returned is incorrect

Now, how can I run that command? And how can I see if the returned info is incorrect?

[tsightler]

You can immediately narrow it down by removing any VMs that are less than 128GB as, at least based on the information so far, they should be immune to the problem. If you have 1500VMs that are greater than 128GB, well, that doesn't help much!

I don't see how querying the API alone would be useful as you'd need something to compare it to. The only way I could see to do that would be to scan the VMDK, at least the parts beyond 128GB. I don't see any practical way to do this that wouldn't be more painful that just resetting CBT.

[joergr]

@Gabe and Tom:

Me personally, I did a scan over EVERY vm to examine where CBT is active: get-vm | ?{$_.ExtensionData.Config.ChangeTrackingEnabled -eq $true}

After that i ran the script targeting every one of these i found. At the end of the day 100% safety matters for me and i also like to have a good sleep

In our datacenter it got about 200 VMs per hour this way (but i did it during a working day so the helper-snaphots sometimes took a little longer to consolidate). I guess during weekend time this will be much faster. But then again, it depends on your VM-size, purpose, storage and on your ESXi hosts very much.

But ofcourse you can 'AND-filter' the CBT-VMs combined with a vdisk size check (was posted in this thread, too). Again - for me personally - i had a better gut feeling by applying a CBT-reset to every CBT-enabled VM - no matter what size.

Best regards,
Joerg

[Reimold]

After that i ran the script targeting every one of these i found. At the end of the day 100% safety matters for me and i also like to have a good sleep

But if I reset the CBT on a lot of VM´s at the same time, all Veeam Jobs will take much longer because of the VMDK beeing read completly. At least in our environment the backup window will be exceeded (Proxys and repositories will be overbooked) - and I cannot let run SAP or Exchange with an active snapshot during production hours. So I have to reset them in small Groups.

Dirk

[dellock6]

Correct. Also, Joerg, resetting CBT "before" a Vm crosses the 128Gb threshold is useless, you force a disk scan with Veeam, but at the time the VM will cross the 128GB size in the future you will have again to reset the CBT, unless until we release a workaround or VMware creates a fix... I hope honestly they will fix it quickly, but I still have another bug open in my list that is taking months (did we forgot this one? http://kb.vmware.com/kb/2068424 is still open since last january, 10 months...) so I would probably wait first for our workaround

[joergr]

Correct. Also, Joerg, resetting CBT "before" a Vm crosses the 128Gb threshold is useless, you force a disk scan with Veeam, but at the time the VM will cross the 128GB size in the future you will have again to reset the CBT, unless until we release a workaround or VMware creates a fix... I hope honestly they will fix it quickly, but I still have another bug open in my list that is taking months (did we forgot this one? http://kb.vmware.com/kb/2068424 is still open since last january, 10 months...) so I would probably wait first for our workaround

Of course you are right Luca - always in mind and assumptive that VMware is 100% and absolutely guaranteed right about this 128GB "barrier".

By the time till the patch comes out i told my colleagues to report when a vmdk is enlarged so i could reset cbt especially for this one vm.

Best regards,
Joerg

[joergr]

But if I reset the CBT on a lot of VM´s at the same time, all Veeam Jobs will take much longer because of the VMDK beeing read completly. At least in our environment the backup window will be exceeded (Proxys and repositories will be overbooked) - and I cannot let run SAP or Exchange with an active snapshot during production hours. So I have to reset them in small Groups.

Dirk

@Dirk:

Of course you have to respect the special characteristics of your adjacencies. It could be possible to go step-by-step (you could for example write the not resetted vms to a variable and keep it till the next day and then continue). Me, personally, i first took some test runs with 3-5 vms just to be sure a) it worked with the config and snapshots and b) the time is indeed exactly the time of an active full. Before i kicked off the reset script i calculated (by the history of all backup jobs and all veeam servers respecting all connected san infrastructure systems - i could count on these values because i am doing active fulls every week - thus i relatively exactly knew the time windows) the time my active full would be exceeding the backup window. With this in mind i kicked of some selected jobs which would cause the windows to exceed and took active fulls before the actual backups ran. Thus - these selected jobs were done fast as everytime when the main backup window came. All this went well.

So - of course and as you already mentioned, you have to adapt to the special characteristics of your environment.

Best Regards,
Joerg

[joergr]

have another bug open in my list that is taking months (did we forgot this one? http://kb.vmware.com/kb/2068424 is still open since last january, 10 months...) so I would probably wait first for our workaround

Oh my Yes this bug is bugging me very intense for a long time

[geofftx]

Still working on verification strategies, but is it fair to say that if you check your latest backups using Veeam Instant Recovery, running a chkdsk on any VM volume larger that 128G, that if chkdsk returns no errors then you have not encountered the bug and all prior restore points will also be good? (Assuming, of course, that CBT has never been reset at any point in that machine's Veeam backup history.)

And if the volumes check clean is there any reason to perform a CBT reset?

Geoff

[aaron_meza]

It''s not just 128GB boundaries. Crossing the 256GB, 512GB, and 1024GB boundaries result in corruption as well. In many cases, corruption is undetectable if you are restoring fulll VM. chkdsk will pass. the only way to detect corruption is to know the checksum from the file on the original volume and compare that to recovered file checksum.

[aaron_meza]

Has anyone actually come across this bug 'in the wild' yet and have personal experience of how it manifests?

Keen to know whether this will trash the entire 128Gb+ disk's structure and file headers, making it effectively unusable/unmountable, or whether theoretically, if i'm able to use the windows guest File-Level-Restore wizard which creates vmdk mountpoints in C:\veeamflr on my backup server, and it successfully enumerates the entire drive's contents and directory structure, i should be in the clear?

Unfortunalty, no. I have produced a scenario where the entire disk structure is correct, chkdsk passes. There are simple zeroed out areas of data in a file. No way to detect without knowing the prior contents/checksum of the file.

[MrSpock]

I have now made a CBT reset on all my VM's. On one host I stumbled on the error stated in this article: http://www.veeam.com/kb1113

Let us take a look at step 7 to 9:

7. Set the "scsi0:x.ctkEnabled" value back to true for each disk of the VM in question
8. Power the VM on
9. Rerun Backup or Replication job to re-enable CBT

At step 7 I did also set "ctkEnabled" to true. That means that CBT will already be enabled when next backup session starts. Veeam Backup will not re-enable CBT as stated in step 9 (I suppose).

Question: The CBT data does not contain all changes since the last backup session. Will Veeam Backup detect that CBT has been reset and re-enabled since the last backup job and make a full scan of the disk?

Best regards,

Johan

[joergr]

@Johan: If you followed exactly the rules in the VEEAM KB you mentioned, especially no5, you are good, because the old ctks are gone.

Joerg

[MrSpock]

@Johan: If you followed exactly the rules in the VEEAM KB you mentioned, especially no5, you are good, because the old ctks are gone.

Joerg

Joerg,

Yes, I am safe regarding the VMWare CBT bug, but my question is about what Veeam Backup will do when CBT has been disabled and re-enabled since the last backup session. Will Veeam Backup use the CBT data or ignore it?

Best regards,

Johan

[joergr]

Johan: Even you had re-enabled cbt manually by setting it according to the veeam kb article you mentioned there is actually no real ctk data on the volume - thus veeam can not use any cbt data at this time - at least not with this first backup run. So there is nothing to ignore because there simply is nothing there. The cbt data is needed after your first backup after the cbt reset - thats how cbt works - to tell the backup provider which blocks have changed since the last backup.

Joerg

[loelly]

So, I happen to have some TBs of file servers and need a quick confirmation that I understood this correctly:

After disabling/enabling CBT, the following backup run will scan the VMDKs entirely but only copy the missing deltas to the backup repository? So I do not need to provide some more TBs to the backup repository. It will be a time consuming, but not space consuming run, right?

[joergr]

You got it 100% right. The time is the time of an active full BUT the space remains the usual.

[loelly]

Thanks for the quick confirmation!

[lando_uk]

Any VM's that I know have had their disks expanded, I can just do a storage vmotion them to sort this out yes? This will reset CBT?

[joergr]

Yes. http://kb.vmware.com/selfservice/micros ... Id=2048201

But only until ESXi 5.5U2. Be aware of that.

[lando_uk]

If I do the CBT reset (or a storage vmotion) I presume the next synthetic full backup wont fix the issue, I'd have to run a active full on the job that the VM I fixed is in?

[loelly]

No, you do not need to run an Active Full. Just reset CBT as proposed, Veeam B&R will do an integrity check on the next run, save just the deltas and you do not need to waste a whole bunch of storage in the backup repository (temporarily). Your backups will take longer, comparable to an Active Full. See posts above.

[ian0x0r]

Subscribing to thread.

[kwells]

Hi,

I would like to use the powerCLI script supplied by Veeam,, but have never used powercli before, so have no idea how to run it. Is there a simple idiots guide somewhere please. Also I see that some people are saying they are applying the script to a few VMs at a time, but the way I am reading the script comments it seems to get a list of all the VMs that have CBT then makes the change against them all at once. Am I just completely misunderstanding how this works?

Thanks in advance for any help you can supply