[KB2136854] There's a new CBT bug in ESXi 6

[Gostev]

All, we are testing the patch and will recommend the best "way out" based on its actual behavior (for example, I would expect the patch to automatically reset CBT for all VMs upon installation - in which case no additional steps should be required aside of deploying the patch). Stay tuned for updates. Thanks!

[jbarrow.viracoribt]

Hot off the presses!

http://kb.vmware.com/selfservice/micros ... Id=2136854

[dellock6]

well, with all the issues around CBT in vSphere 6.0, I'd say until the patch is tested, that VMware released a patch, not that the bug is fixed
Anton already wrote in the official thread about this issue (http://forums.veeam.com/vmware-vsphere- ... ml#p169965) that we are testing the patch, so have a look at that thread for updated info.

Luca

[cerberus]

That's great, thanks Veeam We are holding off on going to 6.0 until there is a confirmation the CBT bugfix provided by VMware is working for Veeam backups using CBT.

[Gostev]

All, we have completed the first day of testing in the same exact lab and using the same heavy write I/O test that made the original issue easily reproducible. After a few TB of increments, the above-mention patch appears to fully resolve the original issue when installed on ESX 6.0 Update 1a build 3073146.

However, we found that simply installing the patch is not sufficient, and CBT reset is required for all of your VMs. This is because existing CBT map files may contain issues created earlier due to the original bug, which may result in inconsistent full backups in future. Having CBT reset will also force the following job run use "full scan" incremental pass, thus fixing any existing inconsistencies in backups and replicas, as discussed earlier in this topic.

Provided CBT reset has been performed, Active Full backups is not required.

Performing Active Full backups by itself cannot be considered as a substitute to CBT reset with this particular CBT issue.

Thanks!

[mdmd]

So that means powering off every VM and resetting the CBT ? A full active wont do? This is going to be VERY time consuming and requires a lot of out of hours work, who should I send the bill too....

VMware really have gone down hill.

[Gostev]

Did you read the KB article I have linked in my previous post? Alternative CBT reset method is fully automated and does not require VM to be powered off (in fact, it requires them to be powered on and will not process powered off ones).

[jelloir]

Alternative CBT reset method is fully automated and does not require VM to be powered off (in fact, it requires them to be powered on and will not process powered off ones).

Just ran this (ResetCBT.ps1) and all worked well.

[Pat490]

Does ESX have to be Update 1A or can I install the patch directly on Update 1 Build 3029758?

[VladV]

I used this older method https://www.veeam.com/kb1940 . It processes all powered on vms with no existent snapshots.

[mdmd]

Thanks Gostev, didn't see the link, I ran the ResetCBT.ps1 and it seemed to work but I did get a wanrning in between creating and removing snapshots (it did create) I presume this isn't a problem ?

[mdmd]

Also, as we are running Reverse incremental backups, I presume I don't need to run a active full backup?

[veremin]

As Anton's mentioned several posts above, as long as CBT reset has been performed, Active full is not needed.

Provided CBT reset has been performed, Active Full backups is not required.

[mdking]

Please excuse me if someone has already asked this. But if we are using Reverse Incremental backups and we perform the Vmware upgrades along with the CBT reset.

What happens to all of the older incremental files? Would they still be usable?

Thanks

[pad343848]

Guys,

After installing update on ESX and perform full active backup all issue will gone?
Or i still need to run custom script from VEEAM KB?

I would like to avoid scripts and manual job.

What option do i have ?

[Gostev]

Hi Sergey, please read my previous post.

[jamerson]

Hi Guys,
as I understand the CBT bug is only on ESX 6.0 not on the 5.5 ?
After the patch 3247720 is applied.
do I have to run the script only if I still need the old backups ? otherwise no need ?
if I don't need the old backups ,do I have to do something so the VEEAM will start taking care of the new backup ?

[Gostev]

Hi, Julien. Veeam KB article included in the first post of this topic answers all your questions. Also, read my previous post in case you need more details. Thanks!

[Gostev]

All, please appreciate that this topic has many subscribers, so do keep all posts here strictly on topic. I've deleted all off-topic posts regarding environment-specific issues with performing CBT reset.

CBT reset is a common task, and there are existing topics discussing various procedures and scripts you can refer to. You can also create a new topic, if you'd like. Additionally, we have the following support KB articles providing a few different methods of performing CBT reset > KB1113, KB1940.

If you still have issues performing CBT reset, feel free to open a support case for assistance.

Thanks!

[thuizenga]

Thanks for the information, all my backup jobs have been reset.

What about replication jobs? Do i just run the CBT reset on the source VM's or is there a different procedure that needs to be done to make sure the destination machines aren't corrupted? Am i also suppose to run the CBT against the replicated VM's and if so do i need to commit existing snapshots first. Is there a script that handles powered off machines, or do i have to do the "Primary Reset CBT (Requires Power-Off)".

Thanks,

[teknomage]

Can I ask a question regarding the resting of the CBT data?

I understand that resetting CBT is required and I've run those scripts before, but I'm curious to the difference between running the scripts and toggling the "Changed block tracking" option in individual backup jobs. Does this only tell the job to use/ignore CBT data, but do nothing with the setting on the individual machines? Also, is the disabling CBT in the Veeam job itself not sufficient to avoid possible issues?

-Mike

[Gostev]

What about replication jobs? Do i just run the CBT reset on the source VM's or is there a different procedure that needs to be done to make sure the destination machines aren't corrupted? Am i also suppose to run the CBT against the replicated VM's and if so do i need to commit existing snapshots first.

Yes, you are supposed to run the CBT against the VMs you are replicating as well. There is no need to commit any snapshots though, as you only need to reset CBT for "source" (production) VMs, which should not have any snapshots. You don't need to reset CBT for "target" (replica) VMs which you are probably talking about here.

I understand that resetting CBT is required and I've run those scripts before, but I'm curious to the difference between running the scripts and toggling the "Changed block tracking" option in individual backup jobs.

Great question, Mike! The difference is that resetting CBT recreates and re-populates CBT map file, which is essential to do when they may contain inconsistency due to the bug discussed in this topic. The option you are talking about simply makes the job not use CBT data - and while disabling this option solves the issue, this is not a type of setup that most people will be happy to live with forever due to the associated performance penalty of "full scan" incremental runs.

[thuizenga]

Thanks Gostev, you are correct i am strictly talking about the target systems. From a technical perspective when CBT is reset on a replica machine does this invoke the process where source and destinations blocks are compared which would be similar to the initial replica seeding process?

[Gostev]

That is correct, as this is the only way to perform an incremental pass without knowing which blocks have changed. Thanks!

[prabhash.jena]

Hi,

I have a general question regarding CBT issue on ESXi 6.0.

Is it same to presume that my environment is not affected with this bug at all if I have not get the message "CBT data is invalid, failing over to legacy incremental backup. No action is required, next job run should start using CBT again. If CBT data remains invalid, follow KB1113 to perform CBT reset. Usual cause is power loss." during the days the bug existed. I have around 30 jobs and I don't see this message on any job, I have checked job history as well.
My colleague has now applied the patch, do we need to carry out some tasks to ensure we have correct backups?

Thanks.
Prabhash Jena

[Gostev]

Hi. Assuming you are talking about KB2136854, this particular issue causes silent backup data corruption without any error messages logged. Every environment running vSphere 6 is affected, and must perform CBT reset on all existing VMs after installing the patch from KB3247720. Thanks!

[cffit]

The VMware KB article mentions that the individual VMs need to be migrated or restarted. I ran the suggested script that disables CBT, then takes and deletes a snapshot, then re-enables CBT. I did not reboot the VMs. Is it still necessary to restart the individual VMs in this case?

[cffit]

This is from the VMware KB per my last post saying the VMs have to be rebooted:

Release Date: Nov 25, 2015

Download Filename:
ESXi600-201511001.zip

Build:
3247720

Download Size:
358.9 MB

Host Reboot Required: Yes
Virtual Machine Migration or Shutdown Required: Yes

[Gostev]

VMs have to be rebooted

Sorry, but can you quote that, because I must be blind

[cffit]

In this KB:
http://kb.vmware.com/selfservice/search ... Id=2137545

"Virtual Machine Migration or Shutdown Required: Yes"