Discussions specific to the VMware vSphere hypervisor
Gostev
SVP, Product Management
Posts: 24939
Liked: 3622 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: [KB2136854] There's a new CBT bug in ESXi 6

Post by Gostev » Nov 25, 2015 8:45 pm 3 people like this post

All, we are testing the patch and will recommend the best "way out" based on its actual behavior (for example, I would expect the patch to automatically reset CBT for all VMs upon installation - in which case no additional steps should be required aside of deploying the patch). Stay tuned for updates. Thanks!

jbarrow.viracoribt
Expert
Posts: 184
Liked: 18 times
Joined: Feb 15, 2013 9:31 pm
Full Name: Jonathan Barrow
Contact:

[MERGED] VMWARE 6.0 CBT Bug is Fixed!

Post by jbarrow.viracoribt » Nov 25, 2015 9:56 pm 1 person likes this post


dellock6
Veeam Software
Posts: 5753
Liked: 1644 times
Joined: Jul 26, 2009 3:39 pm
Full Name: Luca Dell'Oca
Location: Varese, Italy
Contact:

Re: VMWARE 6.0 CBT Bug is Fixed!

Post by dellock6 » Nov 25, 2015 9:59 pm 1 person likes this post

well, with all the issues around CBT in vSphere 6.0, I'd say until the patch is tested, that VMware released a patch, not that the bug is fixed ;)
Anton already wrote in the official thread about this issue (http://forums.veeam.com/vmware-vsphere- ... ml#p169965) that we are testing the patch, so have a look at that thread for updated info.

Luca
Luca Dell'Oca
Principal EMEA Cloud Architect @ Veeam Software

@dellock6
https://www.virtualtothecore.com/
vExpert 2011 -> 2019
Veeam VMCE #1

cerberus
Enthusiast
Posts: 60
Liked: 5 times
Joined: Aug 28, 2015 2:45 pm
Full Name: MD
Contact:

Re: [KB2136854] There's a new CBT bug in ESXi 6

Post by cerberus » Nov 26, 2015 3:27 pm

That's great, thanks Veeam :) We are holding off on going to 6.0 until there is a confirmation the CBT bugfix provided by VMware is working for Veeam backups using CBT.

Gostev
SVP, Product Management
Posts: 24939
Liked: 3622 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: [KB2136854] There's a new CBT bug in ESXi 6

Post by Gostev » Nov 26, 2015 8:49 pm 4 people like this post

All, we have completed the first day of testing in the same exact lab and using the same heavy write I/O test that made the original issue easily reproducible. After a few TB of increments, the above-mention patch appears to fully resolve the original issue when installed on ESX 6.0 Update 1a build 3073146.

However, we found that simply installing the patch is not sufficient, and CBT reset is required for all of your VMs. This is because existing CBT map files may contain issues created earlier due to the original bug, which may result in inconsistent full backups in future. Having CBT reset will also force the following job run use "full scan" incremental pass, thus fixing any existing inconsistencies in backups and replicas, as discussed earlier in this topic.

Provided CBT reset has been performed, Active Full backups is not required.

Performing Active Full backups by itself cannot be considered as a substitute to CBT reset with this particular CBT issue.

Thanks!

mdmd
Enthusiast
Posts: 37
Liked: 1 time
Joined: Jan 06, 2014 10:29 am
Full Name: Mike
Contact:

Re: [KB2136854] There's a new CBT bug in ESXi 6

Post by mdmd » Nov 26, 2015 9:26 pm

So that means powering off every VM and resetting the CBT ? A full active wont do? This is going to be VERY time consuming and requires a lot of out of hours work, who should I send the bill too....

VMware really have gone down hill.

Gostev
SVP, Product Management
Posts: 24939
Liked: 3622 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: [KB2136854] There's a new CBT bug in ESXi 6

Post by Gostev » Nov 26, 2015 9:39 pm 1 person likes this post

Did you read the KB article I have linked in my previous post? Alternative CBT reset method is fully automated and does not require VM to be powered off (in fact, it requires them to be powered on and will not process powered off ones).

jelloir
Novice
Posts: 5
Liked: 1 time
Joined: May 07, 2015 11:45 pm
Contact:

Re: [KB2136854] There's a new CBT bug in ESXi 6

Post by jelloir » Nov 26, 2015 10:29 pm 1 person likes this post

Gostev wrote:Alternative CBT reset method is fully automated and does not require VM to be powered off (in fact, it requires them to be powered on and will not process powered off ones).
Just ran this (ResetCBT.ps1) and all worked well.

Pat490
Expert
Posts: 161
Liked: 25 times
Joined: Apr 28, 2015 7:18 am
Full Name: Patrick
Location: Germany
Contact:

Re: [KB2136854] There's a new CBT bug in ESXi 6

Post by Pat490 » Nov 27, 2015 7:10 am

Does ESX have to be Update 1A or can I install the patch directly on Update 1 Build 3029758?

VladV
Expert
Posts: 223
Liked: 25 times
Joined: Apr 30, 2013 7:38 am
Full Name: Vlad Valeriu Velciu
Contact:

Re: [KB2136854] There's a new CBT bug in ESXi 6

Post by VladV » Nov 27, 2015 9:54 am

I used this older method https://www.veeam.com/kb1940 . It processes all powered on vms with no existent snapshots.

mdmd
Enthusiast
Posts: 37
Liked: 1 time
Joined: Jan 06, 2014 10:29 am
Full Name: Mike
Contact:

Re: [KB2136854] There's a new CBT bug in ESXi 6

Post by mdmd » Nov 27, 2015 2:58 pm

Thanks Gostev, didn't see the link, I ran the ResetCBT.ps1 and it seemed to work but I did get a wanrning in between creating and removing snapshots (it did create) I presume this isn't a problem ?

Image

mdmd
Enthusiast
Posts: 37
Liked: 1 time
Joined: Jan 06, 2014 10:29 am
Full Name: Mike
Contact:

Re: [KB2136854] There's a new CBT bug in ESXi 6

Post by mdmd » Nov 27, 2015 3:11 pm

Also, as we are running Reverse incremental backups, I presume I don't need to run a active full backup?

veremin
Product Manager
Posts: 17005
Liked: 1455 times
Joined: Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin
Contact:

Re: [KB2136854] There's a new CBT bug in ESXi 6

Post by veremin » Nov 27, 2015 3:28 pm

As Anton's mentioned several posts above, as long as CBT reset has been performed, Active full is not needed.
Gostev wrote:Provided CBT reset has been performed, Active Full backups is not required.

mdking
Enthusiast
Posts: 34
Liked: never
Joined: Dec 03, 2014 2:14 pm
Full Name: Matt King
Contact:

Re: [KB2136854] There's a new CBT bug in ESXi 6

Post by mdking » Nov 28, 2015 8:38 am

Please excuse me if someone has already asked this. But if we are using Reverse Incremental backups and we perform the Vmware upgrades along with the CBT reset.

What happens to all of the older incremental files? Would they still be usable?

Thanks

pad343848
Novice
Posts: 5
Liked: never
Joined: Sep 17, 2015 1:06 am
Full Name: Sergey Sakharov
Contact:

Re: [KB2136854] There's a new CBT bug in ESXi 6

Post by pad343848 » Nov 28, 2015 10:17 am

Guys,

After installing update on ESX and perform full active backup all issue will gone?
Or i still need to run custom script from VEEAM KB?

I would like to avoid scripts and manual job.

What option do i have ?

Gostev
SVP, Product Management
Posts: 24939
Liked: 3622 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: [KB2136854] There's a new CBT bug in ESXi 6

Post by Gostev » Nov 28, 2015 12:22 pm

Hi Sergey, please read my previous post.

jamerson
Expert
Posts: 313
Liked: 18 times
Joined: May 01, 2013 9:54 pm
Full Name: Julien
Contact:

Re: [KB2136854] There's a new CBT bug in ESXi 6

Post by jamerson » Nov 30, 2015 12:44 am

Hi Guys,
as I understand the CBT bug is only on ESX 6.0 not on the 5.5 ?
After the patch 3247720 is applied.
do I have to run the script only if I still need the old backups ? otherwise no need ?
if I don't need the old backups ,do I have to do something so the VEEAM will start taking care of the new backup ?

Gostev
SVP, Product Management
Posts: 24939
Liked: 3622 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: [KB2136854] There's a new CBT bug in ESXi 6

Post by Gostev » Nov 30, 2015 1:00 am

Hi, Julien. Veeam KB article included in the first post of this topic answers all your questions. Also, read my previous post in case you need more details. Thanks!

Gostev
SVP, Product Management
Posts: 24939
Liked: 3622 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: [KB2136854] There's a new CBT bug in ESXi 6

Post by Gostev » Nov 30, 2015 11:38 am 1 person likes this post

All, please appreciate that this topic has many subscribers, so do keep all posts here strictly on topic. I've deleted all off-topic posts regarding environment-specific issues with performing CBT reset.

CBT reset is a common task, and there are existing topics discussing various procedures and scripts you can refer to. You can also create a new topic, if you'd like. Additionally, we have the following support KB articles providing a few different methods of performing CBT reset > KB1113, KB1940.

If you still have issues performing CBT reset, feel free to open a support case for assistance.

Thanks!

thuizenga
Influencer
Posts: 24
Liked: never
Joined: May 08, 2010 2:00 am
Full Name: Travis h
Contact:

Re: [KB2136854] There's a new CBT bug in ESXi 6

Post by thuizenga » Nov 30, 2015 6:09 pm

Thanks for the information, all my backup jobs have been reset.

What about replication jobs? Do i just run the CBT reset on the source VM's or is there a different procedure that needs to be done to make sure the destination machines aren't corrupted? Am i also suppose to run the CBT against the replicated VM's and if so do i need to commit existing snapshots first. Is there a script that handles powered off machines, or do i have to do the "Primary Reset CBT (Requires Power-Off)".

Thanks,

teknomage
Service Provider
Posts: 23
Liked: 2 times
Joined: Jul 21, 2010 8:55 pm
Full Name: Mike
Location: Fargo, ND
Contact:

Re: [KB2136854] There's a new CBT bug in ESXi 6

Post by teknomage » Nov 30, 2015 6:34 pm

Can I ask a question regarding the resting of the CBT data?

I understand that resetting CBT is required and I've run those scripts before, but I'm curious to the difference between running the scripts and toggling the "Changed block tracking" option in individual backup jobs. Does this only tell the job to use/ignore CBT data, but do nothing with the setting on the individual machines? Also, is the disabling CBT in the Veeam job itself not sufficient to avoid possible issues?

-Mike

Gostev
SVP, Product Management
Posts: 24939
Liked: 3622 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: [KB2136854] There's a new CBT bug in ESXi 6

Post by Gostev » Nov 30, 2015 7:41 pm 1 person likes this post

thuizenga wrote:What about replication jobs? Do i just run the CBT reset on the source VM's or is there a different procedure that needs to be done to make sure the destination machines aren't corrupted? Am i also suppose to run the CBT against the replicated VM's and if so do i need to commit existing snapshots first.
Yes, you are supposed to run the CBT against the VMs you are replicating as well. There is no need to commit any snapshots though, as you only need to reset CBT for "source" (production) VMs, which should not have any snapshots. You don't need to reset CBT for "target" (replica) VMs which you are probably talking about here.
teknomage wrote:I understand that resetting CBT is required and I've run those scripts before, but I'm curious to the difference between running the scripts and toggling the "Changed block tracking" option in individual backup jobs.
Great question, Mike! The difference is that resetting CBT recreates and re-populates CBT map file, which is essential to do when they may contain inconsistency due to the bug discussed in this topic. The option you are talking about simply makes the job not use CBT data - and while disabling this option solves the issue, this is not a type of setup that most people will be happy to live with forever due to the associated performance penalty of "full scan" incremental runs.

thuizenga
Influencer
Posts: 24
Liked: never
Joined: May 08, 2010 2:00 am
Full Name: Travis h
Contact:

Re: [KB2136854] There's a new CBT bug in ESXi 6

Post by thuizenga » Nov 30, 2015 8:37 pm

Thanks Gostev, you are correct i am strictly talking about the target systems. From a technical perspective when CBT is reset on a replica machine does this invoke the process where source and destinations blocks are compared which would be similar to the initial replica seeding process?

Gostev
SVP, Product Management
Posts: 24939
Liked: 3622 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: [KB2136854] There's a new CBT bug in ESXi 6

Post by Gostev » Nov 30, 2015 10:29 pm

That is correct, as this is the only way to perform an incremental pass without knowing which blocks have changed. Thanks!

prabhash.jena
Enthusiast
Posts: 35
Liked: 2 times
Joined: Jan 16, 2015 1:32 pm
Full Name: Prabhash Kumar Jena
Contact:

[MERGED] ESXi 6.0 CBT issue

Post by prabhash.jena » Dec 01, 2015 2:05 pm

Hi,

I have a general question regarding CBT issue on ESXi 6.0.

Is it same to presume that my environment is not affected with this bug at all if I have not get the message "CBT data is invalid, failing over to legacy incremental backup. No action is required, next job run should start using CBT again. If CBT data remains invalid, follow KB1113 to perform CBT reset. Usual cause is power loss." during the days the bug existed. I have around 30 jobs and I don't see this message on any job, I have checked job history as well.
My colleague has now applied the patch, do we need to carry out some tasks to ensure we have correct backups?

Thanks.
Prabhash Jena

Gostev
SVP, Product Management
Posts: 24939
Liked: 3622 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: [KB2136854] There's a new CBT bug in ESXi 6

Post by Gostev » Dec 01, 2015 2:14 pm

Hi. Assuming you are talking about KB2136854, this particular issue causes silent backup data corruption without any error messages logged. Every environment running vSphere 6 is affected, and must perform CBT reset on all existing VMs after installing the patch from KB3247720. Thanks!

cffit
Expert
Posts: 338
Liked: 34 times
Joined: Jan 20, 2012 2:36 pm
Full Name: Christensen Farms
Contact:

Re: [KB2136854] There's a new CBT bug in ESXi 6

Post by cffit » Dec 02, 2015 4:23 pm

The VMware KB article mentions that the individual VMs need to be migrated or restarted. I ran the suggested script that disables CBT, then takes and deletes a snapshot, then re-enables CBT. I did not reboot the VMs. Is it still necessary to restart the individual VMs in this case?

cffit
Expert
Posts: 338
Liked: 34 times
Joined: Jan 20, 2012 2:36 pm
Full Name: Christensen Farms
Contact:

Re: [KB2136854] There's a new CBT bug in ESXi 6

Post by cffit » Dec 02, 2015 5:44 pm

This is from the VMware KB per my last post saying the VMs have to be rebooted:

Release Date: Nov 25, 2015

Download Filename:
ESXi600-201511001.zip

Build:
3247720

Download Size:
358.9 MB

Host Reboot Required: Yes
Virtual Machine Migration or Shutdown Required: Yes

Gostev
SVP, Product Management
Posts: 24939
Liked: 3622 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: [KB2136854] There's a new CBT bug in ESXi 6

Post by Gostev » Dec 02, 2015 10:41 pm

cffit wrote:VMs have to be rebooted
Sorry, but can you quote that, because I must be blind :shock:

cffit
Expert
Posts: 338
Liked: 34 times
Joined: Jan 20, 2012 2:36 pm
Full Name: Christensen Farms
Contact:

Re: [KB2136854] There's a new CBT bug in ESXi 6

Post by cffit » Dec 02, 2015 11:15 pm

In this KB:
http://kb.vmware.com/selfservice/search ... Id=2137545

"Virtual Machine Migration or Shutdown Required: Yes"

Locked

Who is online

Users browsing this forum: Baidu [Spider], Bing [Bot] and 8 guests