-
- Chief Product Officer
- Posts: 31809
- Liked: 7301 times
- Joined: Jan 01, 2006 1:01 am
- Location: Baar, Switzerland
- Contact:
Re: [KB2136854] There's a new CBT bug in ESXi 6
All, we are testing the patch and will recommend the best "way out" based on its actual behavior (for example, I would expect the patch to automatically reset CBT for all VMs upon installation - in which case no additional steps should be required aside of deploying the patch). Stay tuned for updates. Thanks!
-
- Expert
- Posts: 184
- Liked: 18 times
- Joined: Feb 15, 2013 9:31 pm
- Full Name: Jonathan Barrow
- Contact:
-
- VeeaMVP
- Posts: 6166
- Liked: 1971 times
- Joined: Jul 26, 2009 3:39 pm
- Full Name: Luca Dell'Oca
- Location: Varese, Italy
- Contact:
Re: VMWARE 6.0 CBT Bug is Fixed!
well, with all the issues around CBT in vSphere 6.0, I'd say until the patch is tested, that VMware released a patch, not that the bug is fixed
Anton already wrote in the official thread about this issue (http://forums.veeam.com/vmware-vsphere- ... ml#p169965) that we are testing the patch, so have a look at that thread for updated info.
Luca
Anton already wrote in the official thread about this issue (http://forums.veeam.com/vmware-vsphere- ... ml#p169965) that we are testing the patch, so have a look at that thread for updated info.
Luca
Luca Dell'Oca
Principal EMEA Cloud Architect @ Veeam Software
@dellock6
https://www.virtualtothecore.com/
vExpert 2011 -> 2022
Veeam VMCE #1
Principal EMEA Cloud Architect @ Veeam Software
@dellock6
https://www.virtualtothecore.com/
vExpert 2011 -> 2022
Veeam VMCE #1
-
- Expert
- Posts: 164
- Liked: 17 times
- Joined: Aug 28, 2015 2:45 pm
- Full Name: Mirza
- Contact:
Re: [KB2136854] There's a new CBT bug in ESXi 6
That's great, thanks Veeam We are holding off on going to 6.0 until there is a confirmation the CBT bugfix provided by VMware is working for Veeam backups using CBT.
-
- Chief Product Officer
- Posts: 31809
- Liked: 7301 times
- Joined: Jan 01, 2006 1:01 am
- Location: Baar, Switzerland
- Contact:
Re: [KB2136854] There's a new CBT bug in ESXi 6
All, we have completed the first day of testing in the same exact lab and using the same heavy write I/O test that made the original issue easily reproducible. After a few TB of increments, the above-mention patch appears to fully resolve the original issue when installed on ESX 6.0 Update 1a build 3073146.
However, we found that simply installing the patch is not sufficient, and CBT reset is required for all of your VMs. This is because existing CBT map files may contain issues created earlier due to the original bug, which may result in inconsistent full backups in future. Having CBT reset will also force the following job run use "full scan" incremental pass, thus fixing any existing inconsistencies in backups and replicas, as discussed earlier in this topic.
Provided CBT reset has been performed, Active Full backups is not required.
Performing Active Full backups by itself cannot be considered as a substitute to CBT reset with this particular CBT issue.
Thanks!
However, we found that simply installing the patch is not sufficient, and CBT reset is required for all of your VMs. This is because existing CBT map files may contain issues created earlier due to the original bug, which may result in inconsistent full backups in future. Having CBT reset will also force the following job run use "full scan" incremental pass, thus fixing any existing inconsistencies in backups and replicas, as discussed earlier in this topic.
Provided CBT reset has been performed, Active Full backups is not required.
Performing Active Full backups by itself cannot be considered as a substitute to CBT reset with this particular CBT issue.
Thanks!
-
- Enthusiast
- Posts: 38
- Liked: 2 times
- Joined: Jan 06, 2014 10:29 am
- Full Name: Mike
- Contact:
Re: [KB2136854] There's a new CBT bug in ESXi 6
So that means powering off every VM and resetting the CBT ? A full active wont do? This is going to be VERY time consuming and requires a lot of out of hours work, who should I send the bill too....
VMware really have gone down hill.
VMware really have gone down hill.
-
- Chief Product Officer
- Posts: 31809
- Liked: 7301 times
- Joined: Jan 01, 2006 1:01 am
- Location: Baar, Switzerland
- Contact:
Re: [KB2136854] There's a new CBT bug in ESXi 6
Did you read the KB article I have linked in my previous post? Alternative CBT reset method is fully automated and does not require VM to be powered off (in fact, it requires them to be powered on and will not process powered off ones).
-
- Novice
- Posts: 5
- Liked: 1 time
- Joined: May 07, 2015 11:45 pm
- Contact:
Re: [KB2136854] There's a new CBT bug in ESXi 6
Just ran this (ResetCBT.ps1) and all worked well.Gostev wrote:Alternative CBT reset method is fully automated and does not require VM to be powered off (in fact, it requires them to be powered on and will not process powered off ones).
-
- Expert
- Posts: 170
- Liked: 29 times
- Joined: Apr 28, 2015 7:18 am
- Full Name: Patrick
- Location: Germany
- Contact:
Re: [KB2136854] There's a new CBT bug in ESXi 6
Does ESX have to be Update 1A or can I install the patch directly on Update 1 Build 3029758?
-
- Expert
- Posts: 224
- Liked: 25 times
- Joined: Apr 30, 2013 7:38 am
- Full Name: Vlad Valeriu Velciu
- Contact:
Re: [KB2136854] There's a new CBT bug in ESXi 6
I used this older method https://www.veeam.com/kb1940 . It processes all powered on vms with no existent snapshots.
-
- Enthusiast
- Posts: 38
- Liked: 2 times
- Joined: Jan 06, 2014 10:29 am
- Full Name: Mike
- Contact:
Re: [KB2136854] There's a new CBT bug in ESXi 6
Thanks Gostev, didn't see the link, I ran the ResetCBT.ps1 and it seemed to work but I did get a wanrning in between creating and removing snapshots (it did create) I presume this isn't a problem ?
-
- Enthusiast
- Posts: 38
- Liked: 2 times
- Joined: Jan 06, 2014 10:29 am
- Full Name: Mike
- Contact:
Re: [KB2136854] There's a new CBT bug in ESXi 6
Also, as we are running Reverse incremental backups, I presume I don't need to run a active full backup?
-
- Product Manager
- Posts: 20411
- Liked: 2300 times
- Joined: Oct 26, 2012 3:28 pm
- Full Name: Vladimir Eremin
- Contact:
Re: [KB2136854] There's a new CBT bug in ESXi 6
As Anton's mentioned several posts above, as long as CBT reset has been performed, Active full is not needed.
Gostev wrote:Provided CBT reset has been performed, Active Full backups is not required.
-
- Enthusiast
- Posts: 35
- Liked: never
- Joined: Dec 03, 2014 2:14 pm
- Full Name: Matt King
- Contact:
Re: [KB2136854] There's a new CBT bug in ESXi 6
Please excuse me if someone has already asked this. But if we are using Reverse Incremental backups and we perform the Vmware upgrades along with the CBT reset.
What happens to all of the older incremental files? Would they still be usable?
Thanks
What happens to all of the older incremental files? Would they still be usable?
Thanks
-
- Novice
- Posts: 5
- Liked: never
- Joined: Sep 17, 2015 1:06 am
- Full Name: Sergey Sakharov
- Contact:
Re: [KB2136854] There's a new CBT bug in ESXi 6
Guys,
After installing update on ESX and perform full active backup all issue will gone?
Or i still need to run custom script from VEEAM KB?
I would like to avoid scripts and manual job.
What option do i have ?
After installing update on ESX and perform full active backup all issue will gone?
Or i still need to run custom script from VEEAM KB?
I would like to avoid scripts and manual job.
What option do i have ?
-
- Chief Product Officer
- Posts: 31809
- Liked: 7301 times
- Joined: Jan 01, 2006 1:01 am
- Location: Baar, Switzerland
- Contact:
Re: [KB2136854] There's a new CBT bug in ESXi 6
Hi Sergey, please read my previous post.
-
- Veteran
- Posts: 366
- Liked: 24 times
- Joined: May 01, 2013 9:54 pm
- Full Name: Julien
- Contact:
Re: [KB2136854] There's a new CBT bug in ESXi 6
Hi Guys,
as I understand the CBT bug is only on ESX 6.0 not on the 5.5 ?
After the patch 3247720 is applied.
do I have to run the script only if I still need the old backups ? otherwise no need ?
if I don't need the old backups ,do I have to do something so the VEEAM will start taking care of the new backup ?
as I understand the CBT bug is only on ESX 6.0 not on the 5.5 ?
After the patch 3247720 is applied.
do I have to run the script only if I still need the old backups ? otherwise no need ?
if I don't need the old backups ,do I have to do something so the VEEAM will start taking care of the new backup ?
-
- Chief Product Officer
- Posts: 31809
- Liked: 7301 times
- Joined: Jan 01, 2006 1:01 am
- Location: Baar, Switzerland
- Contact:
Re: [KB2136854] There's a new CBT bug in ESXi 6
Hi, Julien. Veeam KB article included in the first post of this topic answers all your questions. Also, read my previous post in case you need more details. Thanks!
-
- Chief Product Officer
- Posts: 31809
- Liked: 7301 times
- Joined: Jan 01, 2006 1:01 am
- Location: Baar, Switzerland
- Contact:
Re: [KB2136854] There's a new CBT bug in ESXi 6
All, please appreciate that this topic has many subscribers, so do keep all posts here strictly on topic. I've deleted all off-topic posts regarding environment-specific issues with performing CBT reset.
CBT reset is a common task, and there are existing topics discussing various procedures and scripts you can refer to. You can also create a new topic, if you'd like. Additionally, we have the following support KB articles providing a few different methods of performing CBT reset > KB1113, KB1940.
If you still have issues performing CBT reset, feel free to open a support case for assistance.
Thanks!
CBT reset is a common task, and there are existing topics discussing various procedures and scripts you can refer to. You can also create a new topic, if you'd like. Additionally, we have the following support KB articles providing a few different methods of performing CBT reset > KB1113, KB1940.
If you still have issues performing CBT reset, feel free to open a support case for assistance.
Thanks!
-
- Influencer
- Posts: 24
- Liked: never
- Joined: May 08, 2010 2:00 am
- Full Name: Travis h
- Contact:
Re: [KB2136854] There's a new CBT bug in ESXi 6
Thanks for the information, all my backup jobs have been reset.
What about replication jobs? Do i just run the CBT reset on the source VM's or is there a different procedure that needs to be done to make sure the destination machines aren't corrupted? Am i also suppose to run the CBT against the replicated VM's and if so do i need to commit existing snapshots first. Is there a script that handles powered off machines, or do i have to do the "Primary Reset CBT (Requires Power-Off)".
Thanks,
What about replication jobs? Do i just run the CBT reset on the source VM's or is there a different procedure that needs to be done to make sure the destination machines aren't corrupted? Am i also suppose to run the CBT against the replicated VM's and if so do i need to commit existing snapshots first. Is there a script that handles powered off machines, or do i have to do the "Primary Reset CBT (Requires Power-Off)".
Thanks,
-
- Service Provider
- Posts: 25
- Liked: 2 times
- Joined: Jul 21, 2010 8:55 pm
- Full Name: Mike
- Location: Fargo, ND
- Contact:
Re: [KB2136854] There's a new CBT bug in ESXi 6
Can I ask a question regarding the resting of the CBT data?
I understand that resetting CBT is required and I've run those scripts before, but I'm curious to the difference between running the scripts and toggling the "Changed block tracking" option in individual backup jobs. Does this only tell the job to use/ignore CBT data, but do nothing with the setting on the individual machines? Also, is the disabling CBT in the Veeam job itself not sufficient to avoid possible issues?
-Mike
I understand that resetting CBT is required and I've run those scripts before, but I'm curious to the difference between running the scripts and toggling the "Changed block tracking" option in individual backup jobs. Does this only tell the job to use/ignore CBT data, but do nothing with the setting on the individual machines? Also, is the disabling CBT in the Veeam job itself not sufficient to avoid possible issues?
-Mike
-
- Chief Product Officer
- Posts: 31809
- Liked: 7301 times
- Joined: Jan 01, 2006 1:01 am
- Location: Baar, Switzerland
- Contact:
Re: [KB2136854] There's a new CBT bug in ESXi 6
Yes, you are supposed to run the CBT against the VMs you are replicating as well. There is no need to commit any snapshots though, as you only need to reset CBT for "source" (production) VMs, which should not have any snapshots. You don't need to reset CBT for "target" (replica) VMs which you are probably talking about here.thuizenga wrote:What about replication jobs? Do i just run the CBT reset on the source VM's or is there a different procedure that needs to be done to make sure the destination machines aren't corrupted? Am i also suppose to run the CBT against the replicated VM's and if so do i need to commit existing snapshots first.
Great question, Mike! The difference is that resetting CBT recreates and re-populates CBT map file, which is essential to do when they may contain inconsistency due to the bug discussed in this topic. The option you are talking about simply makes the job not use CBT data - and while disabling this option solves the issue, this is not a type of setup that most people will be happy to live with forever due to the associated performance penalty of "full scan" incremental runs.teknomage wrote:I understand that resetting CBT is required and I've run those scripts before, but I'm curious to the difference between running the scripts and toggling the "Changed block tracking" option in individual backup jobs.
-
- Influencer
- Posts: 24
- Liked: never
- Joined: May 08, 2010 2:00 am
- Full Name: Travis h
- Contact:
Re: [KB2136854] There's a new CBT bug in ESXi 6
Thanks Gostev, you are correct i am strictly talking about the target systems. From a technical perspective when CBT is reset on a replica machine does this invoke the process where source and destinations blocks are compared which would be similar to the initial replica seeding process?
-
- Chief Product Officer
- Posts: 31809
- Liked: 7301 times
- Joined: Jan 01, 2006 1:01 am
- Location: Baar, Switzerland
- Contact:
Re: [KB2136854] There's a new CBT bug in ESXi 6
That is correct, as this is the only way to perform an incremental pass without knowing which blocks have changed. Thanks!
-
- Enthusiast
- Posts: 35
- Liked: 2 times
- Joined: Jan 16, 2015 1:32 pm
- Full Name: Prabhash Kumar Jena
- Contact:
[MERGED] ESXi 6.0 CBT issue
Hi,
I have a general question regarding CBT issue on ESXi 6.0.
Is it same to presume that my environment is not affected with this bug at all if I have not get the message "CBT data is invalid, failing over to legacy incremental backup. No action is required, next job run should start using CBT again. If CBT data remains invalid, follow KB1113 to perform CBT reset. Usual cause is power loss." during the days the bug existed. I have around 30 jobs and I don't see this message on any job, I have checked job history as well.
My colleague has now applied the patch, do we need to carry out some tasks to ensure we have correct backups?
Thanks.
Prabhash Jena
I have a general question regarding CBT issue on ESXi 6.0.
Is it same to presume that my environment is not affected with this bug at all if I have not get the message "CBT data is invalid, failing over to legacy incremental backup. No action is required, next job run should start using CBT again. If CBT data remains invalid, follow KB1113 to perform CBT reset. Usual cause is power loss." during the days the bug existed. I have around 30 jobs and I don't see this message on any job, I have checked job history as well.
My colleague has now applied the patch, do we need to carry out some tasks to ensure we have correct backups?
Thanks.
Prabhash Jena
-
- Chief Product Officer
- Posts: 31809
- Liked: 7301 times
- Joined: Jan 01, 2006 1:01 am
- Location: Baar, Switzerland
- Contact:
Re: [KB2136854] There's a new CBT bug in ESXi 6
Hi. Assuming you are talking about KB2136854, this particular issue causes silent backup data corruption without any error messages logged. Every environment running vSphere 6 is affected, and must perform CBT reset on all existing VMs after installing the patch from KB3247720. Thanks!
-
- Veteran
- Posts: 338
- Liked: 35 times
- Joined: Jan 20, 2012 2:36 pm
- Full Name: Christensen Farms
- Contact:
Re: [KB2136854] There's a new CBT bug in ESXi 6
The VMware KB article mentions that the individual VMs need to be migrated or restarted. I ran the suggested script that disables CBT, then takes and deletes a snapshot, then re-enables CBT. I did not reboot the VMs. Is it still necessary to restart the individual VMs in this case?
-
- Veteran
- Posts: 338
- Liked: 35 times
- Joined: Jan 20, 2012 2:36 pm
- Full Name: Christensen Farms
- Contact:
Re: [KB2136854] There's a new CBT bug in ESXi 6
This is from the VMware KB per my last post saying the VMs have to be rebooted:
Release Date: Nov 25, 2015
Download Filename:
ESXi600-201511001.zip
Build:
3247720
Download Size:
358.9 MB
Host Reboot Required: Yes
Virtual Machine Migration or Shutdown Required: Yes
Release Date: Nov 25, 2015
Download Filename:
ESXi600-201511001.zip
Build:
3247720
Download Size:
358.9 MB
Host Reboot Required: Yes
Virtual Machine Migration or Shutdown Required: Yes
-
- Chief Product Officer
- Posts: 31809
- Liked: 7301 times
- Joined: Jan 01, 2006 1:01 am
- Location: Baar, Switzerland
- Contact:
Re: [KB2136854] There's a new CBT bug in ESXi 6
Sorry, but can you quote that, because I must be blindcffit wrote:VMs have to be rebooted
-
- Veteran
- Posts: 338
- Liked: 35 times
- Joined: Jan 20, 2012 2:36 pm
- Full Name: Christensen Farms
- Contact:
Re: [KB2136854] There's a new CBT bug in ESXi 6
In this KB:
http://kb.vmware.com/selfservice/search ... Id=2137545
"Virtual Machine Migration or Shutdown Required: Yes"
http://kb.vmware.com/selfservice/search ... Id=2137545
"Virtual Machine Migration or Shutdown Required: Yes"
Who is online
Users browsing this forum: No registered users and 53 guests