Host-based backup of VMware vSphere VMs.
sphilp
Enthusiast
Posts: 36
Liked: 9 times
Joined: May 28, 2009 7:52 pm
Full Name: Steve Philp
Contact:

ESXi 6.0 CBT issue

Post by sphilp » 6 people like this post

I don't know if anyone else has run into a new issue with CBT on ESXi 6.0, but wanted to pass along the info in case you do...

We upgraded to vSphere 6 following the release of Veeam Update 2. We immediately started seeing issues with CBT messages during backups and replication jobs on some, but not all, guests within jobs. We worked through the normal "reset CBT" steps of running the Powershell script and also manually resetting CBT via the guest Settings / Datastore Browser, but the message would continue to show up.

Finally took a look at the vmware.log file in the guest and found the following for guests that were failing to use CBT:

2015-05-01T06:31:37.684Z| vcpu-0| I120: DISKLIB-CBT : Creating cbt node 2ac9f88-cbt failed with error Cannot allocate memory (0xbad0014, Out of memory).
2015-05-01T06:31:37.684Z| vcpu-0| W110: DISKLIB-LIB : Could not attach vmkernel change tracker: ESXi tracking filter failed (0x143c). Disk will be opened, but change tracking info vill be invalidated.

A quick Google search led to the VMware forums and then to a KB article posted last week about CBT being flat-out busted in ESXi 6.0. The KB article is at http://kb.vmware.com/kb/2114076

Hope it helps someone else...
Vitaliy S.
VP, Product Management
Posts: 27055
Liked: 2710 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: ESXi 6.0 CBT issue

Post by Vitaliy S. »

Thanks for sharing this with the community.
Trelor
Enthusiast
Posts: 47
Liked: 15 times
Joined: Apr 27, 2015 6:02 pm
Contact:

Re: ESXi 6.0 CBT issue

Post by Trelor »

Exactly the same issue here thanks for the VMware article.
sphilp
Enthusiast
Posts: 36
Liked: 9 times
Joined: May 28, 2009 7:52 pm
Full Name: Steve Philp
Contact:

Re: ESXi 6.0 CBT issue

Post by sphilp »

Does anyone have suggestions on effectively dealing with the need to turn off CBT on the guests and Veeam jobs? Our backup time went from 45 minutes to just over 10 hours.
benbour
Enthusiast
Posts: 35
Liked: 5 times
Joined: May 05, 2015 9:48 pm
Full Name: Ben Bourcier
Contact:

Re: ESXi 6.0 CBT issue

Post by benbour » 1 person likes this post

I am also looking for a work-around or solution to this. I upgraded my vCenter and ESXi hosts to 6.0 and have many servers in my backup jobs now producing "CBT data invalid" message.

CBT reset, like sphilp, has been tried to no avail. I have a ticket logged with Veeam and VMware. Vmware pointed me to the same KB article http://kb.vmware.com/kb/2114076

I also got this response:

"The issue will be resolved in a patch - however it will be some months - there is no schedule against the patch as yet. "

So it sounds like we're among the first to report this issue, as VMware maybe hadn't tested quite well enough before the release. I struggle to get daily backups of my VMs as they take over 10 hours. This leaves me really only weekends to get a good backup which isn't good enough.

Looks like we're a long way off from a fix!

Thought I'd make an account and share....
TomF
Novice
Posts: 4
Liked: never
Joined: May 06, 2015 12:56 pm
Contact:

Re: ESXi 6.0 CBT issue

Post by TomF »

We downgraded our ESXi installations back to 5.5 while leaving vCenter at 6 -- this solved the problem for us, for now.
lp@albersdruck.de
Enthusiast
Posts: 82
Liked: 33 times
Joined: Mar 25, 2013 7:37 pm
Full Name: Lars Pisanec
Contact:

Re: ESXi 6.0 CBT issue

Post by lp@albersdruck.de »

So basically VMware said "stay away from 6.0 until the patch is available". At least with Veeam I know that such blunders get fixed in a reasonable time frame.
Well so far I only use vCenter 6.0 for the new web client and I plan to upgrade my hosts sometime later this year when Mellanox releases driver support for that. So I can thank Mellanox for delaying my host upgrade :)
benbour
Enthusiast
Posts: 35
Liked: 5 times
Joined: May 05, 2015 9:48 pm
Full Name: Ben Bourcier
Contact:

Re: ESXi 6.0 CBT issue

Post by benbour »

Unfortunately for me, I also upgraded vmware hardware to v 11, so going back is not really an option.
joejoe
Novice
Posts: 5
Liked: never
Joined: Apr 28, 2015 5:38 pm
Contact:

Re: ESXi 6.0 CBT issue

Post by joejoe »

Does this mean I might have a better shot at getting the wonderful folks at Veeam to add an option to disable CBT in the free version now? :) Create snap, backup, delete snap. No other changes to the VM please :wink:
Craigb
Influencer
Posts: 15
Liked: 28 times
Joined: Nov 14, 2012 2:28 am
Full Name: Craig Braithwaite
Contact:

Re: ESXi 6.0 CBT issue

Post by Craigb » 1 person likes this post

Just passing on some happy news, vmware support seem to have a note on the kb2114076 issue that a resolution might be coming in the form of vmware 6.0 express pack 02 from the patch repository, tentatively by end of month (May 2015).

:D
benbour
Enthusiast
Posts: 35
Liked: 5 times
Joined: May 05, 2015 9:48 pm
Full Name: Ben Bourcier
Contact:

Re: ESXi 6.0 CBT issue

Post by benbour »

That would be great! I didn't see a note on the KB, can you show me where you saw that info?

Thanks
saintdle
Veeam Vanguard
Posts: 103
Liked: 17 times
Joined: Aug 05, 2014 1:13 pm
Full Name: Dean lewis
Contact:

Re: ESXi 6.0 CBT issue

Post by saintdle »

I've wrote a quick blog post on this, seen the KB from VMware when it came out due to a VMware guy I follow on twitter.

Just to add my two cents,

I'm working at a customer site where we have deployed a brand new vCenter and hosts. whilst on site (first day) we decided to run vSphere 6.0 instead of 5.5 u2, as we can replicate machines from ESXi 6.0 to ESXi 5.5 so long as we don't upgrade the hardware tools.

So we deployed vCenter 6.0, but hosts were built on 5,5 u2d, added into vCenter. Configured environment and added all production VM's to these hosts off the old hosts, which were on 5.5 u2d.

The VM's were backuped up on the old hosts 5.5u2d, then moved to the new hosts that run 5.5ud, we upgraded the hosts to 6.0, then backuped the VMs afterwords. I.e they were not backed up on the new hosts whilst the code was 5.5 u2d.

Over the weekend, we have noticed out of 25 VMs, three of them display the CBT issue mentioned, the others are all fine!!!! I've tried creating new jobs, doesnt work, and following the CBT reset procedure which also doesnt work.
Technical Architect
Veeam Certified Architect
Veeam Vanguard
  • Personal Technical Blog - www.veducate.co.uk
  • Twitter - @saintdle
benbour
Enthusiast
Posts: 35
Liked: 5 times
Joined: May 05, 2015 9:48 pm
Full Name: Ben Bourcier
Contact:

Re: ESXi 6.0 CBT issue

Post by benbour »

Thanks for the info - I'm in a similar spot. After upgrade both vCenter and ESXi hosts to 6.0, we had some vms remain backing up fine, as though nothing had changed. Others would produce CBT issue. Of course the vms with issue have large disks and take a long time to backup.
saintdle
Veeam Vanguard
Posts: 103
Liked: 17 times
Joined: Aug 05, 2014 1:13 pm
Full Name: Dean lewis
Contact:

Re: ESXi 6.0 CBT issue

Post by saintdle » 1 person likes this post

benbour wrote:Thanks for the info - I'm in a similar spot. After upgrade both vCenter and ESXi hosts to 6.0, we had some vms remain backing up fine, as though nothing had changed. Others would produce CBT issue. Of course the vms with issue have large disks and take a long time to backup.
On this site at the moment the VMs affected are not the largest, one of them is actually the smallest VM on the platform.

But having your biggest VMs affected would align with the VMware KB
Technical Architect
Veeam Certified Architect
Veeam Vanguard
  • Personal Technical Blog - www.veducate.co.uk
  • Twitter - @saintdle
rmcmillan
Novice
Posts: 7
Liked: 1 time
Joined: May 11, 2015 6:52 pm
Full Name: rmcmillan
Contact:

Re: ESXi 6.0 CBT issue

Post by rmcmillan » 1 person likes this post

We have been severely affected by this CBT issue I believe as well.

After updating everything to ESXi 6.0 and Veeam 8 update 2, we ran our first backup against all of our Vmware virtual servers.

The first thing we noticed was that our backups are now super slow...that wasn't the main issue however. :oops:

After a full Exchange backup finished and the snapshot merge took place, 2 of the 5 mailbox stores dismounted.

It took Microsoft to get the stores back to usable and mount. After dealing with that we ran another backup(not knowing what the root cause was) on Thursday night.

This took 4 of the mailbox stores offline, and even microsoft couldn't bring them back, too corrupted...We restored that backup and recovered the databases with eseutil /R, and all databases now mounted and mail was flowing again.

We are now in the process of migrating all user mailboxes to a new virtual machine and new mailbox databases for fear the corruption is deeper seeded. We will use Windows server backup to backup exchange until Vmware fixes this critical bug.

I have cases open with Veeam and Microsoft, but I now believe the issue lies with CBT and snapshot merge with Vmware.

We updated because of a Server 2012 reboot issue which was fixed in 6.0, but the cost was far too high.
benbour
Enthusiast
Posts: 35
Liked: 5 times
Joined: May 05, 2015 9:48 pm
Full Name: Ben Bourcier
Contact:

Re: ESXi 6.0 CBT issue

Post by benbour »

Which version of exchange are you using? if 2007, does Veeam truncate log files?
sphilp
Enthusiast
Posts: 36
Liked: 9 times
Joined: May 28, 2009 7:52 pm
Full Name: Steve Philp
Contact:

Re: ESXi 6.0 CBT issue

Post by sphilp »

rmcmillan - you're not alone in experiencing Exchange corruption. Until your post, we weren't sure the Exchange 2013 issue we experienced was related to the CBT issue or not.

We ended up with 2 out of 3 mailbox databases unable to be mounted, restored to a pre-6 backup and made sure not to back it up until we'd turned off CBT. We haven't experienced the corruption issue since then.

We also experienced some SQL Transaction log issues, but didn't require a VM restore for those (hallelujah). We ended up changing the DB recovery model to simple, taking a full SQL database backup, then switching them back to Full recovery. No further corruption / warnings since we turned off CBT.

VMware's got a massive problem on their hands and I wish they'd hurry up and fix it!
Craigb
Influencer
Posts: 15
Liked: 28 times
Joined: Nov 14, 2012 2:28 am
Full Name: Craig Braithwaite
Contact:

Re: ESXi 6.0 CBT issue

Post by Craigb »

benbour wrote:That would be great! I didn't see a note on the KB, can you show me where you saw that info?

Thanks
Hi Benbour,
It was a comment by one of the support team. While my vCenter was rebooting on a separate issue I asked if there was any news. My environment is on hold until they come out with a resolution.
Cheers ../cb
rmcmillan
Novice
Posts: 7
Liked: 1 time
Joined: May 11, 2015 6:52 pm
Full Name: rmcmillan
Contact:

Re: ESXi 6.0 CBT issue

Post by rmcmillan »

benbour wrote:Which version of exchange are you using? if 2007, does Veeam truncate log files?

Exchange 2013.
rmcmillan
Novice
Posts: 7
Liked: 1 time
Joined: May 11, 2015 6:52 pm
Full Name: rmcmillan
Contact:

Re: ESXi 6.0 CBT issue

Post by rmcmillan »

sphilp wrote:rmcmillan - you're not alone in experiencing Exchange corruption. Until your post, we weren't sure the Exchange 2013 issue we experienced was related to the CBT issue or not.

We ended up with 2 out of 3 mailbox databases unable to be mounted, restored to a pre-6 backup and made sure not to back it up until we'd turned off CBT. We haven't experienced the corruption issue since then.

We also experienced some SQL Transaction log issues, but didn't require a VM restore for those (hallelujah). We ended up changing the DB recovery model to simple, taking a full SQL database backup, then switching them back to Full recovery. No further corruption / warnings since we turned off CBT.

VMware's got a massive problem on their hands and I wish they'd hurry up and fix it!

Thanks for your post, I've opened a ticket with Vmware to try to get some answers on where this patch is. I don't "think" I have SQL issues, but will be check that now! :shock:
rmcmillan
Novice
Posts: 7
Liked: 1 time
Joined: May 11, 2015 6:52 pm
Full Name: rmcmillan
Contact:

Re: ESXi 6.0 CBT issue

Post by rmcmillan »

Just got off the phone with VMware.

They have a patch and its scheduled to be released on Thursday May 14th!
benbour
Enthusiast
Posts: 35
Liked: 5 times
Joined: May 05, 2015 9:48 pm
Full Name: Ben Bourcier
Contact:

Re: ESXi 6.0 CBT issue

Post by benbour »

rmcmillan wrote:Just got off the phone with VMware.

They have a patch and its scheduled to be released on Thursday May 14th!

Excellent!
sphilp
Enthusiast
Posts: 36
Liked: 9 times
Joined: May 28, 2009 7:52 pm
Full Name: Steve Philp
Contact:

Re: ESXi 6.0 CBT issue

Post by sphilp »

Woo hoo!
unsichtbarre
Service Provider
Posts: 226
Liked: 39 times
Joined: Mar 08, 2010 4:05 pm
Full Name: John Borhek
Contact:

Re: ESXi 6.0 CBT issue

Post by unsichtbarre »

I am so glad we held off on vSphere 6 for the time being!

It would not qualify for us because ESXi/AD/VCSA and the vCLI (direct-host connection) seem to be at lease partly broken, but this CBT thing is amazing! I thought VMware was so careful this time around!
John Borhek, Solutions Architect
https://vmsources.com
jmerfeld
Influencer
Posts: 18
Liked: 1 time
Joined: Apr 05, 2013 12:59 am
Full Name: James Merfeld
Contact:

Re: ESXi 6.0 CBT issue

Post by jmerfeld »

We got the pleasure of restoring our exchange db this morning.

thanks Vmware!
joergr
Veteran
Posts: 391
Liked: 39 times
Joined: Jun 08, 2010 2:01 pm
Full Name: Joerg Riether
Contact:

Re: ESXi 6.0 CBT issue

Post by joergr » 1 person likes this post

I posted it to vSphere ESXi 6 community this morning - https://communities.vmware.com/message/2506025#2506025

Best regards,
Joerg
benbour
Enthusiast
Posts: 35
Liked: 5 times
Joined: May 05, 2015 9:48 pm
Full Name: Ben Bourcier
Contact:

Re: ESXi 6.0 CBT issue

Post by benbour »

joergr wrote:I posted it to vSphere ESXi 6 community this morning - https://communities.vmware.com/message/2506025#2506025

Best regards,
Joerg

Thanks for that, I'll be following that thread too.
LorettoCNY
Novice
Posts: 8
Liked: 4 times
Joined: May 12, 2015 12:56 pm
Full Name: Brandon
Contact:

Re: ESXi 6.0 CBT issue

Post by LorettoCNY »

I'm in the same boat as everyone else that has posted. :cry:
NightBird
Expert
Posts: 242
Liked: 57 times
Joined: Apr 28, 2009 8:33 am
Location: Strasbourg, FRANCE
Contact:

Re: ESXi 6.0 CBT issue

Post by NightBird »

You think that it is a Veeam B&R 8 Update 2 bug or vSphere 6 bug ?
foggy
Veeam Software
Posts: 21069
Liked: 2115 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: ESXi 6.0 CBT issue

Post by foggy »

NightBird wrote:You think that it is a Veeam B&R 8 Update 2 bug or vSphere 6 bug ?
If you review the topic more carefully, you'll find that the issue is acknowledged by VMware and they are going to fix it in a short term.
Post Reply

Who is online

Users browsing this forum: No registered users and 73 guests