Discussions specific to the VMware vSphere hypervisor
sphilp
Enthusiast
Posts: 36
Liked: 9 times
Joined: May 28, 2009 7:52 pm
Full Name: Steve Philp
Contact:

ESXi 6.0 CBT issue

Post by sphilp » May 01, 2015 7:11 am 6 people like this post

I don't know if anyone else has run into a new issue with CBT on ESXi 6.0, but wanted to pass along the info in case you do...

We upgraded to vSphere 6 following the release of Veeam Update 2. We immediately started seeing issues with CBT messages during backups and replication jobs on some, but not all, guests within jobs. We worked through the normal "reset CBT" steps of running the Powershell script and also manually resetting CBT via the guest Settings / Datastore Browser, but the message would continue to show up.

Finally took a look at the vmware.log file in the guest and found the following for guests that were failing to use CBT:

2015-05-01T06:31:37.684Z| vcpu-0| I120: DISKLIB-CBT : Creating cbt node 2ac9f88-cbt failed with error Cannot allocate memory (0xbad0014, Out of memory).
2015-05-01T06:31:37.684Z| vcpu-0| W110: DISKLIB-LIB : Could not attach vmkernel change tracker: ESXi tracking filter failed (0x143c). Disk will be opened, but change tracking info vill be invalidated.

A quick Google search led to the VMware forums and then to a KB article posted last week about CBT being flat-out busted in ESXi 6.0. The KB article is at http://kb.vmware.com/kb/2114076

Hope it helps someone else...

Vitaliy S.
Product Manager
Posts: 23230
Liked: 1611 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: ESXi 6.0 CBT issue

Post by Vitaliy S. » May 01, 2015 11:12 am

Thanks for sharing this with the community.

Trelor
Influencer
Posts: 17
Liked: 3 times
Joined: Apr 27, 2015 6:02 pm
Contact:

Re: ESXi 6.0 CBT issue

Post by Trelor » May 04, 2015 3:32 pm

Exactly the same issue here thanks for the VMware article.

sphilp
Enthusiast
Posts: 36
Liked: 9 times
Joined: May 28, 2009 7:52 pm
Full Name: Steve Philp
Contact:

Re: ESXi 6.0 CBT issue

Post by sphilp » May 05, 2015 2:45 am

Does anyone have suggestions on effectively dealing with the need to turn off CBT on the guests and Veeam jobs? Our backup time went from 45 minutes to just over 10 hours.

benbour
Enthusiast
Posts: 35
Liked: 5 times
Joined: May 05, 2015 9:48 pm
Full Name: Ben Bourcier
Contact:

Re: ESXi 6.0 CBT issue

Post by benbour » May 05, 2015 9:57 pm 1 person likes this post

I am also looking for a work-around or solution to this. I upgraded my vCenter and ESXi hosts to 6.0 and have many servers in my backup jobs now producing "CBT data invalid" message.

CBT reset, like sphilp, has been tried to no avail. I have a ticket logged with Veeam and VMware. Vmware pointed me to the same KB article http://kb.vmware.com/kb/2114076

I also got this response:

"The issue will be resolved in a patch - however it will be some months - there is no schedule against the patch as yet. "

So it sounds like we're among the first to report this issue, as VMware maybe hadn't tested quite well enough before the release. I struggle to get daily backups of my VMs as they take over 10 hours. This leaves me really only weekends to get a good backup which isn't good enough.

Looks like we're a long way off from a fix!

Thought I'd make an account and share....

TomF
Novice
Posts: 4
Liked: never
Joined: May 06, 2015 12:56 pm
Contact:

Re: ESXi 6.0 CBT issue

Post by TomF » May 06, 2015 1:33 pm

We downgraded our ESXi installations back to 5.5 while leaving vCenter at 6 -- this solved the problem for us, for now.

lp@albersdruck.de
Enthusiast
Posts: 82
Liked: 33 times
Joined: Mar 25, 2013 7:37 pm
Full Name: Lars Pisanec
Contact:

Re: ESXi 6.0 CBT issue

Post by lp@albersdruck.de » May 06, 2015 3:22 pm

So basically VMware said "stay away from 6.0 until the patch is available". At least with Veeam I know that such blunders get fixed in a reasonable time frame.
Well so far I only use vCenter 6.0 for the new web client and I plan to upgrade my hosts sometime later this year when Mellanox releases driver support for that. So I can thank Mellanox for delaying my host upgrade :)

benbour
Enthusiast
Posts: 35
Liked: 5 times
Joined: May 05, 2015 9:48 pm
Full Name: Ben Bourcier
Contact:

Re: ESXi 6.0 CBT issue

Post by benbour » May 06, 2015 3:40 pm

Unfortunately for me, I also upgraded vmware hardware to v 11, so going back is not really an option.

joejoe
Novice
Posts: 5
Liked: never
Joined: Apr 28, 2015 5:38 pm
Contact:

Re: ESXi 6.0 CBT issue

Post by joejoe » May 06, 2015 5:03 pm

Does this mean I might have a better shot at getting the wonderful folks at Veeam to add an option to disable CBT in the free version now? :) Create snap, backup, delete snap. No other changes to the VM please :wink:

Craigb
Influencer
Posts: 13
Liked: 25 times
Joined: Nov 14, 2012 2:28 am
Full Name: Craig Braithwaite
Contact:

Re: ESXi 6.0 CBT issue

Post by Craigb » May 07, 2015 7:30 am 1 person likes this post

Just passing on some happy news, vmware support seem to have a note on the kb2114076 issue that a resolution might be coming in the form of vmware 6.0 express pack 02 from the patch repository, tentatively by end of month (May 2015).

:D

benbour
Enthusiast
Posts: 35
Liked: 5 times
Joined: May 05, 2015 9:48 pm
Full Name: Ben Bourcier
Contact:

Re: ESXi 6.0 CBT issue

Post by benbour » May 07, 2015 12:31 pm

That would be great! I didn't see a note on the KB, can you show me where you saw that info?

Thanks

saintdle
Veeam Vanguard
Posts: 83
Liked: 15 times
Joined: Aug 05, 2014 1:13 pm
Full Name: Dean lewis
Contact:

Re: ESXi 6.0 CBT issue

Post by saintdle » May 11, 2015 12:18 pm

I've wrote a quick blog post on this, seen the KB from VMware when it came out due to a VMware guy I follow on twitter.

Just to add my two cents,

I'm working at a customer site where we have deployed a brand new vCenter and hosts. whilst on site (first day) we decided to run vSphere 6.0 instead of 5.5 u2, as we can replicate machines from ESXi 6.0 to ESXi 5.5 so long as we don't upgrade the hardware tools.

So we deployed vCenter 6.0, but hosts were built on 5,5 u2d, added into vCenter. Configured environment and added all production VM's to these hosts off the old hosts, which were on 5.5 u2d.

The VM's were backuped up on the old hosts 5.5u2d, then moved to the new hosts that run 5.5ud, we upgraded the hosts to 6.0, then backuped the VMs afterwords. I.e they were not backed up on the new hosts whilst the code was 5.5 u2d.

Over the weekend, we have noticed out of 25 VMs, three of them display the CBT issue mentioned, the others are all fine!!!! I've tried creating new jobs, doesnt work, and following the CBT reset procedure which also doesnt work.
Technical Architect
Veeam Certified Architect
Various other stuff
Personal Technical Blog - www.educationalcentre.co.uk

benbour
Enthusiast
Posts: 35
Liked: 5 times
Joined: May 05, 2015 9:48 pm
Full Name: Ben Bourcier
Contact:

Re: ESXi 6.0 CBT issue

Post by benbour » May 11, 2015 12:31 pm

Thanks for the info - I'm in a similar spot. After upgrade both vCenter and ESXi hosts to 6.0, we had some vms remain backing up fine, as though nothing had changed. Others would produce CBT issue. Of course the vms with issue have large disks and take a long time to backup.

saintdle
Veeam Vanguard
Posts: 83
Liked: 15 times
Joined: Aug 05, 2014 1:13 pm
Full Name: Dean lewis
Contact:

Re: ESXi 6.0 CBT issue

Post by saintdle » May 11, 2015 1:06 pm 1 person likes this post

benbour wrote:Thanks for the info - I'm in a similar spot. After upgrade both vCenter and ESXi hosts to 6.0, we had some vms remain backing up fine, as though nothing had changed. Others would produce CBT issue. Of course the vms with issue have large disks and take a long time to backup.
On this site at the moment the VMs affected are not the largest, one of them is actually the smallest VM on the platform.

But having your biggest VMs affected would align with the VMware KB
Technical Architect
Veeam Certified Architect
Various other stuff
Personal Technical Blog - www.educationalcentre.co.uk

rmcmillan
Novice
Posts: 7
Liked: 1 time
Joined: May 11, 2015 6:52 pm
Full Name: rmcmillan
Contact:

Re: ESXi 6.0 CBT issue

Post by rmcmillan » May 11, 2015 7:05 pm 1 person likes this post

We have been severely affected by this CBT issue I believe as well.

After updating everything to ESXi 6.0 and Veeam 8 update 2, we ran our first backup against all of our Vmware virtual servers.

The first thing we noticed was that our backups are now super slow...that wasn't the main issue however. :oops:

After a full Exchange backup finished and the snapshot merge took place, 2 of the 5 mailbox stores dismounted.

It took Microsoft to get the stores back to usable and mount. After dealing with that we ran another backup(not knowing what the root cause was) on Thursday night.

This took 4 of the mailbox stores offline, and even microsoft couldn't bring them back, too corrupted...We restored that backup and recovered the databases with eseutil /R, and all databases now mounted and mail was flowing again.

We are now in the process of migrating all user mailboxes to a new virtual machine and new mailbox databases for fear the corruption is deeper seeded. We will use Windows server backup to backup exchange until Vmware fixes this critical bug.

I have cases open with Veeam and Microsoft, but I now believe the issue lies with CBT and snapshot merge with Vmware.

We updated because of a Server 2012 reboot issue which was fixed in 6.0, but the cost was far too high.

benbour
Enthusiast
Posts: 35
Liked: 5 times
Joined: May 05, 2015 9:48 pm
Full Name: Ben Bourcier
Contact:

Re: ESXi 6.0 CBT issue

Post by benbour » May 11, 2015 11:15 pm

Which version of exchange are you using? if 2007, does Veeam truncate log files?

sphilp
Enthusiast
Posts: 36
Liked: 9 times
Joined: May 28, 2009 7:52 pm
Full Name: Steve Philp
Contact:

Re: ESXi 6.0 CBT issue

Post by sphilp » May 12, 2015 1:46 am

rmcmillan - you're not alone in experiencing Exchange corruption. Until your post, we weren't sure the Exchange 2013 issue we experienced was related to the CBT issue or not.

We ended up with 2 out of 3 mailbox databases unable to be mounted, restored to a pre-6 backup and made sure not to back it up until we'd turned off CBT. We haven't experienced the corruption issue since then.

We also experienced some SQL Transaction log issues, but didn't require a VM restore for those (hallelujah). We ended up changing the DB recovery model to simple, taking a full SQL database backup, then switching them back to Full recovery. No further corruption / warnings since we turned off CBT.

VMware's got a massive problem on their hands and I wish they'd hurry up and fix it!

Craigb
Influencer
Posts: 13
Liked: 25 times
Joined: Nov 14, 2012 2:28 am
Full Name: Craig Braithwaite
Contact:

Re: ESXi 6.0 CBT issue

Post by Craigb » May 12, 2015 6:12 am

benbour wrote:That would be great! I didn't see a note on the KB, can you show me where you saw that info?

Thanks
Hi Benbour,
It was a comment by one of the support team. While my vCenter was rebooting on a separate issue I asked if there was any news. My environment is on hold until they come out with a resolution.
Cheers ../cb

rmcmillan
Novice
Posts: 7
Liked: 1 time
Joined: May 11, 2015 6:52 pm
Full Name: rmcmillan
Contact:

Re: ESXi 6.0 CBT issue

Post by rmcmillan » May 12, 2015 2:22 pm

benbour wrote:Which version of exchange are you using? if 2007, does Veeam truncate log files?

Exchange 2013.

rmcmillan
Novice
Posts: 7
Liked: 1 time
Joined: May 11, 2015 6:52 pm
Full Name: rmcmillan
Contact:

Re: ESXi 6.0 CBT issue

Post by rmcmillan » May 12, 2015 2:25 pm

sphilp wrote:rmcmillan - you're not alone in experiencing Exchange corruption. Until your post, we weren't sure the Exchange 2013 issue we experienced was related to the CBT issue or not.

We ended up with 2 out of 3 mailbox databases unable to be mounted, restored to a pre-6 backup and made sure not to back it up until we'd turned off CBT. We haven't experienced the corruption issue since then.

We also experienced some SQL Transaction log issues, but didn't require a VM restore for those (hallelujah). We ended up changing the DB recovery model to simple, taking a full SQL database backup, then switching them back to Full recovery. No further corruption / warnings since we turned off CBT.

VMware's got a massive problem on their hands and I wish they'd hurry up and fix it!

Thanks for your post, I've opened a ticket with Vmware to try to get some answers on where this patch is. I don't "think" I have SQL issues, but will be check that now! :shock:

rmcmillan
Novice
Posts: 7
Liked: 1 time
Joined: May 11, 2015 6:52 pm
Full Name: rmcmillan
Contact:

Re: ESXi 6.0 CBT issue

Post by rmcmillan » May 12, 2015 2:48 pm

Just got off the phone with VMware.

They have a patch and its scheduled to be released on Thursday May 14th!

benbour
Enthusiast
Posts: 35
Liked: 5 times
Joined: May 05, 2015 9:48 pm
Full Name: Ben Bourcier
Contact:

Re: ESXi 6.0 CBT issue

Post by benbour » May 12, 2015 2:56 pm

rmcmillan wrote:Just got off the phone with VMware.

They have a patch and its scheduled to be released on Thursday May 14th!

Excellent!

sphilp
Enthusiast
Posts: 36
Liked: 9 times
Joined: May 28, 2009 7:52 pm
Full Name: Steve Philp
Contact:

Re: ESXi 6.0 CBT issue

Post by sphilp » May 12, 2015 3:55 pm

Woo hoo!

unsichtbarre
Expert
Posts: 130
Liked: 24 times
Joined: Mar 08, 2010 4:05 pm
Full Name: John Borhek
Contact:

Re: ESXi 6.0 CBT issue

Post by unsichtbarre » May 12, 2015 4:29 pm

I am so glad we held off on vSphere 6 for the time being!

It would not qualify for us because ESXi/AD/VCSA and the vCLI (direct-host connection) seem to be at lease partly broken, but this CBT thing is amazing! I thought VMware was so careful this time around!
-The Invisible Admin-
http://www.johnborhek.com

jmerfeld
Influencer
Posts: 16
Liked: 1 time
Joined: Apr 05, 2013 12:59 am
Full Name: James Merfeld
Contact:

Re: ESXi 6.0 CBT issue

Post by jmerfeld » May 12, 2015 4:57 pm

We got the pleasure of restoring our exchange db this morning.

thanks Vmware!

joergr
Expert
Posts: 386
Liked: 39 times
Joined: Jun 08, 2010 2:01 pm
Full Name: Joerg Riether
Contact:

Re: ESXi 6.0 CBT issue

Post by joergr » May 12, 2015 6:00 pm 1 person likes this post

I posted it to vSphere ESXi 6 community this morning - https://communities.vmware.com/message/2506025#2506025

Best regards,
Joerg

benbour
Enthusiast
Posts: 35
Liked: 5 times
Joined: May 05, 2015 9:48 pm
Full Name: Ben Bourcier
Contact:

Re: ESXi 6.0 CBT issue

Post by benbour » May 12, 2015 7:57 pm

joergr wrote:I posted it to vSphere ESXi 6 community this morning - https://communities.vmware.com/message/2506025#2506025

Best regards,
Joerg

Thanks for that, I'll be following that thread too.

LorettoCNY
Novice
Posts: 8
Liked: 3 times
Joined: May 12, 2015 12:56 pm
Full Name: Brandon
Contact:

Re: ESXi 6.0 CBT issue

Post by LorettoCNY » May 13, 2015 3:39 am

I'm in the same boat as everyone else that has posted. :cry:

NightBird
Service Provider
Posts: 178
Liked: 32 times
Joined: Apr 28, 2009 8:33 am
Location: Strasbourg, FRANCE
Contact:

Re: ESXi 6.0 CBT issue

Post by NightBird » May 13, 2015 7:28 am

You think that it is a Veeam B&R 8 Update 2 bug or vSphere 6 bug ?

foggy
Veeam Software
Posts: 18458
Liked: 1589 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: ESXi 6.0 CBT issue

Post by foggy » May 13, 2015 7:35 am

NightBird wrote:You think that it is a Veeam B&R 8 Update 2 bug or vSphere 6 bug ?
If you review the topic more carefully, you'll find that the issue is acknowledged by VMware and they are going to fix it in a short term.

Post Reply

Who is online

Users browsing this forum: Google [Bot] and 9 guests