ESXi 6.0 CBT issue

[sphilp]

I don't know if anyone else has run into a new issue with CBT on ESXi 6.0, but wanted to pass along the info in case you do...

We upgraded to vSphere 6 following the release of Veeam Update 2. We immediately started seeing issues with CBT messages during backups and replication jobs on some, but not all, guests within jobs. We worked through the normal "reset CBT" steps of running the Powershell script and also manually resetting CBT via the guest Settings / Datastore Browser, but the message would continue to show up.

Finally took a look at the vmware.log file in the guest and found the following for guests that were failing to use CBT:

2015-05-01T06:31:37.684Z| vcpu-0| I120: DISKLIB-CBT : Creating cbt node 2ac9f88-cbt failed with error Cannot allocate memory (0xbad0014, Out of memory).
2015-05-01T06:31:37.684Z| vcpu-0| W110: DISKLIB-LIB : Could not attach vmkernel change tracker: ESXi tracking filter failed (0x143c). Disk will be opened, but change tracking info vill be invalidated.

A quick Google search led to the VMware forums and then to a KB article posted last week about CBT being flat-out busted in ESXi 6.0. The KB article is at http://kb.vmware.com/kb/2114076

Hope it helps someone else...

[Vitaliy S.]

Thanks for sharing this with the community.

[Trelor]

Exactly the same issue here thanks for the VMware article.

[sphilp]

Does anyone have suggestions on effectively dealing with the need to turn off CBT on the guests and Veeam jobs? Our backup time went from 45 minutes to just over 10 hours.

[benbour]

I am also looking for a work-around or solution to this. I upgraded my vCenter and ESXi hosts to 6.0 and have many servers in my backup jobs now producing "CBT data invalid" message.

CBT reset, like sphilp, has been tried to no avail. I have a ticket logged with Veeam and VMware. Vmware pointed me to the same KB article http://kb.vmware.com/kb/2114076

I also got this response:

"The issue will be resolved in a patch - however it will be some months - there is no schedule against the patch as yet. "

So it sounds like we're among the first to report this issue, as VMware maybe hadn't tested quite well enough before the release. I struggle to get daily backups of my VMs as they take over 10 hours. This leaves me really only weekends to get a good backup which isn't good enough.

Looks like we're a long way off from a fix!

Thought I'd make an account and share....

[TomF]

We downgraded our ESXi installations back to 5.5 while leaving vCenter at 6 -- this solved the problem for us, for now.

[lp@albersdruck.de]

So basically VMware said "stay away from 6.0 until the patch is available". At least with Veeam I know that such blunders get fixed in a reasonable time frame.
Well so far I only use vCenter 6.0 for the new web client and I plan to upgrade my hosts sometime later this year when Mellanox releases driver support for that. So I can thank Mellanox for delaying my host upgrade

[benbour]

Unfortunately for me, I also upgraded vmware hardware to v 11, so going back is not really an option.

[joejoe]

Does this mean I might have a better shot at getting the wonderful folks at Veeam to add an option to disable CBT in the free version now? Create snap, backup, delete snap. No other changes to the VM please

[Craigb]

Just passing on some happy news, vmware support seem to have a note on the kb2114076 issue that a resolution might be coming in the form of vmware 6.0 express pack 02 from the patch repository, tentatively by end of month (May 2015).

[benbour]

That would be great! I didn't see a note on the KB, can you show me where you saw that info?

Thanks

[saintdle]

I've wrote a quick blog post on this, seen the KB from VMware when it came out due to a VMware guy I follow on twitter.

Just to add my two cents,

I'm working at a customer site where we have deployed a brand new vCenter and hosts. whilst on site (first day) we decided to run vSphere 6.0 instead of 5.5 u2, as we can replicate machines from ESXi 6.0 to ESXi 5.5 so long as we don't upgrade the hardware tools.

So we deployed vCenter 6.0, but hosts were built on 5,5 u2d, added into vCenter. Configured environment and added all production VM's to these hosts off the old hosts, which were on 5.5 u2d.

The VM's were backuped up on the old hosts 5.5u2d, then moved to the new hosts that run 5.5ud, we upgraded the hosts to 6.0, then backuped the VMs afterwords. I.e they were not backed up on the new hosts whilst the code was 5.5 u2d.

Over the weekend, we have noticed out of 25 VMs, three of them display the CBT issue mentioned, the others are all fine!!!! I've tried creating new jobs, doesnt work, and following the CBT reset procedure which also doesnt work.

[benbour]

Thanks for the info - I'm in a similar spot. After upgrade both vCenter and ESXi hosts to 6.0, we had some vms remain backing up fine, as though nothing had changed. Others would produce CBT issue. Of course the vms with issue have large disks and take a long time to backup.

[saintdle]

Thanks for the info - I'm in a similar spot. After upgrade both vCenter and ESXi hosts to 6.0, we had some vms remain backing up fine, as though nothing had changed. Others would produce CBT issue. Of course the vms with issue have large disks and take a long time to backup.

On this site at the moment the VMs affected are not the largest, one of them is actually the smallest VM on the platform.

But having your biggest VMs affected would align with the VMware KB

[rmcmillan]

We have been severely affected by this CBT issue I believe as well.

After updating everything to ESXi 6.0 and Veeam 8 update 2, we ran our first backup against all of our Vmware virtual servers.

The first thing we noticed was that our backups are now super slow...that wasn't the main issue however.

After a full Exchange backup finished and the snapshot merge took place, 2 of the 5 mailbox stores dismounted.

It took Microsoft to get the stores back to usable and mount. After dealing with that we ran another backup(not knowing what the root cause was) on Thursday night.

This took 4 of the mailbox stores offline, and even microsoft couldn't bring them back, too corrupted...We restored that backup and recovered the databases with eseutil /R, and all databases now mounted and mail was flowing again.

We are now in the process of migrating all user mailboxes to a new virtual machine and new mailbox databases for fear the corruption is deeper seeded. We will use Windows server backup to backup exchange until Vmware fixes this critical bug.

I have cases open with Veeam and Microsoft, but I now believe the issue lies with CBT and snapshot merge with Vmware.

We updated because of a Server 2012 reboot issue which was fixed in 6.0, but the cost was far too high.

[benbour]

Which version of exchange are you using? if 2007, does Veeam truncate log files?

[sphilp]

rmcmillan - you're not alone in experiencing Exchange corruption. Until your post, we weren't sure the Exchange 2013 issue we experienced was related to the CBT issue or not.

We ended up with 2 out of 3 mailbox databases unable to be mounted, restored to a pre-6 backup and made sure not to back it up until we'd turned off CBT. We haven't experienced the corruption issue since then.

We also experienced some SQL Transaction log issues, but didn't require a VM restore for those (hallelujah). We ended up changing the DB recovery model to simple, taking a full SQL database backup, then switching them back to Full recovery. No further corruption / warnings since we turned off CBT.

VMware's got a massive problem on their hands and I wish they'd hurry up and fix it!

[Craigb]

That would be great! I didn't see a note on the KB, can you show me where you saw that info?

Thanks

Hi Benbour,
It was a comment by one of the support team. While my vCenter was rebooting on a separate issue I asked if there was any news. My environment is on hold until they come out with a resolution.
Cheers ../cb

[rmcmillan]

Which version of exchange are you using? if 2007, does Veeam truncate log files?

Exchange 2013.

[rmcmillan]

rmcmillan - you're not alone in experiencing Exchange corruption. Until your post, we weren't sure the Exchange 2013 issue we experienced was related to the CBT issue or not.

We ended up with 2 out of 3 mailbox databases unable to be mounted, restored to a pre-6 backup and made sure not to back it up until we'd turned off CBT. We haven't experienced the corruption issue since then.

We also experienced some SQL Transaction log issues, but didn't require a VM restore for those (hallelujah). We ended up changing the DB recovery model to simple, taking a full SQL database backup, then switching them back to Full recovery. No further corruption / warnings since we turned off CBT.

VMware's got a massive problem on their hands and I wish they'd hurry up and fix it!

Thanks for your post, I've opened a ticket with Vmware to try to get some answers on where this patch is. I don't "think" I have SQL issues, but will be check that now!

[rmcmillan]

Just got off the phone with VMware.

They have a patch and its scheduled to be released on Thursday May 14th!

[benbour]

Just got off the phone with VMware.

They have a patch and its scheduled to be released on Thursday May 14th!

Excellent!

[sphilp]

Woo hoo!

[unsichtbarre]

I am so glad we held off on vSphere 6 for the time being!

It would not qualify for us because ESXi/AD/VCSA and the vCLI (direct-host connection) seem to be at lease partly broken, but this CBT thing is amazing! I thought VMware was so careful this time around!

[jmerfeld]

We got the pleasure of restoring our exchange db this morning.

thanks Vmware!

[joergr]

I posted it to vSphere ESXi 6 community this morning - https://communities.vmware.com/message/2506025#2506025

Best regards,
Joerg

[benbour]

I posted it to vSphere ESXi 6 community this morning - https://communities.vmware.com/message/2506025#2506025

Best regards,
Joerg

Thanks for that, I'll be following that thread too.

[LorettoCNY]

I'm in the same boat as everyone else that has posted.

[NightBird]

You think that it is a Veeam B&R 8 Update 2 bug or vSphere 6 bug ?

[foggy]

You think that it is a Veeam B&R 8 Update 2 bug or vSphere 6 bug ?

If you review the topic more carefully, you'll find that the issue is acknowledged by VMware and they are going to fix it in a short term.