ESXi 6.0 CBT issue

VMware specific discussions

ESXi 6.0 CBT issue

Veeam Logoby sphilp » Fri May 01, 2015 7:11 am 6 people like this post

I don't know if anyone else has run into a new issue with CBT on ESXi 6.0, but wanted to pass along the info in case you do...

We upgraded to vSphere 6 following the release of Veeam Update 2. We immediately started seeing issues with CBT messages during backups and replication jobs on some, but not all, guests within jobs. We worked through the normal "reset CBT" steps of running the Powershell script and also manually resetting CBT via the guest Settings / Datastore Browser, but the message would continue to show up.

Finally took a look at the vmware.log file in the guest and found the following for guests that were failing to use CBT:

2015-05-01T06:31:37.684Z| vcpu-0| I120: DISKLIB-CBT : Creating cbt node 2ac9f88-cbt failed with error Cannot allocate memory (0xbad0014, Out of memory).
2015-05-01T06:31:37.684Z| vcpu-0| W110: DISKLIB-LIB : Could not attach vmkernel change tracker: ESXi tracking filter failed (0x143c). Disk will be opened, but change tracking info vill be invalidated.

A quick Google search led to the VMware forums and then to a KB article posted last week about CBT being flat-out busted in ESXi 6.0. The KB article is at http://kb.vmware.com/kb/2114076

Hope it helps someone else...
sphilp
Enthusiast
 
Posts: 36
Liked: 9 times
Joined: Thu May 28, 2009 7:52 pm
Full Name: Steve Philp

Re: ESXi 6.0 CBT issue

Veeam Logoby Vitaliy S. » Fri May 01, 2015 11:12 am

Thanks for sharing this with the community.
Vitaliy S.
Veeam Software
 
Posts: 19542
Liked: 1098 times
Joined: Mon Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov

Re: ESXi 6.0 CBT issue

Veeam Logoby Trelor » Mon May 04, 2015 3:32 pm

Exactly the same issue here thanks for the VMware article.
Trelor
Novice
 
Posts: 8
Liked: 1 time
Joined: Mon Apr 27, 2015 6:02 pm

Re: ESXi 6.0 CBT issue

Veeam Logoby sphilp » Tue May 05, 2015 2:45 am

Does anyone have suggestions on effectively dealing with the need to turn off CBT on the guests and Veeam jobs? Our backup time went from 45 minutes to just over 10 hours.
sphilp
Enthusiast
 
Posts: 36
Liked: 9 times
Joined: Thu May 28, 2009 7:52 pm
Full Name: Steve Philp

Re: ESXi 6.0 CBT issue

Veeam Logoby benbour » Tue May 05, 2015 9:57 pm 1 person likes this post

I am also looking for a work-around or solution to this. I upgraded my vCenter and ESXi hosts to 6.0 and have many servers in my backup jobs now producing "CBT data invalid" message.

CBT reset, like sphilp, has been tried to no avail. I have a ticket logged with Veeam and VMware. Vmware pointed me to the same KB article http://kb.vmware.com/kb/2114076

I also got this response:

"The issue will be resolved in a patch - however it will be some months - there is no schedule against the patch as yet. "

So it sounds like we're among the first to report this issue, as VMware maybe hadn't tested quite well enough before the release. I struggle to get daily backups of my VMs as they take over 10 hours. This leaves me really only weekends to get a good backup which isn't good enough.

Looks like we're a long way off from a fix!

Thought I'd make an account and share....
benbour
Enthusiast
 
Posts: 33
Liked: 2 times
Joined: Tue May 05, 2015 9:48 pm
Full Name: Ben Bourcier

Re: ESXi 6.0 CBT issue

Veeam Logoby TomF » Wed May 06, 2015 1:33 pm

We downgraded our ESXi installations back to 5.5 while leaving vCenter at 6 -- this solved the problem for us, for now.
TomF
Novice
 
Posts: 4
Liked: never
Joined: Wed May 06, 2015 12:56 pm

Re: ESXi 6.0 CBT issue

Veeam Logoby lp@albersdruck.de » Wed May 06, 2015 3:22 pm

So basically VMware said "stay away from 6.0 until the patch is available". At least with Veeam I know that such blunders get fixed in a reasonable time frame.
Well so far I only use vCenter 6.0 for the new web client and I plan to upgrade my hosts sometime later this year when Mellanox releases driver support for that. So I can thank Mellanox for delaying my host upgrade :)
lp@albersdruck.de
Enthusiast
 
Posts: 81
Liked: 31 times
Joined: Mon Mar 25, 2013 7:37 pm
Full Name: Lars Pisanec

Re: ESXi 6.0 CBT issue

Veeam Logoby benbour » Wed May 06, 2015 3:40 pm

Unfortunately for me, I also upgraded vmware hardware to v 11, so going back is not really an option.
benbour
Enthusiast
 
Posts: 33
Liked: 2 times
Joined: Tue May 05, 2015 9:48 pm
Full Name: Ben Bourcier

Re: ESXi 6.0 CBT issue

Veeam Logoby joejoe » Wed May 06, 2015 5:03 pm

Does this mean I might have a better shot at getting the wonderful folks at Veeam to add an option to disable CBT in the free version now? :) Create snap, backup, delete snap. No other changes to the VM please :wink:
joejoe
Novice
 
Posts: 4
Liked: never
Joined: Tue Apr 28, 2015 5:38 pm

Re: ESXi 6.0 CBT issue

Veeam Logoby Craigb » Thu May 07, 2015 7:30 am 1 person likes this post

Just passing on some happy news, vmware support seem to have a note on the kb2114076 issue that a resolution might be coming in the form of vmware 6.0 express pack 02 from the patch repository, tentatively by end of month (May 2015).

:D
Craigb
Influencer
 
Posts: 12
Liked: 18 times
Joined: Wed Nov 14, 2012 2:28 am
Full Name: Craig Braithwaite

Re: ESXi 6.0 CBT issue

Veeam Logoby benbour » Thu May 07, 2015 12:31 pm

That would be great! I didn't see a note on the KB, can you show me where you saw that info?

Thanks
benbour
Enthusiast
 
Posts: 33
Liked: 2 times
Joined: Tue May 05, 2015 9:48 pm
Full Name: Ben Bourcier

Re: ESXi 6.0 CBT issue

Veeam Logoby saintdle » Mon May 11, 2015 12:18 pm

I've wrote a quick blog post on this, seen the KB from VMware when it came out due to a VMware guy I follow on twitter.

Just to add my two cents,

I'm working at a customer site where we have deployed a brand new vCenter and hosts. whilst on site (first day) we decided to run vSphere 6.0 instead of 5.5 u2, as we can replicate machines from ESXi 6.0 to ESXi 5.5 so long as we don't upgrade the hardware tools.

So we deployed vCenter 6.0, but hosts were built on 5,5 u2d, added into vCenter. Configured environment and added all production VM's to these hosts off the old hosts, which were on 5.5 u2d.

The VM's were backuped up on the old hosts 5.5u2d, then moved to the new hosts that run 5.5ud, we upgraded the hosts to 6.0, then backuped the VMs afterwords. I.e they were not backed up on the new hosts whilst the code was 5.5 u2d.

Over the weekend, we have noticed out of 25 VMs, three of them display the CBT issue mentioned, the others are all fine!!!! I've tried creating new jobs, doesnt work, and following the CBT reset procedure which also doesnt work.
Senior Engineer
VMCE
Various other stuff
Personal Technical Blog - www.educationalcentre.co.uk
saintdle
Veeam Vanguard
 
Posts: 65
Liked: 13 times
Joined: Tue Aug 05, 2014 1:13 pm
Full Name: Dean lewis

Re: ESXi 6.0 CBT issue

Veeam Logoby benbour » Mon May 11, 2015 12:31 pm

Thanks for the info - I'm in a similar spot. After upgrade both vCenter and ESXi hosts to 6.0, we had some vms remain backing up fine, as though nothing had changed. Others would produce CBT issue. Of course the vms with issue have large disks and take a long time to backup.
benbour
Enthusiast
 
Posts: 33
Liked: 2 times
Joined: Tue May 05, 2015 9:48 pm
Full Name: Ben Bourcier

Re: ESXi 6.0 CBT issue

Veeam Logoby saintdle » Mon May 11, 2015 1:06 pm 1 person likes this post

benbour wrote:Thanks for the info - I'm in a similar spot. After upgrade both vCenter and ESXi hosts to 6.0, we had some vms remain backing up fine, as though nothing had changed. Others would produce CBT issue. Of course the vms with issue have large disks and take a long time to backup.


On this site at the moment the VMs affected are not the largest, one of them is actually the smallest VM on the platform.

But having your biggest VMs affected would align with the VMware KB
Senior Engineer
VMCE
Various other stuff
Personal Technical Blog - www.educationalcentre.co.uk
saintdle
Veeam Vanguard
 
Posts: 65
Liked: 13 times
Joined: Tue Aug 05, 2014 1:13 pm
Full Name: Dean lewis

Re: ESXi 6.0 CBT issue

Veeam Logoby rmcmillan » Mon May 11, 2015 7:05 pm 1 person likes this post

We have been severely affected by this CBT issue I believe as well.

After updating everything to ESXi 6.0 and Veeam 8 update 2, we ran our first backup against all of our Vmware virtual servers.

The first thing we noticed was that our backups are now super slow...that wasn't the main issue however. :oops:

After a full Exchange backup finished and the snapshot merge took place, 2 of the 5 mailbox stores dismounted.

It took Microsoft to get the stores back to usable and mount. After dealing with that we ran another backup(not knowing what the root cause was) on Thursday night.

This took 4 of the mailbox stores offline, and even microsoft couldn't bring them back, too corrupted...We restored that backup and recovered the databases with eseutil /R, and all databases now mounted and mail was flowing again.

We are now in the process of migrating all user mailboxes to a new virtual machine and new mailbox databases for fear the corruption is deeper seeded. We will use Windows server backup to backup exchange until Vmware fixes this critical bug.

I have cases open with Veeam and Microsoft, but I now believe the issue lies with CBT and snapshot merge with Vmware.

We updated because of a Server 2012 reboot issue which was fixed in 6.0, but the cost was far too high.
rmcmillan
Novice
 
Posts: 7
Liked: 1 time
Joined: Mon May 11, 2015 6:52 pm
Full Name: rmcmillan

Next

Return to VMware vSphere



Who is online

Users browsing this forum: SyNtAxx and 21 guests