[KB2136854] There's a new CBT bug in ESXi 6

VMware specific discussions

Re: There's a new CBT bug in ESXi 6

Veeam Logoby willn » Mon Nov 16, 2015 2:03 pm

I am not sure if it happens every time. I have restored Exchange mailbox items as of Friday.
willn
Influencer
 
Posts: 15
Liked: never
Joined: Mon May 07, 2012 1:47 pm
Full Name: William Nelson

Re: There's a new CBT bug in ESXi 6

Veeam Logoby mdmd » Mon Nov 16, 2015 2:38 pm

We run a reverse incremental based backup scheme for all our 100+ VMs and use 6.0u1

I had to do a restore of our SharePoint front end a couple of weeks ago, all went fine. Just ran a chkdsk on that VM, and all reported fine. VM was 100+ GB.

We have also done various file level and exchange mailbox restores without a hitch too.
mdmd
Enthusiast
 
Posts: 32
Liked: 1 time
Joined: Mon Jan 06, 2014 10:29 am
Full Name: Mike

Re: There's a new CBT bug in ESXi 6

Veeam Logoby dellock6 » Mon Nov 16, 2015 4:03 pm 1 person likes this post

The problem with the bug is that is not consistent, it "may" report wrong blocks when reading CBT information, and this is why probably you (and many others) have not seen the error. The suggestion of disabling CBT at all until there's a patch from VMwre is to avoid those small chances of getting caught by the bug.
Luca Dell'Oca
EMEA Cloud Architect @ Veeam Software

@dellock6
http://www.virtualtothecore.com
vExpert 2011-2012-2013-2014-2015-2016
Veeam VMCE #1
dellock6
Veeam Software
 
Posts: 5047
Liked: 1330 times
Joined: Sun Jul 26, 2009 3:39 pm
Location: Varese, Italy
Full Name: Luca Dell'Oca

Re: There's a new CBT bug in ESXi 6

Veeam Logoby lightsout » Mon Nov 16, 2015 4:25 pm

I used this bit of PowerShell:
Code: Select all
get-vbrjob | ? {$_.JobType -eq "Backup" } | Set-VBRJobAdvancedViOptions -UseChangeTracking $false | out-null

To disable CBT on all of my jobs.

I've heard from VMware we'll see a patch for it this month.
lightsout
Expert
 
Posts: 185
Liked: 47 times
Joined: Thu Apr 10, 2014 4:13 pm

Re: There's a new CBT bug in ESXi 6

Veeam Logoby rccl_ecain » Mon Nov 16, 2015 4:52 pm

This part of the KB:

" this causes the change tracking information of I/Os that occur during snapshot consolidation to be lost."

Seems to be suggesting that the VM itself may be corrupted because of the consolidation process, whereas the incorrect data (corrupt data) is written back into the VM. I have a few VM's that are experiencing this type of issue it seems, were an Oracle DB server becomes corrupt after a backup. Has anyone else seen this? IF this shows up more, the only choice I have is to disable all backups.
Retired 'Cloud Admiral'. Might actually be on a ship.
rccl_ecain
Influencer
 
Posts: 12
Liked: 14 times
Joined: Tue Jan 06, 2015 10:46 pm
Location: Miramr, FL
Full Name: Ethan Cain

Re: There's a new CBT bug in ESXi 6

Veeam Logoby fourg » Mon Nov 16, 2015 4:57 pm 1 person likes this post

lightsout wrote:
Code: Select all
get-vbrjob | ? {$_.JobType -eq "Backup" } | Set-VBRJobAdvancedViOptions -UseChangeTracking $false | out-null

I came here for this, thank you kind sir!
fourg
Novice
 
Posts: 4
Liked: 1 time
Joined: Wed Sep 16, 2015 10:11 pm
Full Name: Brent F

Re: There's a new CBT bug in ESXi 6

Veeam Logoby ndolson » Mon Nov 16, 2015 7:13 pm

rccl_ecain wrote:This part of the KB:

" this causes the change tracking information of I/Os that occur during snapshot consolidation to be lost."

Seems to be suggesting that the VM itself may be corrupted because of the consolidation process, whereas the incorrect data (corrupt data) is written back into the VM.

That was my concern...you can't roll back into the parent VMDK what no longer exists. :?
ndolson
Influencer
 
Posts: 13
Liked: 2 times
Joined: Thu Jan 08, 2015 3:56 pm
Full Name: Neal

Re: [KB2136854] There's a new CBT bug in ESXi 6

Veeam Logoby jmerfeld » Mon Nov 16, 2015 7:15 pm

I opened a case with VMware about this issue.. Here is the response from them. I guess CBT disable until then..
VMware Support wrote:With regards to your question on when a patch is released on the issue, it was advised a tentative release should be by December 2nd week and no exact date was given.

But a workround on the issue is suggested on http://kb.vmware.com/kb/2136854

Any questions. do reply and i will be glad to help
jmerfeld
Influencer
 
Posts: 16
Liked: 1 time
Joined: Fri Apr 05, 2013 12:59 am
Full Name: James Merfeld

Re: There's a new CBT bug in ESXi 6

Veeam Logoby Gostev » Tue Nov 17, 2015 1:14 am

rccl_ecain wrote:This part of the KB:

" this causes the change tracking information of I/Os that occur during snapshot consolidation to be lost."

Seems to be suggesting that the VM itself may be corrupted because of the consolidation process, whereas the incorrect data (corrupt data) is written back into the VM.

No, you misunderstood this part. This talks about change tracking information specifically, which goes into a separate file (CTK). Your issue reminds one of the "early" vSphere 6 issues, but assuming you are fully patched to vSphere 6 U1a level, then there are no known issues matching your description.
Gostev
Veeam Software
 
Posts: 21390
Liked: 2349 times
Joined: Sun Jan 01, 2006 1:01 am
Location: Baar, Switzerland

Re: There's a new CBT bug in ESXi 6

Veeam Logoby tinto1970 » Tue Nov 17, 2015 8:07 am

lightsout wrote:I've heard from VMware we'll see a patch for it this month.

so soon?
great!!! :evil:
Alessandro Tinivelli aka Tinto
@tinto1970
tinto1970
Enthusiast
 
Posts: 69
Liked: 25 times
Joined: Thu Sep 26, 2013 8:40 am
Location: Bologna, Italy
Full Name: Alessandro Tinivelli

Re: There's a new CBT bug in ESXi 6

Veeam Logoby v.Eremin » Tue Nov 17, 2015 10:15 am 2 people like this post

lightsout wrote:I used this bit of PowerShell to disable CBT on all of my jobs.

For those of you using both backup and replication jobs the script has to be modified slightly:

Code: Select all
get-vbrjob | ? {$_.JobType -eq "Backup" -or $_.JobType -eq "Replica"} | Set-VBRJobAdvancedViOptions -UseChangeTracking $false | out-null

Thanks.
v.Eremin
Veeam Software
 
Posts: 13266
Liked: 968 times
Joined: Fri Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin

Re: [KB2136854] There's a new CBT bug in ESXi 6

Veeam Logoby cerberus » Tue Nov 17, 2015 6:24 pm

I've read this thread from start to end, and I just want to clarify on the impact of disabling CBT.

Currently we are in process of upgrading to vSphere 6.0 U1a and I noticed this CBT bug thread. Our Veeam nightly backup job is setup for "Reverse incremental" because we offload to tape and require daily tape jobs to have the full backup. This works great. We are doing a SAN based backup, all traffic goes over the fiber channel network.

Now with this CBT bug, we're currently not affected by it because we have not upgraded our production environment to vSphere 6 (only our Lab), so when we do upgrade I need to uncheck "Use changed block tracking data" and also uncheck "Enable CBT for all protected VMs automatically" under Advanced Settings for the backup to disk job?

How does Veeam backup the blocks inside my VMs with CBT disabled for a job that is Reverse Incremental? Is this now doing a full backup over the FC network?

I am trying to avoid a full backup as they are lengthy and resource intensive. At this point we're probably going to wait until VMware fixes the bug before we upgrade production with vSphere 6.

[EDIT] Bummer I can't seem to edit my own post.. i have my answer.. without CBT, Veeam has to read the entire disk and figure out what blocks are changed (hence the increase in disk I/O)... please ignore my rookie question.
cerberus
Enthusiast
 
Posts: 33
Liked: 3 times
Joined: Fri Aug 28, 2015 2:45 pm
Full Name: MD

Re: [KB2136854] There's a new CBT bug in ESXi 6

Veeam Logoby cffit » Tue Nov 17, 2015 9:07 pm

Can someone help explain what kind of impact to expect when you turn CBT off? Say for instance tonight I would normally run an incremental backup and I turn CBT off. Will that essentially take the same amount of time to backup as a Full Active backup does? Or more?

Do you need to start with a full backup job that has CBT turned off before you can do an incremental with CBT off?

I'd just like to know what kind of impact to expect and I realize it can very. Just some thoughts would be appreciated.
cffit
Expert
 
Posts: 338
Liked: 33 times
Joined: Fri Jan 20, 2012 2:36 pm
Full Name: Christensen Farms

Re: [KB2136854] There's a new CBT bug in ESXi 6

Veeam Logoby Gostev » Tue Nov 17, 2015 9:28 pm 1 person likes this post

Wow, some coincidences are simply unbelievable... believe or not, but I came to this topic with the sole purpose to share an excellent blog post on this exact topic. And what I see? Your post requesting this exact information! What are the chances?!

Well, there you go > Image-level backups can be done even without CBT

Kudos to our very own dellock6, his blog post could not be more timely.
Gostev
Veeam Software
 
Posts: 21390
Liked: 2349 times
Joined: Sun Jan 01, 2006 1:01 am
Location: Baar, Switzerland

Re: [KB2136854] There's a new CBT bug in ESXi 6

Veeam Logoby emachabert » Tue Nov 17, 2015 9:57 pm 1 person likes this post

Wow, clicking the link given by Anton, reading Luca's blog post and then clicking the link to Josh's blog -> can't believe what I've just read. WTF !
Veeamizing your IT since 2009/ Vanguard 2015,2016,2017
emachabert
Veeam Vanguard
 
Posts: 354
Liked: 163 times
Joined: Wed Nov 17, 2010 11:42 am
Location: France
Full Name: Eric Machabert

PreviousNext

Return to VMware vSphere



Who is online

Users browsing this forum: allankjaer, Bing [Bot], Google [Bot], rgreen83 and 13 guests