[KB2136854] There's a new CBT bug in ESXi 6

VMware specific discussions

Re: [KB2136854] There's a new CBT bug in ESXi 6

Veeam Logoby cffit » Tue Nov 17, 2015 10:34 pm

I assume that changing from CBT to non-CBT backups you aren't really covered until you have done a full backup without CBT enabled correct? Having a full backup that was done with CBT enabled and then incrementals without CBT enabled doesn't eliminate the risk given the full backup is always going to impact every incremental after it?

And if that's the case, how does this affect backup copies since they never really do a full backup after the first backup?
cffit
Expert
 
Posts: 338
Liked: 33 times
Joined: Fri Jan 20, 2012 2:36 pm
Full Name: Christensen Farms

Re: [KB2136854] There's a new CBT bug in ESXi 6

Veeam Logoby Gostev » Tue Nov 17, 2015 11:36 pm

Your assumption is incorrect, moreover I have already answered this question on the second page ;)

Anyway, here is the extended version of my answer from this week's forum digest for your convenience:
Gostev wrote:Disabling CBT is sufficient to both prevent and remediate the issue, because this will make jobs physically compare latest state of disk in backup or replica with its actual state, and transfer any non-matching blocks over as a part of incremental backup (along with actually changed blocks), thus fixing any corruption that may already be in place.
Gostev
Veeam Software
 
Posts: 21390
Liked: 2349 times
Joined: Sun Jan 01, 2006 1:01 am
Location: Baar, Switzerland

Re: [KB2136854] There's a new CBT bug in ESXi 6

Veeam Logoby alex1002 » Wed Nov 18, 2015 3:20 am

Today I tried veeam in my lab and tried to restart it with cbt off. The cpu usage on the esxi servers went so high causing VMware to halt vms. The Alarms were saying high
Cpu usage. Everything make to a complete halt while backups were running. Is this normal behaviour?
alex1002
Enthusiast
 
Posts: 25
Liked: 1 time
Joined: Tue Jan 27, 2015 6:17 pm
Full Name: Alex

Re: [KB2136854] There's a new CBT bug in ESXi 6

Veeam Logoby Gostev » Wed Nov 18, 2015 1:19 pm 1 person likes this post

No, this is not normal, as calculating hashes requires little to no CPU resources. And this is the only extra operation that proxy does when CBT is disabled, comparing to incremental runs with CBT enabled.
Gostev
Veeam Software
 
Posts: 21390
Liked: 2349 times
Joined: Sun Jan 01, 2006 1:01 am
Location: Baar, Switzerland

Re: [KB2136854] There's a new CBT bug in ESXi 6

Veeam Logoby cffit » Wed Nov 18, 2015 7:24 pm

If I wanted to check our backups by booting up a SureBackup job and then running chkdsk, do I need to run the full chkdsk (which will take forever) or for this purpose can a more limited chkdsk scan be done with some of the command switches?

Thanks!
cffit
Expert
 
Posts: 338
Liked: 33 times
Joined: Fri Jan 20, 2012 2:36 pm
Full Name: Christensen Farms

Re: [KB2136854] There's a new CBT bug in ESXi 6

Veeam Logoby Gostev » Wed Nov 18, 2015 7:44 pm

I don't believe you need a full scan, because it is designed for finding physical disk issues (specifically, bad sectors). Regular CHKDSK run, which checks if actual file allocation on disk matches NTFS MFT expectations, is usually enough to pick up major inconsistencies and corruptions. Thanks!
Gostev
Veeam Software
 
Posts: 21390
Liked: 2349 times
Joined: Sun Jan 01, 2006 1:01 am
Location: Baar, Switzerland

Re: [KB2136854] There's a new CBT bug in ESXi 6

Veeam Logoby alex1002 » Thu Nov 19, 2015 6:59 pm

Gostev wrote:No, this is not normal, as calculating hashes requires little to no CPU resources. And this is the only extra operation that proxy does when CBT is disabled, comparing to incremental runs with CBT enabled.

What do you suggest?
alex1002
Enthusiast
 
Posts: 25
Liked: 1 time
Joined: Tue Jan 27, 2015 6:17 pm
Full Name: Alex

Re: [KB2136854] There's a new CBT bug in ESXi 6

Veeam Logoby Gostev » Thu Nov 19, 2015 7:14 pm 3 people like this post

To open a support case for troubleshooting, of course :)
Gostev
Veeam Software
 
Posts: 21390
Liked: 2349 times
Joined: Sun Jan 01, 2006 1:01 am
Location: Baar, Switzerland

Re: [KB2136854] There's a new CBT bug in ESXi 6

Veeam Logoby BLWL » Mon Nov 23, 2015 10:57 am

For you that disabled CBT, how have that affected your source storage?

We run a NetApp MetroCluster storage on two sites. Veeam using storage snapshots. Before I disabled CBT, we could see some 100% max disk utilization during backups for shorter periods. No complaints though from users.

After disabling CBT, we now see much longer periods, 2-6 hours of 100% disk util. Both sites affected. Some VM's having problems. Moving jobs have helped a bit, but I still need to adjust in Veeam I belive.

Would decrease of concurrent tasks be the most suitable thing to adjust?


Thanks,
/BLWL
BLWL
Influencer
 
Posts: 15
Liked: 3 times
Joined: Tue Jan 27, 2015 7:24 am
Full Name: Bjorn L

Re: [KB2136854] There's a new CBT bug in ESXi 6

Veeam Logoby foggy » Mon Nov 23, 2015 12:52 pm

Yes, you can play with reducing concurrent tasks limit to decrease the load on primary storage.
But a far better instrument to control storage load is our Backup I/O Control functionality.
foggy
Veeam Software
 
Posts: 14728
Liked: 1078 times
Joined: Mon Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson

Re: [KB2136854] There's a new CBT bug in ESXi 6

Veeam Logoby sphilp » Wed Nov 25, 2015 4:17 pm 3 people like this post

VMware appears to have issued a patch for this, it came through Update Manager an hour or so ago for us.

Patch ID: ESXi600-201511401-BG
KB Article: http://kb.vmware.com/kb/2137546

I'm curious about what the "recovery" process for our backups and replication should be. Is the suggestion that we run an Active Full against each job to get a "known good" set of backups after the VMware patch has been applied?
sphilp
Enthusiast
 
Posts: 36
Liked: 9 times
Joined: Thu May 28, 2009 7:52 pm
Full Name: Steve Philp

Re: [KB2136854] There's a new CBT bug in ESXi 6

Veeam Logoby sthorpe » Wed Nov 25, 2015 4:46 pm

Even though VMware have posted a patch, our backups would still be 'inconsistent' (synthetic full) as if we were hit by the bug then we wouldn't know we have have missed any blocks.
We will still need to perform a full backup - Does this reset the CBT at that point?

FYI turning off CBT on one of our 'real life servers' increased the backup time from 13mins to 3hours 7mins it also meant rather than reading 40Gb of 'changed data' it read the whole 2.1Tb disk increasing I/O on the backend SAN. Turn off CBT in the real world is not a workable solution.

Steve
sthorpe
Lurker
 
Posts: 1
Liked: never
Joined: Wed Nov 25, 2015 4:33 pm
Full Name: Steve Thorpe

Re: [KB2136854] There's a new CBT bug in ESXi 6

Veeam Logoby ar2015 » Wed Nov 25, 2015 5:35 pm

Hi,

Will CBT need to be reset to continue with good backups once the patch has been applied?? Gostevs post "Veeam Logoby Gostev » Sat Nov 14, 2015 4:41 pm" suggests there may be an issue with full backups as they utilise CBT, therefore if there are potentially missing blocks in existing CBT will a new Full Backup be susceptible to the same issue. Alternatively does CBT get reset at the point a full is made ??
ar2015
Lurker
 
Posts: 1
Liked: never
Joined: Wed Nov 25, 2015 5:23 pm
Full Name: Adam Richards

Re: [KB2136854] There's a new CBT bug in ESXi 6

Veeam Logoby cerberus » Wed Nov 25, 2015 5:49 pm

sphilp wrote:VMware appears to have issued a patch for this, it came through Update Manager an hour or so ago for us.

Patch ID: ESXi600-201511401-BG
KB Article: http://kb.vmware.com/kb/2137546

I'm curious about what the "recovery" process for our backups and replication should be. Is the suggestion that we run an Active Full against each job to get a "known good" set of backups after the VMware patch has been applied?


I see the patch now too, wohoo :)

Off-topic but does anyone on here know if using a non-VMware ESXi 6.0 image (in my case, I am using Dell-ESXi-6.0U1-3073146-A01 for my ESXi hosts), can we still apply this patch release by VMware or do we have to wait for Dell?
cerberus
Enthusiast
 
Posts: 32
Liked: 3 times
Joined: Fri Aug 28, 2015 2:45 pm
Full Name: MD

Re: [KB2136854] There's a new CBT bug in ESXi 6

Veeam Logoby neuvoja » Wed Nov 25, 2015 5:54 pm

You do not have to wait for Dell, they usually only update the custom iso with major updates. Just use VUM to patch it to your hosts.
neuvoja
Service Provider
 
Posts: 1
Liked: never
Joined: Mon Nov 16, 2015 5:04 pm
Location: Finland
Full Name: Jani Neuvonen

PreviousNext

Return to VMware vSphere



Who is online

Users browsing this forum: No registered users and 11 guests