A CBT Story

[ejleipold]

Hi Guys,

I'm always asking questions on here, so I thought today I'd give back and tell you a tale of what happened in my production environment recently.
So we had a power loss at our main office on a Friday arvo last week, not unusual for our area, not a problem as we have a backup generator that kicks in automatically within 10 seconds to take over the load.
Unfortunately, it appears that some muppet had earlier, hit the emergency stop button on the generator which knocked it out of automatic mode, so when the power went out, the UPS's lasted 20 minutes then everything went dark.
My whole production environment got shutdown dirty. Ew.

So I raced into my office when I saw the UPS notifications, fixed the generator and got everything powered back up.
I then spent the next 45 minutes slowly bringing all my servers online and praying that nothing had corrupted.
Luckily, everything appeared to have surprised the dirty shutdown!

That night though when I ran my Veeam backups, I got warnings about the CBT (Changed block tracking) in the form of "Cannot use CBT: Soap fault. Error caused by file "
The backups still completed, but it was pretty damn slow.

So I did some digging and it appears that the CBT somehow got corrupted from the dirty shutdown

Luckily I was able to find this KB article (http://www.veeam.com/KB1113) which outlines how to reset CBT.

Which basically involves shutting down each vm (oh yay) disabling CBT on the VM, removing the old CBT files from the vmfs then rebooting the VM a couple of times.

Yay fixed didnt need that weekend anyway

Morale of the story, never underestimate the power of human stupidity.

[J1mbo]

I guess there's a trade off - update the CBT first, and lose the data block. Or the opposite, and lose the CBT stuff which we might say has less value. Maybe CBT is purposely invalidated on a dirty shutdown, for that reason.

[ejleipold]

Probably, either way it doesn't seem to have broken anything too bad, at least as far as I can tell, for now. :/ Oh, we're getting a break glass panel put over that stop button now, so hopefully its muppet proof and I wont have to go through this debacle again >_<

[Gostev]

Actually, resetting CBT manually on a VM is rarely needed. Your following job run would have used CBT anyway - even without performing CBT reset. That first (slow) job run establishes new consistent CBT point automatically. This run is slow because it performs "snap&scan" incremental (physically reads every block, and compares it to its previously state).

[ejleipold]

Hey Gostev,

Nope, it warns every time I run the job now, until I reset the CBT on that VM :/

[Gostev]

OK, then this is one of those rare cases when you do indeed have to go through this pain

[ejleipold]

I have an amazing talent of finding the exceptions I'm actually in the process of rebuilding my whole vCentre anyway now as I tried to fix a small issue and broke it worse Not having a good week.

[Gostev]

Let the Veeam Backup be with you

[juz_far]

I just wanted to congratulate you on being able to use the description 'muppet' in the forums. It's not used often enough..

[chimera]

OK, then this is one of those rare cases when you do indeed have to go through this pain

Been through that pain as well. In our case, we lost iSCSI connections from 1 x host to a new SAN that we'd moved a couple VM's too, the host became completely unresponsive, had to manually power off... anyways to cut a long story short, the 2 x VM's (1 of which was critical) needed CBT resetting as per that article. So similar symptoms to what you would have experienced - corrupt ctk file no doubt. After resetting, they backed up flawlessly again. Sigh of relief

[boromic]

I just went through the exact same CBT issue that you are describing, but the cause was different. Go back and check your VMs for snapshots created by Veeam that were left there when you lost power to your environment. If you have jobs currently running, be sure that the VMs with snapshots are currently being processed by Veam...if they aren't, remove the snapsphots. I had a vCenter server problem that lasted for several weeks were the vCenter service was crashing leaving Veeam job to fail because it could no longer talk to it. I was also getting extremely slow performance on my remote backup job after upgrading to 6.1 (hopefully the new hotfix will address this) and I was having to kill backup jobs processing my VM with large vmdks...when vCenter isn't running, Veeam can't tell the hosts to remove the snapshots from the VMs. The VMs with orphaned snapshot reported a CBT error the first backup (or when an additional snapshot was left) and then it went away. My exchange server had 3 nested Veeam snapshots, and post VM processing snapshot removal was taking almost an hour. My two large VMs were adding almost 2.5 hours to processing because of slow snapshot removal. I believe you are also going to get a CBT error when you remove the orphaned snapshots.

[nickgc]

I had a power outage over the weekend and was not able to shut down our VMs cleanly. Everything powered up ok, but like the first thread of this post I got Veeam warnings regarding CBT:

Code: Select all

Cannot use CBT: Soap fault. Error caused by file /vmfs/volumes/... ... .vmdkDetail: '', endpoint: ''

Given that it was just a warning, the information in this post, plus the fact that Veeam claimed to have finished the job successfully my inclination was to let it run one more time. After the second set of backups the warnings are no longer there and all jobs reporting success.

[Gostev]

This is exactly how it should normally work.

[mykl_74]

I have come across this CBT problem as well, but for yet another different reason. We recently purchased a new ibm v3700 SAN. I migrated all my VMs to the new device, by shutting down the VM, and using the migrate option in vSphere to move the VMDKs. (I don't have storage vmotion ) Now I am running backups with Veeam for the first time against the new san, and almost every single one of my VMs is complaining of CBT errors. Is this expected when moving to new storage?

[Gostev]

Unexpected for me as well...

[pcrebe]

Hi,
I've migrated all VMs from an Dell MD3000i (iSCSI) to a NetApp FAS2220 (NFS) from one week and every replica or backup cycle show wornings for CBT error.

Any touning to do?

Thanks

[foggy]

Carlo, what Veeam B&R version are you using and how did you migrate VMs to the new storage? Anyway, the next jobs run should complete without CBT warnings.

[pcrebe]

Hi foggy,
B&R v7 and vSphere 4.1 Ess. plus. I've migrated the VMs with a cold migration task.
The CBT warning is on alternate days. The first day the job has the worning, the day after no and the thirtd day it has the warning again and so on.

Thanks

[Vitaliy S.]

So to make sure I got it right, you have these warning messages on the first, third and fifth job passes? Do you see it for all VMs?

[pcrebe]

Yes you've understand not all vm
day 1,3,5 vm A,B,C
day 2,4,6 vm D,E,F

Today is the first day that all vm are backed up with CBT so no wornings. I've sopped Veeam ONE service that is on the same B&R server and parallel jobs. Tomorrow i'll restart the service to see what happen.

Thanks

[veremin]

Actually, neither Veeam One service nor parallel job processing should affect the Veeam ability to use CBT. So, kindly, observe your environment for several days and see whether the CBT warning shows up again or not.

If it does, the best option will be to open a ticket with our support team and let them investigate it directly.

Thanks.

[pcrebe]

When i've time i open a support case.

All VM's on NFS NetApp storage have alternative warnings instead all VM's on iSCSI Dell storage are ok.

Thanks

[pcrebe]

case number is 00534156.