VMware CBT bug KB 2090639

[jftuga]

I just installed Veeam 7 along with patch 4 on a new VM last week. I started backing up about 50 Windows VMs of various sizes including ones with disks greater than 128 GB. Today, I installed the 4 DLL files mentioned in Gostev's weekly newsletter. None of these VM drives have changed sizes since the initial Veeam install (last week). Do I still need to manually reset CBT?

Thanks,
-John

[foggy]

Yes, since some of the drives could have been resized back in the past (prior to Veeam B&R installation) and might have corrupt CBT data associated with them.

[bctirado]

So will upgrading to v8 take care of the CBT issue or will we still need to do some manual work after the upgrade to still reset it?

Does a full backup reset CBT or do we need do the manual steps listed in: http://www.veeam.com/kb1113 which requires the VM to be powered down.

Any suggestions on a strategy that would take care of this in the best way possible would be great, we have hundreds of VMs and like 30+ Veeam servers so whatever we need to do will be a pretty large investment in time.

[Vitaliy S.]

Hi Brian,

This was covered in Anton's weekly digest email, please take a look:

Good news, we have managed to squeeze a workaround for VMware CBT bug into the v8 release (hotfix for v7 is coming soon). Jobs will now reset CBT on a VM upon detecting any virtual disk configuration change. Given the time constraints, results of our own testing and continuous updates to the corresponding VMware KB article, we felt this is the safest way to go. Keep in mind that this new functionality will address future virtual disk expansion cases only, so you do need to reset CBT manually on all VMs before installing v8 (unless you have already done so) according to the steps outlined in our KB1940. Another good source of alternative scripts (as well as answers to all possible related questions) is this dedicated forum thread > VMware CBT bug KB 2090639

The best strategy would be detect the list of VMs that had their virtual disk extended (if you own Veeam ONE/MP you can do that automatically), reset the CBT and run jobs as you normally do.

Thanks!

[bctirado]

Thanks Vitaliy,

Three other questions that came up that I haven't seen scanning through this thread.

1. In order to run the powershell commands do we need to have the powershell option for veeam installed or does it rely on something like PowerCLI installed?
2. Does the bug impact v 6.1.0.204 we have a handful of those out there still and just wanted to double-check.
3. Are we safe to install v8 and then run the powershell script or do we need to run the powersell script before installing v8?

Thanks!
Brian

[KiwiJJ]

Hi,
In Anton's latest newsletter he says

"Hopefully, final update on VMware CBT data corruption issue (Veeam KB1940). We now have a hot fix for B&R 7.0 Patch #4 available for download. This will make all jobs reset VM CBT automatically upon detecting a change in virtual disk configuration (B&R v8 does the same out of box)."

I do not use CBT so does this mean that when I make a change to the virtual disk configuration that it will turn it on ?

Thanks,

John

[lightsout]

All,

So on Gostev's digest for the week, we've got a hotfix for VB&R v7 available for the CBT corruption available here:

ftp://ftp334455:xWidbjXe@supportftp.veeam.com/

In this FTP directory, I've just got 4 DLLs. I'm assuming we should stop Veeam's services, and replace these files by hand?

Is there a plan to do an installer for this? Or is the advice to look towards upgrading to v8 which has it embedded anyway?

Thanks!

[Gostev]

1. In order to run the powershell commands do we need to have the powershell option for veeam installed or does it rely on something like PowerCLI installed?
2. Does the bug impact v 6.1.0.204 we have a handful of those out there still and just wanted to double-check.
3. Are we safe to install v8 and then run the powershell script or do we need to run the powersell script before installing v8?

1. No, KB1940 script is pure PowerCLI script.
2. The bug impacts every B&R version since 4.0 (which is when we have introduced CBT support). But 6.1 is not supported for over a year now anyway.
3. Does not matter as long as you can make sure virtual disks are not being extended between running the script and installing v8 (or hot fix for v7)

I do not use CBT so does this mean that when I make a change to the virtual disk configuration that it will turn it on ?

No, it will turn it off (in case it is enabled on a VM). Turning CBT on is controlled with the existing UI setting that we had since 4.0, and I assume you have it unchecked if you say you are not using CBT.

[foggy]

In this FTP directory, I've just got 4 DLLs. I'm assuming we should stop Veeam's services, and replace these files by hand?

Yes, and this is exactly what the mentioned KB states.

Is there a plan to do an installer for this? Or is the advice to look towards upgrading to v8 which has it embedded anyway?

No plans to do an installer. You can either replace DLL's in your v7 installation or upgrade to v8.

[bctirado]

Perfect, thanks for the info!

[tom11011]

How do I run these powershell scripts?

When I run mine, I get the message "the term get-vm is not recognized as the name of a cmdlet......."

Do I need to run this in a certain directory on the veeam server?

[veremin]

Do you have PowerCLI installed in your environment? Have you added it to your PS session? Thanks.

[tom11011]

I installed bps_64x per the veeam instructions then opened a powershell through the veeam software.

[veremin]

bps_64x refers to our PowerShell snap-in, not VMware one. The latter can be downloaded from VMware site. Once you install PowerCLI successfully, you should then add it to your PS session, using the following line:

Code: Select all

Add-PSSnapin VMware.VimAutomation.Core

Thanks.

[tom11011]

Thanks.

Before I run the script, I wanted to check something first.

I installed the 4 hotfix dll files last night. Backups kicked off and completed, although they took a really long time and each virtual machine finished with this warning in the log

Cannot use CBT: Soap fault. Error caused by file /vmfs/volumes/500dbdfe-27eae1d3-9a82-001b2134d3ac/TSTSPTTPAV01/TSTSPTTPAV01.vmdkDetail: '', endpoint: ''

Does that mean cbt has already been reset now or should I still run the script? I was under the impression that simply installing these files was not enough?

[foggy]

Seems that something else caused CBT to fail and full VM data was read. Simply installing the hotfix was not intended to cause this, you were supposed to reset CBT manually for all VMs that may have been impacted. If each and every VM was affected by the said behavior, then there's no need to run the script anymore.

[tom11011]

Couple of extra questions.

1.) Are all vm's that are less than 128gigs considered safe?

2.) Does this procedure need to be ran for both backups and replications? If I have a vm that is only replicated and not backed up, do I still run the script? What about if the vm is both backed up and replicated, do I run the script only one time?

3.) What about a vm that has 2 different vm drives? Say for example hard drive 1 is 70gigs and hard drive 2 is 80gigs. Is this vm considered safe?

Thanks.

[foggy]

Couple of extra questions.

The answer is "Yes" to all of these questions.

[tom11011]

Foggy you are ok, I don't care what everyone else says

[ldv]

Hi! Another question about the script - Can I run only the step "Apply the specification to each VM" without the snapshot creation/removal? The snapshot creation/removal will be done with the next veeam backup job anyway.

Thanks.

[foggy]

Laura, to my understanding, this will not work. The fact is that for the CBT setting to take effect, you need to power off the VM or take a snapshot. But Veeam B&R will enable CBT back on the next job run prior to taking the snapshot, so the setting will not be applied. You need to run the entire script.

[helpdeskuk]

Good Afternoon,

I have applied the latest .dll hot fix to prevent new VMs from being affected by the CBT VMWare issue.

I am in the process of resetting the CBT of my existing VMWare VMs to ensure a valid backup and would like a little clarification.

I am following the Veeam KB article 1113 http://www.veeam.com/kb1113.
Step 3 says to "Set the "ctkEnabled" value to false" it then goes on to sat for step 4 "Set the "scsi0:x.ctkEnabled" value to false for each disk of the VM in question"

After removing the -CTK.VMDK files and powering on and off step 7 says to "Set the "scsi0:x.ctkEnabled" value back to true for each disk of the VM in question" but it makes no mention of the "ctkenabled" in step 3. Should this be left as false or returned to true along with "scsi0:x.ctkEnabled" ?

Many Thanks

[Guido]

I ran the script for one VM. It only takes a few seconds to complete. No errors. CBT is off afterwards.
But there are no messages to be found in the Tasks&events logging within VIclient for snapshotting and removing the snapshot. Is this normal?

[Vitaliy S.]

Hi David,

You should enable it as well, good catch. Also please be aware that you can leave all settings as false and Veeam will re-enable CTK for all VMs automatically (given that you have a corresponding option enabled).

P.S. btw, in this topic you can find a script examples that would reset VM CBT data for you > VMware CBT bug KB 209063911

Thanks!

[helpdeskuk]

Many thanks!

[joergr]

@helpdeskuk: I would highly suggest to use this script - it can reset cbt automatically on all of your vms WITHOUT the need for shutting anything down. You can use it while the vms are running.

[foggy]

I ran the script for one VM. It only takes a few seconds to complete. No errors. CBT is off afterwards.
But there are no messages to be found in the Tasks&events logging within VIclient for snapshotting and removing the snapshot. Is this normal?

Looks suspicious. I would check VMware logs for snapshot creation/deletion operations and/or contact tech support to check whether you're doing everything correctly.

[Guido]

If VIclient connected directly to the vsphere server the snapshotting is logged. If connected to the vcenter there is no logging. Thank you.

[foggy]

If VIclient connected directly to the vsphere server the snapshotting is logged. If connected to the vcenter there is no logging. Thank you.

That means you're aiming the script to the host directly, not via vCenter.

[r.mckeon]

Hi, so I just found out about this bug/nightmare. This might be a stupid question but I need some peace of mind.

I recently expanded an ESXi datastore from 100GB to 200GB. I repeat, I only expanded the DATASTORE not the actual vmdk. This means I'm completely safe right?

But, if I find out that one of my vmdks was actually expanded, or I want to go through with a fix for above-mentioned VM anyway, do these overall steps make sense? Also, will the below steps solve for both backups and replica jobs?

• 1. Reset the CBT on the particular VM using the steps in the Solution section here http://www.veeam.com/kb1113
  2. Update the Veeam B&R software for any future expansions using this KB http://www.veeam.com/kb1940