Discussions specific to the VMware vSphere hypervisor
Locked
jftuga
Enthusiast
Posts: 26
Liked: 2 times
Joined: Oct 22, 2010 7:02 pm
Full Name: John Taylor
Location: Athens, GA
Contact:

[MERGED] New Veeam server and recent CBT patch

Post by jftuga » Nov 17, 2014 3:44 pm

I just installed Veeam 7 along with patch 4 on a new VM last week. I started backing up about 50 Windows VMs of various sizes including ones with disks greater than 128 GB. Today, I installed the 4 DLL files mentioned in Gostev's weekly newsletter. None of these VM drives have changed sizes since the initial Veeam install (last week). Do I still need to manually reset CBT?

Thanks,
-John

foggy
Veeam Software
Posts: 18457
Liked: 1589 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: New Veeam server and recent CBT patch

Post by foggy » Nov 17, 2014 4:37 pm 1 person likes this post

Yes, since some of the drives could have been resized back in the past (prior to Veeam B&R installation) and might have corrupt CBT data associated with them.

bctirado
Influencer
Posts: 20
Liked: 1 time
Joined: Oct 27, 2009 9:24 pm
Full Name: Brian
Contact:

[MERGED] Upgrade to v8 and the CBT Issue

Post by bctirado » Nov 17, 2014 6:15 pm

So will upgrading to v8 take care of the CBT issue or will we still need to do some manual work after the upgrade to still reset it?

Does a full backup reset CBT or do we need do the manual steps listed in: http://www.veeam.com/kb1113 which requires the VM to be powered down.

Any suggestions on a strategy that would take care of this in the best way possible would be great, we have hundreds of VMs and like 30+ Veeam servers so whatever we need to do will be a pretty large investment in time.

Vitaliy S.
Product Manager
Posts: 23222
Liked: 1609 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: VMware CBT bug KB 2090639

Post by Vitaliy S. » Nov 17, 2014 6:59 pm

Hi Brian,

This was covered in Anton's weekly digest email, please take a look:
Gostev wrote:Good news, we have managed to squeeze a workaround for VMware CBT bug into the v8 release (hotfix for v7 is coming soon). Jobs will now reset CBT on a VM upon detecting any virtual disk configuration change. Given the time constraints, results of our own testing and continuous updates to the corresponding VMware KB article, we felt this is the safest way to go. Keep in mind that this new functionality will address future virtual disk expansion cases only, so you do need to reset CBT manually on all VMs before installing v8 (unless you have already done so) according to the steps outlined in our KB1940. Another good source of alternative scripts (as well as answers to all possible related questions) is this dedicated forum thread > VMware CBT bug KB 2090639
The best strategy would be detect the list of VMs that had their virtual disk extended (if you own Veeam ONE/MP you can do that automatically), reset the CBT and run jobs as you normally do.

Thanks!

bctirado
Influencer
Posts: 20
Liked: 1 time
Joined: Oct 27, 2009 9:24 pm
Full Name: Brian
Contact:

Re: VMware CBT bug KB 2090639

Post by bctirado » Nov 17, 2014 8:14 pm

Thanks Vitaliy,

Three other questions that came up that I haven't seen scanning through this thread.

1. In order to run the powershell commands do we need to have the powershell option for veeam installed or does it rely on something like PowerCLI installed?
2. Does the bug impact v 6.1.0.204 we have a handful of those out there still and just wanted to double-check.
3. Are we safe to install v8 and then run the powershell script or do we need to run the powersell script before installing v8?

Thanks!
Brian

KiwiJJ
Expert
Posts: 105
Liked: 2 times
Joined: Feb 16, 2010 8:05 pm
Full Name: John Jones
Location: New Zealand
Contact:

[MERGED] VMware CBT issue

Post by KiwiJJ » Nov 17, 2014 8:16 pm

Hi,
In Anton's latest newsletter he says

"Hopefully, final update on VMware CBT data corruption issue (Veeam KB1940). We now have a hot fix for B&R 7.0 Patch #4 available for download. This will make all jobs reset VM CBT automatically upon detecting a change in virtual disk configuration (B&R v8 does the same out of box)."

I do not use CBT so does this mean that when I make a change to the virtual disk configuration that it will turn it on ?

Thanks,

John

lightsout
Expert
Posts: 218
Liked: 58 times
Joined: Apr 10, 2014 4:13 pm
Contact:

[MERGED] Veeam v7 Hotfix for KB1940

Post by lightsout » Nov 17, 2014 8:53 pm

All,

So on Gostev's digest for the week, we've got a hotfix for VB&R v7 available for the CBT corruption available here:

ftp://ftp334455:xWidbjXe@supportftp.veeam.com/

In this FTP directory, I've just got 4 DLLs. I'm assuming we should stop Veeam's services, and replace these files by hand?

Is there a plan to do an installer for this? Or is the advice to look towards upgrading to v8 which has it embedded anyway?

Thanks!

Gostev
SVP, Product Management
Posts: 25135
Liked: 3688 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: VMware CBT bug KB 2090639

Post by Gostev » Nov 17, 2014 8:59 pm

bctirado wrote:1. In order to run the powershell commands do we need to have the powershell option for veeam installed or does it rely on something like PowerCLI installed?
2. Does the bug impact v 6.1.0.204 we have a handful of those out there still and just wanted to double-check.
3. Are we safe to install v8 and then run the powershell script or do we need to run the powersell script before installing v8?
1. No, KB1940 script is pure PowerCLI script.
2. The bug impacts every B&R version since 4.0 (which is when we have introduced CBT support). But 6.1 is not supported for over a year now anyway.
3. Does not matter as long as you can make sure virtual disks are not being extended between running the script and installing v8 (or hot fix for v7)
KiwiJJ wrote:I do not use CBT so does this mean that when I make a change to the virtual disk configuration that it will turn it on ?
No, it will turn it off (in case it is enabled on a VM). Turning CBT on is controlled with the existing UI setting that we had since 4.0, and I assume you have it unchecked if you say you are not using CBT.

foggy
Veeam Software
Posts: 18457
Liked: 1589 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Veeam v7 Hotfix for KB1940

Post by foggy » Nov 17, 2014 9:09 pm

lightsout wrote:In this FTP directory, I've just got 4 DLLs. I'm assuming we should stop Veeam's services, and replace these files by hand?
Yes, and this is exactly what the mentioned KB states.
lightsout wrote:Is there a plan to do an installer for this? Or is the advice to look towards upgrading to v8 which has it embedded anyway?
No plans to do an installer. You can either replace DLL's in your v7 installation or upgrade to v8.

bctirado
Influencer
Posts: 20
Liked: 1 time
Joined: Oct 27, 2009 9:24 pm
Full Name: Brian
Contact:

Re: VMware CBT bug KB 2090639

Post by bctirado » Nov 17, 2014 9:12 pm

Perfect, thanks for the info!

tom11011
Expert
Posts: 144
Liked: 2 times
Joined: Dec 01, 2010 8:40 pm
Full Name: Tom
Contact:

Re: VMware CBT bug KB 2090639

Post by tom11011 » Nov 19, 2014 2:58 am

How do I run these powershell scripts?

When I run mine, I get the message "the term get-vm is not recognized as the name of a cmdlet......."

Do I need to run this in a certain directory on the veeam server?

veremin
Product Manager
Posts: 17070
Liked: 1475 times
Joined: Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin
Contact:

Re: VMware CBT bug KB 2090639

Post by veremin » Nov 19, 2014 9:49 am

Do you have PowerCLI installed in your environment? Have you added it to your PS session? Thanks.

tom11011
Expert
Posts: 144
Liked: 2 times
Joined: Dec 01, 2010 8:40 pm
Full Name: Tom
Contact:

Re: VMware CBT bug KB 2090639

Post by tom11011 » Nov 19, 2014 12:43 pm

I installed bps_64x per the veeam instructions then opened a powershell through the veeam software.

veremin
Product Manager
Posts: 17070
Liked: 1475 times
Joined: Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin
Contact:

Re: VMware CBT bug KB 2090639

Post by veremin » Nov 19, 2014 12:46 pm

bps_64x refers to our PowerShell snap-in, not VMware one. The latter can be downloaded from VMware site. Once you install PowerCLI successfully, you should then add it to your PS session, using the following line:

Code: Select all

Add-PSSnapin VMware.VimAutomation.Core
Thanks.

tom11011
Expert
Posts: 144
Liked: 2 times
Joined: Dec 01, 2010 8:40 pm
Full Name: Tom
Contact:

Re: VMware CBT bug KB 2090639

Post by tom11011 » Nov 19, 2014 3:01 pm

Thanks.

Before I run the script, I wanted to check something first.

I installed the 4 hotfix dll files last night. Backups kicked off and completed, although they took a really long time and each virtual machine finished with this warning in the log

Cannot use CBT: Soap fault. Error caused by file /vmfs/volumes/500dbdfe-27eae1d3-9a82-001b2134d3ac/TSTSPTTPAV01/TSTSPTTPAV01.vmdkDetail: '', endpoint: ''

Does that mean cbt has already been reset now or should I still run the script? I was under the impression that simply installing these files was not enough?

foggy
Veeam Software
Posts: 18457
Liked: 1589 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: VMware CBT bug KB 2090639

Post by foggy » Nov 19, 2014 3:45 pm

Seems that something else caused CBT to fail and full VM data was read. Simply installing the hotfix was not intended to cause this, you were supposed to reset CBT manually for all VMs that may have been impacted. If each and every VM was affected by the said behavior, then there's no need to run the script anymore.

tom11011
Expert
Posts: 144
Liked: 2 times
Joined: Dec 01, 2010 8:40 pm
Full Name: Tom
Contact:

Re: VMware CBT bug KB 2090639

Post by tom11011 » Nov 19, 2014 4:18 pm

Couple of extra questions.

1.) Are all vm's that are less than 128gigs considered safe?

2.) Does this procedure need to be ran for both backups and replications? If I have a vm that is only replicated and not backed up, do I still run the script? What about if the vm is both backed up and replicated, do I run the script only one time?

3.) What about a vm that has 2 different vm drives? Say for example hard drive 1 is 70gigs and hard drive 2 is 80gigs. Is this vm considered safe?

Thanks.

foggy
Veeam Software
Posts: 18457
Liked: 1589 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: VMware CBT bug KB 2090639

Post by foggy » Nov 19, 2014 4:30 pm

tom11011 wrote:Couple of extra questions.
The answer is "Yes" to all of these questions.

tom11011
Expert
Posts: 144
Liked: 2 times
Joined: Dec 01, 2010 8:40 pm
Full Name: Tom
Contact:

Re: VMware CBT bug KB 2090639

Post by tom11011 » Nov 19, 2014 5:08 pm

Foggy you are ok, I don't care what everyone else says :)

ldv
Novice
Posts: 9
Liked: 1 time
Joined: Sep 25, 2013 6:43 am
Full Name: Laura
Contact:

Re: VMware CBT bug KB 2090639

Post by ldv » Nov 20, 2014 11:28 am

Hi! Another question about the script - Can I run only the step "Apply the specification to each VM" without the snapshot creation/removal? The snapshot creation/removal will be done with the next veeam backup job anyway.

Thanks.

foggy
Veeam Software
Posts: 18457
Liked: 1589 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: VMware CBT bug KB 2090639

Post by foggy » Nov 20, 2014 3:22 pm

Laura, to my understanding, this will not work. The fact is that for the CBT setting to take effect, you need to power off the VM or take a snapshot. But Veeam B&R will enable CBT back on the next job run prior to taking the snapshot, so the setting will not be applied. You need to run the entire script.

helpdeskuk
Novice
Posts: 7
Liked: 1 time
Joined: Nov 14, 2011 4:39 pm
Full Name: David Thomas
Contact:

[MERGED] CBT Reset KB article clarification

Post by helpdeskuk » Nov 21, 2014 2:36 pm

Good Afternoon,

I have applied the latest .dll hot fix to prevent new VMs from being affected by the CBT VMWare issue.

I am in the process of resetting the CBT of my existing VMWare VMs to ensure a valid backup and would like a little clarification.

I am following the Veeam KB article 1113 http://www.veeam.com/kb1113.
Step 3 says to "Set the "ctkEnabled" value to false" it then goes on to sat for step 4 "Set the "scsi0:x.ctkEnabled" value to false for each disk of the VM in question"

After removing the -CTK.VMDK files and powering on and off step 7 says to "Set the "scsi0:x.ctkEnabled" value back to true for each disk of the VM in question" but it makes no mention of the "ctkenabled" in step 3. Should this be left as false or returned to true along with "scsi0:x.ctkEnabled" ?

Many Thanks

Guido
Influencer
Posts: 24
Liked: never
Joined: Jul 17, 2012 10:56 am
Full Name: Guido
Contact:

Re: VMware CBT bug KB 2090639

Post by Guido » Nov 21, 2014 2:37 pm

I ran the script for one VM. It only takes a few seconds to complete. No errors. CBT is off afterwards.
But there are no messages to be found in the Tasks&events logging within VIclient for snapshotting and removing the snapshot. Is this normal?

Vitaliy S.
Product Manager
Posts: 23222
Liked: 1609 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: CBT Reset KB article clarification

Post by Vitaliy S. » Nov 21, 2014 3:01 pm

Hi David,

You should enable it as well, good catch. Also please be aware that you can leave all settings as false and Veeam will re-enable CTK for all VMs automatically (given that you have a corresponding option enabled).

P.S. btw, in this topic you can find a script examples that would reset VM CBT data for you > VMware CBT bug KB 209063911

Thanks!

helpdeskuk
Novice
Posts: 7
Liked: 1 time
Joined: Nov 14, 2011 4:39 pm
Full Name: David Thomas
Contact:

Re: CBT Reset KB article clarification

Post by helpdeskuk » Nov 21, 2014 3:09 pm

Many thanks!

joergr
Expert
Posts: 386
Liked: 39 times
Joined: Jun 08, 2010 2:01 pm
Full Name: Joerg Riether
Contact:

Re: CBT Reset KB article clarification

Post by joergr » Nov 22, 2014 1:21 pm

@helpdeskuk: I would highly suggest to use this script - it can reset cbt automatically on all of your vms WITHOUT the need for shutting anything down. You can use it while the vms are running.

foggy
Veeam Software
Posts: 18457
Liked: 1589 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: VMware CBT bug KB 2090639

Post by foggy » Nov 23, 2014 8:40 pm

Guido wrote:I ran the script for one VM. It only takes a few seconds to complete. No errors. CBT is off afterwards.
But there are no messages to be found in the Tasks&events logging within VIclient for snapshotting and removing the snapshot. Is this normal?
Looks suspicious. I would check VMware logs for snapshot creation/deletion operations and/or contact tech support to check whether you're doing everything correctly.

Guido
Influencer
Posts: 24
Liked: never
Joined: Jul 17, 2012 10:56 am
Full Name: Guido
Contact:

Re: VMware CBT bug KB 2090639

Post by Guido » Nov 24, 2014 9:21 am

If VIclient connected directly to the vsphere server the snapshotting is logged. If connected to the vcenter there is no logging. Thank you.

foggy
Veeam Software
Posts: 18457
Liked: 1589 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: VMware CBT bug KB 2090639

Post by foggy » Nov 24, 2014 12:18 pm

Guido wrote:If VIclient connected directly to the vsphere server the snapshotting is logged. If connected to the vcenter there is no logging. Thank you.
That means you're aiming the script to the host directly, not via vCenter.

r.mckeon
Enthusiast
Posts: 59
Liked: 2 times
Joined: May 27, 2014 8:25 am
Full Name: Rick McKeon
Contact:

Re: VMware CBT bug KB 2090639

Post by r.mckeon » Nov 26, 2014 2:03 pm

Hi, so I just found out about this bug/nightmare. This might be a stupid question but I need some peace of mind.

I recently expanded an ESXi datastore from 100GB to 200GB. I repeat, I only expanded the DATASTORE not the actual vmdk. This means I'm completely safe right?

But, if I find out that one of my vmdks was actually expanded, or I want to go through with a fix for above-mentioned VM anyway, do these overall steps make sense? Also, will the below steps solve for both backups and replica jobs?

Locked

Who is online

Users browsing this forum: No registered users and 15 guests