Discussions specific to the VMware vSphere hypervisor
Locked
digitlman
Enthusiast
Posts: 94
Liked: 3 times
Joined: Jun 10, 2010 6:32 pm
Contact:

Re: VMware CBT bug KB 2090639

Post by digitlman » Nov 10, 2014 5:01 pm

How about some help for those of us not good with Powershell?

I run the 32 bit vmware powershell, and I try and copy each line from the KB into the command box. The first and 2nd line work , but not the middle section. Putting them all on one line doesn't work either.

chjones
Expert
Posts: 106
Liked: 27 times
Joined: Oct 30, 2012 7:53 pm
Full Name: Chris Jones
Contact:

Re: VMware CBT bug KB 2090639

Post by chjones » Nov 10, 2014 7:58 pm

You need to run the scripts like you would a batch file or a vbscript, Powershell is no different. Copy all of the code into notepad and save it as a .ps1 file. Then in powershell you run the script. For example if you saved the file as C:\CBT-Fix.ps1, in the powershell prompt you would navigate to C:\ and run the command "./CBT-Fix.ps1" and it will execute all of the code (if you want to run a script you must put "./" preceding the file name)

Stephenv
Influencer
Posts: 12
Liked: 1 time
Joined: Dec 01, 2011 5:04 pm
Full Name: Stephen
Contact:

Re: VMware CBT bug KB 2090639

Post by Stephenv » Nov 10, 2014 9:07 pm

Per the Sunday's email from GOSTEV Veeam v8 has a work around for this bug?

does this mean if i upgrade i do not need to disable and enable the CBT on a VM with this problem?

Stephen

chjones
Expert
Posts: 106
Liked: 27 times
Joined: Oct 30, 2012 7:53 pm
Full Name: Chris Jones
Contact:

Re: VMware CBT bug KB 2090639

Post by chjones » Nov 10, 2014 9:23 pm 1 person likes this post

The fix in v8 only works for any VMDKs that are increased in size AFTER v8 is installed. Any VMs already in your environment that have had their sizes changed in the past may be affected. You still need to run the CBT fix for existing VMs. The fix in v8 will only reset CBT automatically for you from that point forward.

Better to be safe than sorry is my motto when it comes to backups!

kimhansen
Novice
Posts: 7
Liked: 2 times
Joined: Jul 20, 2009 7:05 pm
Full Name: Kim Alexander Hansen
Contact:

Re: VMware CBT bug KB 2090639

Post by kimhansen » Nov 11, 2014 7:29 am 1 person likes this post

Maybe it is time for Veeam to add a option to periodically do a "full verify" on backups - which ignores CBT? This kind of thing would have been discovered and caught much ealier.. and give everybody some piece of mind? I understand the theory around surebackup, but even with this CBT bug, some OSes would boot and behave "normally".

dellock6
Veeam Software
Posts: 5683
Liked: 1603 times
Joined: Jul 26, 2009 3:39 pm
Full Name: Luca Dell'Oca
Location: Varese, Italy
Contact:

Re: VMware CBT bug KB 2090639

Post by dellock6 » Nov 11, 2014 8:11 am 1 person likes this post

Technically, you can already ignore CBT in a backup job and use disk scan.
But the problem here is not only ignoring CBT, because at next execution the bug will hit anyway, you have to properly disable and re-enable it in order to remove corrupted informations.
Luca Dell'Oca
Principal EMEA Cloud Architect @ Veeam Software

@dellock6
https://www.virtualtothecore.com/
vExpert 2011 -> 2019
Veeam VMCE #1

kimhansen
Novice
Posts: 7
Liked: 2 times
Joined: Jul 20, 2009 7:05 pm
Full Name: Kim Alexander Hansen
Contact:

Re: VMware CBT bug KB 2090639

Post by kimhansen » Nov 11, 2014 8:27 am

Luca - if that was a reply to me, I don't fully buy that argument. If the backup used CBT information but the verify did not use any CBT information. It would definately show that verification of the backup failed - every time you did a verify. It would be a indication that something is wrong? I think this bug whould have been caught years ago...

In any case, it is just a suggestion and something I have missed somewhat to be able to do. If the verify feature was "periodic", you could for example run a verify once a week.

dellock6
Veeam Software
Posts: 5683
Liked: 1603 times
Joined: Jul 26, 2009 3:39 pm
Full Name: Luca Dell'Oca
Location: Varese, Italy
Contact:

Re: VMware CBT bug KB 2090639

Post by dellock6 » Nov 11, 2014 8:33 am 2 people like this post

Everyone trusted completely CBT before the bug came out, however we did suggested for a long time to run surebackup at least for the most critical VMs exactly for this reason. Even without the CBT bug, different issues can come and they only way to catch any possible problem is to effectively test a restore, and surebackup is designed for this. Sorry if it seems a marketing answer, but the whole reason for designing surebackup is exactly this, and in a way the CBT bug is the biggest validation so far for this feature.
Luca Dell'Oca
Principal EMEA Cloud Architect @ Veeam Software

@dellock6
https://www.virtualtothecore.com/
vExpert 2011 -> 2019
Veeam VMCE #1

kimhansen
Novice
Posts: 7
Liked: 2 times
Joined: Jul 20, 2009 7:05 pm
Full Name: Kim Alexander Hansen
Contact:

Re: VMware CBT bug KB 2090639

Post by kimhansen » Nov 11, 2014 9:09 am

The point here is that surebackup restore is not a guarantee that that the data is fully uncorrputed. You may be able to boot even with this CBT bug. So ideally you should have both. When it comes to CBT reliability, there surely are ways for the CBT to get corrupted. For example SAN crash / power failures etc.. If I'm the only one thinking it would be a good idea/feature to have - but I can live with that :)

v334m
Influencer
Posts: 12
Liked: never
Joined: Aug 06, 2014 8:13 am
Full Name: A
Contact:

Re: VMware CBT bug KB 2090639

Post by v334m » Nov 11, 2014 10:42 am

I want to avoid snapping all my VMs at once, in batches or during biz hours... Can i do the following?

Disable CBT via Powershell on all VMs
Let the next Veeam job run (incremental)
Veeam will then enable CBT and snap/thaw

After this is done, i can upgrade to v8 and sleep soundly again?

nreutemann
Enthusiast
Posts: 47
Liked: 6 times
Joined: Mar 06, 2012 11:45 pm
Full Name: Nicolas Reutemann
Contact:

Re: VMware CBT bug KB 2090639

Post by nreutemann » Nov 11, 2014 10:56 am

I run the script on "biz hours", no impact, even on SQL servers. (I fully trust on Gostev words)

Before I run the script for all the VMs, I try over some VMs less critical, to see the impact and, after 15 trys I dont see any trouble, so, I run the script over all the infraestructure.

I dont run the incremental, I wait until the friday (when we run Incremental + Synthetic), run the script, upgrade to v8 and force an Active Full. The complete run took more time, but Im pretty sure, now, I got a good set of backups files.

v334m
Influencer
Posts: 12
Liked: never
Joined: Aug 06, 2014 8:13 am
Full Name: A
Contact:

Re: VMware CBT bug KB 2090639

Post by v334m » Nov 11, 2014 10:58 am

I definitely cant run snaps during biz hours, VMs are disconnected for a few seconds, unfortunately in my business sector, this is unacceptable.

dellock6
Veeam Software
Posts: 5683
Liked: 1603 times
Joined: Jul 26, 2009 3:39 pm
Full Name: Luca Dell'Oca
Location: Varese, Italy
Contact:

Re: VMware CBT bug KB 2090639

Post by dellock6 » Nov 11, 2014 11:11 am

Kim, I'm not saying you do not have a point here.
But as of now, CBT bug has shown up only when vmdk size is changed and crosses one 128GB boundary. So, also our workaround in V8 is more then what's technically needed to be safe. A complete disk scan, in my personal opinion (so not necessary the overall Veeam position on this) is an overkill, and the CBT reset upon disk size change is enough. And it should be applied now regardless a VM has just crossed the boundary, because if you have any VM above that size you don't know when it did changed the size or if it was deployed from the beginning above the boundary.
I'm more scared about old retention points, those kept for archival purposes, there's indeed a rick some of its content is unusable, thus making the whole idea of archiving useless...but at least we can fix the actual restore points.

PS: if you want to run a disk scan, technically you can already disable at all CBT in the backup job and let Veeam scan all blocks.
Luca Dell'Oca
Principal EMEA Cloud Architect @ Veeam Software

@dellock6
https://www.virtualtothecore.com/
vExpert 2011 -> 2019
Veeam VMCE #1

nreutemann
Enthusiast
Posts: 47
Liked: 6 times
Joined: Mar 06, 2012 11:45 pm
Full Name: Nicolas Reutemann
Contact:

Re: VMware CBT bug KB 2090639

Post by nreutemann » Nov 11, 2014 11:37 am

v334m wrote:I definitely cant run snaps during biz hours, VMs are disconnected for a few seconds, unfortunately in my business sector, this is unacceptable.
I dont have any disconnection. I leave a ping running on some random VMs, and only have 1 ping with 250ms when the commit succeed.

But, maybe, my infrastructure its no big enough to have a big impact.

cffit
Expert
Posts: 338
Liked: 34 times
Joined: Jan 20, 2012 2:36 pm
Full Name: Christensen Farms
Contact:

Re: VMware CBT bug KB 2090639

Post by cffit » Nov 11, 2014 2:54 pm

Can someone from VEEAM please post a Powershell script that will work with one specified VM at a time? I think that would help a lot of people out after reading through this post. A lot of us want to be cautious and try the script out on a single VM or a few test VMs even though the script provided on here earlier that does all VMs at once is said to be safe to run, if we could have a script that runs against one VM at a time that would be nice.

I think it's important that this script is provided and verified by someone from VEEAM and not a forum end user. While I appreciate the other end user input on here, something this critical I want to hear directly from VEEAM.

Also, once this is done, as I understand, there is no way to verify that our VMs are safe moving forward? We just have to make sure we run the script on every VM and that is our assurance?

mrt
Enthusiast
Posts: 47
Liked: 2 times
Joined: Feb 10, 2011 7:27 pm
Contact:

Re: VMware CBT bug KB 2090639

Post by mrt » Nov 11, 2014 2:58 pm 1 person likes this post

cffit wrote:Can someone from VEEAM please post a Powershell script that will work with one specified VM at a time?
this script posted earlier in the thread does just that and worked great for me: http://forums.veeam.com/vmware-vsphere- ... ml#p123778
It's not "from Veeam" but it is straightforward

digitlman
Enthusiast
Posts: 94
Liked: 3 times
Joined: Jun 10, 2010 6:32 pm
Contact:

Re: VMware CBT bug KB 2090639

Post by digitlman » Nov 11, 2014 3:12 pm 1 person likes this post

chjones wrote:You need to run the scripts like you would a batch file or a vbscript, Powershell is no different. Copy all of the code into notepad and save it as a .ps1 file. Then in powershell you run the script. For example if you saved the file as C:\CBT-Fix.ps1, in the powershell prompt you would navigate to C:\ and run the command "./CBT-Fix.ps1" and it will execute all of the code (if you want to run a script you must put "./" preceding the file name)
That worked like a charm. Thanks!

cffit
Expert
Posts: 338
Liked: 34 times
Joined: Jan 20, 2012 2:36 pm
Full Name: Christensen Farms
Contact:

Re: VMware CBT bug KB 2090639

Post by cffit » Nov 11, 2014 6:19 pm

mrt wrote: this script posted earlier in the thread does just that and worked great for me: http://forums.veeam.com/vmware-vsphere- ... ml#p123778
It's not "from Veeam" but it is straightforward
Ok I did this and it appears to work. Is there a line I can run right after I've done this to verify if CBT is in fact disabled for sure? When I go in though the vSphere client and look at the properties it still states that the disk CBT is turned on. I know the value in the script applies to the whole VM, but I just want a way to verify that after this script is run that CBT is in fact disabled for sure. Thanks!

xadamz23
Influencer
Posts: 17
Liked: 3 times
Joined: Dec 08, 2009 10:24 pm
Contact:

Re: VMware CBT bug KB 2090639

Post by xadamz23 » Nov 11, 2014 6:52 pm

Code: Select all

get-vm your_vm_name | ?{$_.ExtensionData.Config.ChangeTrackingEnabled -eq $true}
It should return to a prompt without displaying anything if CBT is currently disabled on the VM. If CBT is still enabled it will produce output.

bearflatfish
Novice
Posts: 5
Liked: 1 time
Joined: Dec 27, 2013 5:46 am
Full Name: Chia-hung Tseng
Contact:

Re: VMware CBT bug KB 2090639

Post by bearflatfish » Nov 12, 2014 5:44 am

Dears,

I met a problem when running the backup job after using the script of KB1940.
The script was run normally, with the CTK disable and snapshot created/removed.
The job always failed at the first execution after CTK has been reset. The error messages were as following:
"Job has failed unexpectedly"
And the following INCR or manually active backup are completed successfully.

The task log shows there's something wrong at the first backup session after CBT reset.

Can someone give me some hints?

Regards,
Joe

veremin
Product Manager
Posts: 16669
Liked: 1390 times
Joined: Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin
Contact:

Re: VMware CBT bug KB 2090639

Post by veremin » Nov 12, 2014 10:00 am

Hi,

Kindly, avoid posting log snippets, as it's against our forum rules. Please, contact our support team directly for any technical issue encountered.

Thanks.

bearflatfish
Novice
Posts: 5
Liked: 1 time
Joined: Dec 27, 2013 5:46 am
Full Name: Chia-hung Tseng
Contact:

Re: VMware CBT bug KB 2090639

Post by bearflatfish » Nov 13, 2014 1:10 am

apologize for my mistake.
Because I'm verifying the impact of this VMware CBT bug for our customer.
I'm not sure if I can open a support case with my evaluation key.
So that I would like ask someone experienced.
Maybe I'll open a support request later to see if what they can help me.
Thanks for your reminder.

veremin
Product Manager
Posts: 16669
Liked: 1390 times
Joined: Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin
Contact:

Re: VMware CBT bug KB 2090639

Post by veremin » Nov 13, 2014 11:52 am

According to our policy , customers even with evaluation license are eligible for support. So, feel free to contact them. Thanks.

Vitaliy S.
Product Manager
Posts: 22760
Liked: 1519 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: VMware CBT bug KB 2090639

Post by Vitaliy S. » Nov 14, 2014 3:59 pm 4 people like this post

Guys,

If you're using Veeam ONE or Veeam MP for SCOM, then here is a small trick that would allow you to detect ALL VMs that had their virtual disks extended and could be potentially affected by this CBT bug.

Veeam ONE:

1. Navigate to VMware configuration tracking report pack and select Infrastructure Changes by Object report
2. In the parameters specify "Virtual Machine" as a reporting object
3. In the object properties choose "VM disk size"

Veeam MP:

1. Select Configuration Tracking and Alert correlation report
2. Choose "group changes by property"
3. In the reporting scope select "VMware virtual machines" as an object class
4. In the parameters choose "VMware Virtual Machine: Disk Allocation GB" property

Please be aware that you can select any reporting period in these reports. Hope you will find this tip useful.

Thanks!

EBoucq
Expert
Posts: 232
Liked: 15 times
Joined: Jan 21, 2011 10:10 am
Full Name: Eric Boucq
Contact:

[MERGED] VMware CBT data corruption: hot fix installation

Post by EBoucq » Nov 17, 2014 10:47 am

Hello,

Following Gostev's weekly post, could you confirm that the hotfix for B&R v7 patch#4 to fix the VMware CBT data corruption issue just consists in .DLL copy in the Veeam directory ?
Thank you.

Regards,
Eric

johan.h
Expert
Posts: 536
Liked: 110 times
Joined: Jun 05, 2013 9:45 am
Full Name: Johan Huttenga
Contact:

Re: VMware CBT data corruption: hot fix installation

Post by johan.h » Nov 17, 2014 10:57 am 1 person likes this post

You should replace the .dll files and then restart all Veeam services.

EBoucq
Expert
Posts: 232
Liked: 15 times
Joined: Jan 21, 2011 10:10 am
Full Name: Eric Boucq
Contact:

Re: VMware CBT data corruption: hot fix installation

Post by EBoucq » Nov 17, 2014 11:02 am

Thank you.

johan.h
Expert
Posts: 536
Liked: 110 times
Joined: Jun 05, 2013 9:45 am
Full Name: Johan Huttenga
Contact:

Re: VMware CBT data corruption: hot fix installation

Post by johan.h » Nov 17, 2014 11:08 am

No worries :). (I like keeping an old copy of .dll files, but it should be fine)

m.novelli
Veeam ProPartner
Posts: 353
Liked: 40 times
Joined: Dec 29, 2009 12:48 pm
Full Name: Marco Novelli
Location: Asti - Italy
Contact:

Re: VMware CBT bug KB 2090639

Post by m.novelli » Nov 17, 2014 2:06 pm

Hi guys, one question about the script to reset CBT: I have an host with both production VM and some replicated VM from another host
Can I run the script against all VM or should exclude the replicated (target) VM?
Thanks

Marco

foggy
Veeam Software
Posts: 18026
Liked: 1531 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: VMware CBT bug KB 2090639

Post by foggy » Nov 17, 2014 2:34 pm

You can reset CBT on all of the VMs. Even if you're backing up replica VMs, CBT still cannot be used for them.

Locked

Who is online

Users browsing this forum: Baidu [Spider], Bing [Bot] and 27 guests