VMware CBT bug KB 2090639

[digitlman]

How about some help for those of us not good with Powershell?

I run the 32 bit vmware powershell, and I try and copy each line from the KB into the command box. The first and 2nd line work , but not the middle section. Putting them all on one line doesn't work either.

[chjones]

You need to run the scripts like you would a batch file or a vbscript, Powershell is no different. Copy all of the code into notepad and save it as a .ps1 file. Then in powershell you run the script. For example if you saved the file as C:\CBT-Fix.ps1, in the powershell prompt you would navigate to C:\ and run the command "./CBT-Fix.ps1" and it will execute all of the code (if you want to run a script you must put "./" preceding the file name)

[Stephenv]

Per the Sunday's email from GOSTEV Veeam v8 has a work around for this bug?

does this mean if i upgrade i do not need to disable and enable the CBT on a VM with this problem?

Stephen

[chjones]

The fix in v8 only works for any VMDKs that are increased in size AFTER v8 is installed. Any VMs already in your environment that have had their sizes changed in the past may be affected. You still need to run the CBT fix for existing VMs. The fix in v8 will only reset CBT automatically for you from that point forward.

Better to be safe than sorry is my motto when it comes to backups!

[kimhansen]

Maybe it is time for Veeam to add a option to periodically do a "full verify" on backups - which ignores CBT? This kind of thing would have been discovered and caught much ealier.. and give everybody some piece of mind? I understand the theory around surebackup, but even with this CBT bug, some OSes would boot and behave "normally".

[dellock6]

Technically, you can already ignore CBT in a backup job and use disk scan.
But the problem here is not only ignoring CBT, because at next execution the bug will hit anyway, you have to properly disable and re-enable it in order to remove corrupted informations.

[kimhansen]

Luca - if that was a reply to me, I don't fully buy that argument. If the backup used CBT information but the verify did not use any CBT information. It would definately show that verification of the backup failed - every time you did a verify. It would be a indication that something is wrong? I think this bug whould have been caught years ago...

In any case, it is just a suggestion and something I have missed somewhat to be able to do. If the verify feature was "periodic", you could for example run a verify once a week.

[dellock6]

Everyone trusted completely CBT before the bug came out, however we did suggested for a long time to run surebackup at least for the most critical VMs exactly for this reason. Even without the CBT bug, different issues can come and they only way to catch any possible problem is to effectively test a restore, and surebackup is designed for this. Sorry if it seems a marketing answer, but the whole reason for designing surebackup is exactly this, and in a way the CBT bug is the biggest validation so far for this feature.

[kimhansen]

The point here is that surebackup restore is not a guarantee that that the data is fully uncorrputed. You may be able to boot even with this CBT bug. So ideally you should have both. When it comes to CBT reliability, there surely are ways for the CBT to get corrupted. For example SAN crash / power failures etc.. If I'm the only one thinking it would be a good idea/feature to have - but I can live with that

[v334m]

I want to avoid snapping all my VMs at once, in batches or during biz hours... Can i do the following?

Disable CBT via Powershell on all VMs
Let the next Veeam job run (incremental)
Veeam will then enable CBT and snap/thaw

After this is done, i can upgrade to v8 and sleep soundly again?

[nreutemann]

I run the script on "biz hours", no impact, even on SQL servers. (I fully trust on Gostev words)

Before I run the script for all the VMs, I try over some VMs less critical, to see the impact and, after 15 trys I dont see any trouble, so, I run the script over all the infraestructure.

I dont run the incremental, I wait until the friday (when we run Incremental + Synthetic), run the script, upgrade to v8 and force an Active Full. The complete run took more time, but Im pretty sure, now, I got a good set of backups files.

[v334m]

I definitely cant run snaps during biz hours, VMs are disconnected for a few seconds, unfortunately in my business sector, this is unacceptable.

[dellock6]

Kim, I'm not saying you do not have a point here.
But as of now, CBT bug has shown up only when vmdk size is changed and crosses one 128GB boundary. So, also our workaround in V8 is more then what's technically needed to be safe. A complete disk scan, in my personal opinion (so not necessary the overall Veeam position on this) is an overkill, and the CBT reset upon disk size change is enough. And it should be applied now regardless a VM has just crossed the boundary, because if you have any VM above that size you don't know when it did changed the size or if it was deployed from the beginning above the boundary.
I'm more scared about old retention points, those kept for archival purposes, there's indeed a rick some of its content is unusable, thus making the whole idea of archiving useless...but at least we can fix the actual restore points.

PS: if you want to run a disk scan, technically you can already disable at all CBT in the backup job and let Veeam scan all blocks.

[nreutemann]

I definitely cant run snaps during biz hours, VMs are disconnected for a few seconds, unfortunately in my business sector, this is unacceptable.

I dont have any disconnection. I leave a ping running on some random VMs, and only have 1 ping with 250ms when the commit succeed.

But, maybe, my infrastructure its no big enough to have a big impact.

[cffit]

Can someone from VEEAM please post a Powershell script that will work with one specified VM at a time? I think that would help a lot of people out after reading through this post. A lot of us want to be cautious and try the script out on a single VM or a few test VMs even though the script provided on here earlier that does all VMs at once is said to be safe to run, if we could have a script that runs against one VM at a time that would be nice.

I think it's important that this script is provided and verified by someone from VEEAM and not a forum end user. While I appreciate the other end user input on here, something this critical I want to hear directly from VEEAM.

Also, once this is done, as I understand, there is no way to verify that our VMs are safe moving forward? We just have to make sure we run the script on every VM and that is our assurance?

[mrt]

Can someone from VEEAM please post a Powershell script that will work with one specified VM at a time?

this script posted earlier in the thread does just that and worked great for me: http://forums.veeam.com/vmware-vsphere- ... ml#p123778
It's not "from Veeam" but it is straightforward

[digitlman]

You need to run the scripts like you would a batch file or a vbscript, Powershell is no different. Copy all of the code into notepad and save it as a .ps1 file. Then in powershell you run the script. For example if you saved the file as C:\CBT-Fix.ps1, in the powershell prompt you would navigate to C:\ and run the command "./CBT-Fix.ps1" and it will execute all of the code (if you want to run a script you must put "./" preceding the file name)

That worked like a charm. Thanks!

[cffit]

this script posted earlier in the thread does just that and worked great for me: http://forums.veeam.com/vmware-vsphere- ... ml#p123778
It's not "from Veeam" but it is straightforward

Ok I did this and it appears to work. Is there a line I can run right after I've done this to verify if CBT is in fact disabled for sure? When I go in though the vSphere client and look at the properties it still states that the disk CBT is turned on. I know the value in the script applies to the whole VM, but I just want a way to verify that after this script is run that CBT is in fact disabled for sure. Thanks!

[xadamz23]

Code: Select all

get-vm your_vm_name | ?{$_.ExtensionData.Config.ChangeTrackingEnabled -eq $true}

It should return to a prompt without displaying anything if CBT is currently disabled on the VM. If CBT is still enabled it will produce output.

[bearflatfish]

Dears,

I met a problem when running the backup job after using the script of KB1940.
The script was run normally, with the CTK disable and snapshot created/removed.
The job always failed at the first execution after CTK has been reset. The error messages were as following:
"Job has failed unexpectedly"
And the following INCR or manually active backup are completed successfully.

The task log shows there's something wrong at the first backup session after CBT reset.

Can someone give me some hints?

Regards,
Joe

[veremin]

Hi,

Kindly, avoid posting log snippets, as it's against our forum rules. Please, contact our support team directly for any technical issue encountered.

Thanks.

[bearflatfish]

apologize for my mistake.
Because I'm verifying the impact of this VMware CBT bug for our customer.
I'm not sure if I can open a support case with my evaluation key.
So that I would like ask someone experienced.
Maybe I'll open a support request later to see if what they can help me.
Thanks for your reminder.

[veremin]

According to our policy , customers even with evaluation license are eligible for support. So, feel free to contact them. Thanks.

[Vitaliy S.]

Guys,

If you're using Veeam ONE or Veeam MP for SCOM, then here is a small trick that would allow you to detect ALL VMs that had their virtual disks extended and could be potentially affected by this CBT bug.

Veeam ONE:

1. Navigate to VMware configuration tracking report pack and select Infrastructure Changes by Object report
2. In the parameters specify "Virtual Machine" as a reporting object
3. In the object properties choose "VM disk size"

Veeam MP:

1. Select Configuration Tracking and Alert correlation report
2. Choose "group changes by property"
3. In the reporting scope select "VMware virtual machines" as an object class
4. In the parameters choose "VMware Virtual Machine: Disk Allocation GB" property

Please be aware that you can select any reporting period in these reports. Hope you will find this tip useful.

Thanks!

[EBoucq]

Hello,

Following Gostev's weekly post, could you confirm that the hotfix for B&R v7 patch#4 to fix the VMware CBT data corruption issue just consists in .DLL copy in the Veeam directory ?
Thank you.

Regards,
Eric

[johan.h]

You should replace the .dll files and then restart all Veeam services.

[EBoucq]

Thank you.

[johan.h]

No worries . (I like keeping an old copy of .dll files, but it should be fine)

[m.novelli]

Hi guys, one question about the script to reset CBT: I have an host with both production VM and some replicated VM from another host
Can I run the script against all VM or should exclude the replicated (target) VM?
Thanks

Marco

[foggy]

You can reset CBT on all of the VMs. Even if you're backing up replica VMs, CBT still cannot be used for them.