CBT failing on drives other than Hard Disk 1

[rhnb]

I've got a call open with tech support (5159848) but thought I'd post here in case anyone else has seen this issue and knows of a fix (I could do with it urgently!)

Since upgrading to v6, most (not all) of my Windows VM's which have more than 1 vmdk are not using CBT on the 2nd/3rd/... disk. The error I see is...

11/12/2011 23:58:22 :: Cannot use CBT: Soap fault. A specified parameter was not correct. . deviceKeyDetail: '<InvalidArgumentFault xmlns="urn:internalvim25" xsi:type="InvalidArgument"><invalidProperty>deviceKey</invalidProperty></InvalidArgumentFault>', endpoint: ''

I've tried one suggestion (setting all the ctkenabled stuff to false then doing another couple of backups), but that hasn't fixed it (only been able to try that on one VM as the VM needs to be powered off to change these settings). I can confirm that CBT gets re-enabled again.

I can't see anything different between VMs that work and ones that don't, and I'm seeing it on a mixture of Windows2003 and Windows2008 VM's. Might just be coincidence, but am not seeing it on any Linux VM's. I didn't see this issue with v5, and it's causing havoc with my backup window!

Any ideas much appreciated.
Cheeers... allan.

[jhdryden]

Same problem here. Also opened a case with support

[aweiss]

Hi
Same problem here. I still need to disable cbt on the VM to make some tests.
Rhnb : did you powered off VM, disable CTK generaly and on all disks, powered on VM, powered off again and powered on a last time ? You should be able to see that ctk.vmdk files have been deleted. Then try to run a Veeam backup again to see if ctk.mdk files get recreated. I'm gonna do it soon. No case opened yet at support.

[aweiss]

By the way, on my case, i think it is related to VM that previoulsy had pRDM. Recently, my customer converted them to vmdk but we found that sometimes ctk would stay disabled on these disks.

[rhnb]

aweiss: Here's what I did...
Powered off the VM
Disabled CTK generally and on all disks (scsi0:n.ctkenabled=false)
Powered on the VM.
Did a backup (no issues as it was a full).
Did a second backup (failed CBT on 2nd disk).
Looked at the settings again - ctk was now enabled as it should be.

What I didn't check was whether the ctk.vmdk files had disappeared once I'd restarted the VM (does it do that when you set ctk to false?) Even if it doesn't it's not a file I'd want to simply 'delete' without some assurance

In my case, I've got 29 VM's which are showing this behaviour.

[aweiss]

Last time i had an issue with instant recovery a VM, Veeam support gave me theses steps to reset CBT and it was important to power off,on,off and on
Here's a cut and paste of my ticket for a previous problem i had 3 months ago.
*******************
we can try resetting the CBT for the VM
here is an instruction on how you can reset CBT on the guest VM:
1) power off VM
2) follow the steps shown below:

- You need to set "ctkEnabled" value to false
- You need to set "scsi0:x.ctkEnabled" value to false (false should be set for each disk of the VM in question)

3) power on VM
4) power off VM again (that step is needed to update CTK DataBase)
5) power on VM (from that time you may re-run the job and CTK will be automatically enabled).
**************

Let me know but if you have this on all VM ... it could be a V6 related issue
By the way, does your Veeam Database is on an external SQL server or embedded with the Veeam Server ?

Thks
Alex

[rhnb]

Thanks Alex
I've now done that and will see if that works (won't know 'til tomorrow as a load of backups are currently queued up). I can confirm that once you power up the VM (after setting ctk to false), the ctk.vmdk file disappears.

I just had a reply from tech support and they've asked me to try a 'workaround' by adding one of my ESXi hosts in Servers and create a job to see if it then works. They say it could be a problem with my vCenter Server and the ESXi hosts?
I'm going to try it, but if it works it's not a fix I can live with (no vMotion!).

This doesn't happen on ALL VM's. I have some VM's with multiple disks that are using CBT fine. It's not related to how old the VM is either. Got some old/some new VMs in this mix.

SQL database is local to the Veeam Backup server (which is not a proxy).

[Gostev]

Thanks for including your support case ID, and please keep us posted on your findings!

[aweiss]

Hi Anton
Seems my SQL VM has been backed up with success. I have no more CBT issu. I need to double Check with my customer if he followed my advise of disabling CBT before the backup job ran or if the issue disappeared because of something else. I'll keep you updated tomorrow. Rgds. Alex

[rhnb]

Well, some progress!

I added in one of my ESXi hosts as a server, created a test job and backed up one of the CBT failing VM's - that worked.
I then put that VM in a test job which uses my vCenter Server and backed it up a couple of times. The first was a full (of course) and the second an incremental which used CBT.

So, to fix my issue it looks as though I'll have to plough through 29 VM's, back them up using the ESXi host(s) and then it looks like the CBT problem will be sorted. No idea why this fixes it, and why it's only some VM's. I just hope it's a permanent fix! Fingers crossed.

At least I don;t have to take the VM down, remove CBT, power on/off a couple of times etc. I can do all this while the VMs are up. Going to take a long time though

[aweiss]

Hi
I confirm that disabling CBT and re enable it made it for me. All jobs are OK now.
Affected VM were the one that previously use pRDM and that were migrated to classic vmdk.
Alex

[Gostev]

Alexandre, thanks fort the update. Do you know if this environment get upgraded to vSphere 5 recently? I have heard this on a multiple occasions now that upgrading to vSphere 5 seem to mess up CBT data on some VMs. Thanks.

[aweiss]

My customer is in vSphere 4.1 U1. We have just migrated from veeam 5 to 6. All vSphere 5 install i make are always from scratch even if customer is in 3.5 or 4.1

[rhnb]

So, to fix my issue it looks as though I'll have to plough through 29 VM's, back them up using the ESXi host(s) and then it looks like the CBT problem will be sorted. No idea why this fixes it, and why it's only some VM's. I just hope it's a permanent fix! Fingers crossed.

Spoke too soon, it hasn't fixed the issue and tech support are still looking at it.
The bulk of the machines that were failing CBT still are (a couple are now working) BUT, just to complicate things, a couple that WERE working are now failing. I've also tried the recommended way of removing CBT but that didn't fix it either.

[Gostev]

I too was curious why would going through ESXi hosts help as in this case the same CBT data is used. Really, the only reliable way to fix CBT issues is to go through CBT reset process during your next maintenance window.