CBT failing on drives other than Hard Disk 1

[rhnb]

I've got a call open with tech support (5159848) but thought I'd post here in case anyone else has seen this issue and knows of a fix (I could do with it urgently!)

Since upgrading to v6, most (not all) of my Windows VM's which have more than 1 vmdk are not using CBT on the 2nd/3rd/... disk. The error I see is...

11/12/2011 23:58:22 :: Cannot use CBT: Soap fault. A specified parameter was not correct. . deviceKeyDetail: '<InvalidArgumentFault xmlns="urn:internalvim25" xsi:type="InvalidArgument"><invalidProperty>deviceKey</invalidProperty></InvalidArgumentFault>', endpoint: ''

I've tried one suggestion (setting all the ctkenabled stuff to false then doing another couple of backups), but that hasn't fixed it (only been able to try that on one VM as the VM needs to be powered off to change these settings). I can confirm that CBT gets re-enabled again.

I can't see anything different between VMs that work and ones that don't, and I'm seeing it on a mixture of Windows2003 and Windows2008 VM's. Might just be coincidence, but am not seeing it on any Linux VM's. I didn't see this issue with v5, and it's causing havoc with my backup window!

Any ideas much appreciated.
Cheeers... allan.

[jhdryden]

Same problem here. Also opened a case with support

[aweiss]

Hi
Same problem here. I still need to disable cbt on the VM to make some tests.
Rhnb : did you powered off VM, disable CTK generaly and on all disks, powered on VM, powered off again and powered on a last time ? You should be able to see that ctk.vmdk files have been deleted. Then try to run a Veeam backup again to see if ctk.mdk files get recreated. I'm gonna do it soon. No case opened yet at support.

[aweiss]

By the way, on my case, i think it is related to VM that previoulsy had pRDM. Recently, my customer converted them to vmdk but we found that sometimes ctk would stay disabled on these disks.

[rhnb]

aweiss: Here's what I did...
Powered off the VM
Disabled CTK generally and on all disks (scsi0:n.ctkenabled=false)
Powered on the VM.
Did a backup (no issues as it was a full).
Did a second backup (failed CBT on 2nd disk).
Looked at the settings again - ctk was now enabled as it should be.

What I didn't check was whether the ctk.vmdk files had disappeared once I'd restarted the VM (does it do that when you set ctk to false?) Even if it doesn't it's not a file I'd want to simply 'delete' without some assurance

In my case, I've got 29 VM's which are showing this behaviour.

[aweiss]

Last time i had an issue with instant recovery a VM, Veeam support gave me theses steps to reset CBT and it was important to power off,on,off and on
Here's a cut and paste of my ticket for a previous problem i had 3 months ago.
*******************
we can try resetting the CBT for the VM
here is an instruction on how you can reset CBT on the guest VM:
1) power off VM
2) follow the steps shown below:

- You need to set "ctkEnabled" value to false
- You need to set "scsi0:x.ctkEnabled" value to false (false should be set for each disk of the VM in question)

3) power on VM
4) power off VM again (that step is needed to update CTK DataBase)
5) power on VM (from that time you may re-run the job and CTK will be automatically enabled).
**************

Let me know but if you have this on all VM ... it could be a V6 related issue
By the way, does your Veeam Database is on an external SQL server or embedded with the Veeam Server ?

Thks
Alex

[rhnb]

Thanks Alex
I've now done that and will see if that works (won't know 'til tomorrow as a load of backups are currently queued up). I can confirm that once you power up the VM (after setting ctk to false), the ctk.vmdk file disappears.

I just had a reply from tech support and they've asked me to try a 'workaround' by adding one of my ESXi hosts in Servers and create a job to see if it then works. They say it could be a problem with my vCenter Server and the ESXi hosts?
I'm going to try it, but if it works it's not a fix I can live with (no vMotion!).

This doesn't happen on ALL VM's. I have some VM's with multiple disks that are using CBT fine. It's not related to how old the VM is either. Got some old/some new VMs in this mix.

SQL database is local to the Veeam Backup server (which is not a proxy).

[Gostev]

Thanks for including your support case ID, and please keep us posted on your findings!

[aweiss]

Hi Anton
Seems my SQL VM has been backed up with success. I have no more CBT issu. I need to double Check with my customer if he followed my advise of disabling CBT before the backup job ran or if the issue disappeared because of something else. I'll keep you updated tomorrow. Rgds. Alex

[rhnb]

Well, some progress!

I added in one of my ESXi hosts as a server, created a test job and backed up one of the CBT failing VM's - that worked.
I then put that VM in a test job which uses my vCenter Server and backed it up a couple of times. The first was a full (of course) and the second an incremental which used CBT.

So, to fix my issue it looks as though I'll have to plough through 29 VM's, back them up using the ESXi host(s) and then it looks like the CBT problem will be sorted. No idea why this fixes it, and why it's only some VM's. I just hope it's a permanent fix! Fingers crossed.

At least I don;t have to take the VM down, remove CBT, power on/off a couple of times etc. I can do all this while the VMs are up. Going to take a long time though

[aweiss]

Hi
I confirm that disabling CBT and re enable it made it for me. All jobs are OK now.
Affected VM were the one that previously use pRDM and that were migrated to classic vmdk.
Alex

[Gostev]

Alexandre, thanks fort the update. Do you know if this environment get upgraded to vSphere 5 recently? I have heard this on a multiple occasions now that upgrading to vSphere 5 seem to mess up CBT data on some VMs. Thanks.

[aweiss]

My customer is in vSphere 4.1 U1. We have just migrated from veeam 5 to 6. All vSphere 5 install i make are always from scratch even if customer is in 3.5 or 4.1

[rhnb]

So, to fix my issue it looks as though I'll have to plough through 29 VM's, back them up using the ESXi host(s) and then it looks like the CBT problem will be sorted. No idea why this fixes it, and why it's only some VM's. I just hope it's a permanent fix! Fingers crossed.

Spoke too soon, it hasn't fixed the issue and tech support are still looking at it.
The bulk of the machines that were failing CBT still are (a couple are now working) BUT, just to complicate things, a couple that WERE working are now failing. I've also tried the recommended way of removing CBT but that didn't fix it either.

[Gostev]

I too was curious why would going through ESXi hosts help as in this case the same CBT data is used. Really, the only reliable way to fix CBT issues is to go through CBT reset process during your next maintenance window.

[rhnb]

I've tried the reset sequence on 4 VM's now and that hasn't helped I'm afraid (I wish it had!). Plus though the bulk of VM's failing CBT remain the same, it does change now and again. I take your point about the CBT data, that's common whether you're using vCenter or ESXi host, so it points more to vCenter server being the issue perhaps.
We're about to update our vCenter server so it will be interesting to see if that has any effect.

[jhdryden]

As suggested by Veeam support, the following worked for me (deleting CTK and running 2 replication jobs)

There is a stuck snapshot on the VM causing change block tracking not to work:

You will need to manually create a snapshot and then go into the snapshot manager and click on Delete All. After, please re-run the job, If change v=block tracking still is not working for the listed VM remove the -ctk.vmdk file (in the datastore file) for the VM it’s not working on and run the job twice (Once to re-create the CTK file and the second to use it). By doing that, it will re-set the change block tracking to work properly.

[dustyrooster]

It's definately a snapshot issue. To solve the problem myself I did a V2V conversion, which I know is a little heavy handed but I wanted to make sure the snapshots where all consolidated. Works fine but will be trying the previous method as well.

[mysidia]

I think I may be running into the same kind of thing, CBT failing on drives other than Hard Disk 1. So far on any VM I tried to backup with Veeam 6; Environment vCenter 5 all hosts ESXi5; vCenter 5 was a clean install. I went through the CTK reset procedure mentioned by aweiss. No dice, same behavior persists.

Also, if I create a brand new Version 8 VM with 2 blank hard drives, power it on, install an OS; create a new backup job, add the new VM to the backup job. Run the backup job twice, the same warning appears on the brand new VM, and CBT is not used for the reverse incremental backup. The warning is raised immediately on the 2nd run of the new backup job
"Cannot use CBT: Soap fault. A specified parameter was not correct. . deviceKeyDetail:"

After investigation I see that in fact "2001" seems to be a valid device key for the disk device on the target VM.
That is.. the error doesn't make sense because 2001 in fact is the device key associated with the VMDK in the VM's
config spec.
On the very same VMDK that Veeam says CBT couldn't be used on, I inspected the VM's CBT status by invoking
QueryChangedDiskAreas(), using the vSphere PowerCLI, and I see what appears a proper response shown....

And I am able to fetch lists of changed blocks on the Veeam snapshot through Powershell while the backup is in progress, without an error or exception being raised..... So Powershell can see the changed blocks, but Veeam gets a SOAP error from vCenter when trying to query the same information?

[Gostev]

Hi, please avoid posting log snippets as per forum rules on the forum. For any technical issues, you should contact support and provide full debug logs for investigation, and include your support case ID instead of logs. Thanks!

[Gostev]

@mysidia developers had investigated some debug logs provided, and I've been told that this is a known VMware issue not related to v6 (existed in all previous versions as well). CBT query works fine for the first time it is called (before processing the first disk) - which is why the first disk is always processed correctly. However, for certain VMs, for cause yet unknown, it starts failing once we get to the following disks (you won't see this in your PowerCLI test because your script does not mount disks with VDDK). And indeed, the CTK cleanup will not help you with this specific problem.

While we cannot fix the issue with CBT failures, we do have some ideas on possible workaround, and are planning to prototype and test that. Thanks!

[mysidia]

@Gostev, thank you, I have a support case for this issue, # 5162648.
I'd be very interested to hear of any workarounds that don't require downgrading VMs to VM Hardware version 7

Well... when you say certain VMs.. I see the CBT error happening basically on all VMs. Including new VMs...
If I right click a cluster, create a brand new VM with 2 fresh VMDKs, Version 8 virtual hardware, the CBT failure predictably comes up at the 2nd backup for any new VM I make. When I create a new VM with version 7 hardware and try to back that up, the CBT seems to work correctly on freshly created Virtual Hardware Version 7 VMs.

(you won't see this in your PowerCLI test because your script does not mount disks with VDDK)

Well, that's very interesting... I find that indeed, when I let Veeam create the temporary snapshot,
and wait for Veeam to apply its "reconfigure virtual machine" tasks on the source proxy VM, presumably hot-adding the VMDKs,

_After_ veeam does its hot adds / mounts, I am able to see the same error raised through power shell, to a query that worked moments before.
http://pastebin.com/W8fwZsbS

If I create a snapshot of my own, or make queryChangedDiskAreas queries during a backup after Veeam creates its snapshot, but before Veeam does its "reconfigure VMs" on the hot-add backup proxy VM, the query made from powershell gets a Change block response, instead of an error.

[Gostev]

The workaround we are planning should not require you to change anything. Thanks for posting support case ID, I will ask to provide you a pilot version of code with workaround in place once it is ready. I already know it is not a simple change (we would have to touch much code there).

[juergens]

Glad to hear, i'm not alone with this problem.
In our environment are failing nearly all vm's with more than one disk. Just one server with 2 disks runs with cbt.

i tried the cbt reset without success and our hw is v7.
i want to cross reference to my support case #5146558 to be included in any development, pilot or workaround scenarios.

if there are details needed, don't hesitate, i'll provide any information i can to resolve this misery.

[simon617]

Looking forward to it! Any idea when the worksaround will be available? 1 week/1month?

[Gostev]

Based on what I know, due to complexity, at least 1 month (that is, if our idea works as expected).
Pretty major changes to the Holy Grail (data mover agent) are required.

[n2it-dk]

Ill add myself to the queue, having client issues aswell... Ive tried suggested solutions with no luck, for my part it allso seems to be related to multiple vmdk´s

Until a fix is ready, what would be recomended ?

Have a great new year.

[Gostev]

The workaround is not to use hot add processing mode. Thanks!

[juergens]

With the new v6, transport mode is globally configured at backup proxy level. As far as i know it was possible in v5 to define the transport level per job.
I don't want to set all my jobs to network mode. is it possible to set specific jobs to network mode anymore?

[foggy]

You should deploy additional proxy, set it to use Network mode and explicitly select this proxy for those specific jobs (at the Storage step of the job wizard). Thanks.