CBT failing on drives other than Hard Disk 1

[mysidia]

Is there any information on when Hot-Add mode might be fixed?
Direct SAN access is not an option in my case, because the storage array is NFS, not VMFS based storage.

CBT works when the backup proxy is configured for 'Network' mode; however,
when the backup proxy is configured for Network mode, the throughput rate is poor, where a full backup is being performed.

It seems like the backup software just needs to be querying the change block data after taking the snapshot
but before performing the Hot Adds.

[Gostev]

Is there any information on when Hot-Add mode might be fixed?

See just 4 posts above

[rhnb]

Just catching up with this thread after holidays (I originally opened this thread)...

I'm a bit confused as I've got this response from tech support, and I just can't believe what I'm reading. The response implies that basically, CBT is unreliable and should be avoided??? (See the last respose to some questions I'd asked).

------
> ◦Information about the snapshot being compared is accessed using SCSI hot add transport mode
I guess this is a bit of a showstopper! So basically is it saying that if we use virtual appliance mode it can't be guaranteed that cbt will work? Does that imply that if I set it to use Network mode that CBT would always work?

- no, it seems like they say that when using Hot-add then CBT may fail.

From this I'm assuming you're seeing a lot of customers with this CBT problem? If not, the question has to be why and what we are doing differently.

I'm also wondering why this seems to have suddenly started happening. Was it our change from Veeam5 to Veeam6?

- No difference, CBT is on VMware ESX vStorage APIs.

Will the update to VDDK 1.2.1 fix this issue? and while you can't give me a date, are we talking a month, 6 months?

- Don't think so. This question should be addressed to VMware.

I'd just like to have some idea so that I can take some action to alleviate my backup window problem. I'll create more proxies and split the jobs into smaller chunks.

Summing up, CBT is a native feature of VMware vStorage APIs. Different backup applications can use it. CBT may fail. E.g., on Friday one of my Customers (we had Webex for 5h 30 min) had an issue - VM restored from a backup loaded in Recovery Mode. Because disk C: was corrupted by CBT, disks of the source VM were backed up(with CBT On) and corrupted C: backed up .vmdk-file, because .ctk - file was incorrect, most likely it was against an incorrect time point(there were 4 snapshots of VMware on the source, it could confuse .ctk-file). But backup job had no errors or warnings, because ESX's API told Veeam that CBT is ok and Veeam used it. Actually, CBT is just a table of changes against the last snapshot of VMware, sometimes it gives us serious problems(corrupted backup for e.g.). I would disable if in Veeam backup job properties to be sure that my backed up disks are ok(this option is in advanced job properties, 4th tab), it will make backup process longer but more safety.
--------

I'm a tad disappointed that tech support didn't read this thread as they seem to be implying it's a VMware problem and I need to contact them. There are obviously others with this issue.

Something I can confirm, at least with our setup, is that predominantly the same VM's fail each time, but occasionally VM's that didn't work, start working and ones that had always worked start failing. I hope there is a fix for this, and soon. It must be something that changed between v5 and v6 - never had the problem before and nothing else in our environment has changed.

[Gostev]

This response makes no sense to me either... whoever wrote this has very poor knowledge of VMware CBT. My favorite remark is about "CTK file" getting "confused" by multiple snapshots presence (when in reality, its own CTK file is created for each and every snapshot). Recommendation to disable CBT, as well as speculations on CBT unreliability are ridiculous as well (and have nothing in common with reality). I have forwarded this to the support management in case they want to do anything about this.

However, to address your point, the issue is actually confirmed to be not specific to the Veeam version (as it can be reproduced without Veeam present). Reading through my response above, we do have a plan for the fix, but due to the amount of changes required it will not be very quick. Meanwhile, my suggestion would be to use alternative transport modes for the affected VMs.

Thanks.

[hunaid]

having same issue only on one vm every day, however repeating the replication process succeeded !!!!!!!!!!!!!

[fedusia]

+1 I have the same issue on one vm every day.

[stevenrodenburg1]

Same here. On a specific VM that has 2 VMDK's, the first VMDK is done with CBT without a hitch. Disk number 2 often, but not always, failes with CBT, just like the FAQ says.

The server VM in question is idle 99% of the day. The application installed on that second VMDK generates very little I/O. Could not find a pattern why 60% of the jobs end with the CBT warning and 40% of it's jobs, both backup- and replication-jobs, run flawlessly.

All other "multi disk" VM's are fine. It's just this one VM.

I too sensed a "CBT should be avoided" kind of reply but that response is unfounded. I mean, V5 never had a problem, then came V6 and the trouble started, then all off a sudden it's VMware's fault....
One should be carefull with such replies but i just read (above) that Anton already took action internally.

We will have to wait for the fix. I have confidence it will be ok. Veeam always came through.

[tsightler]

I too sensed a "CBT should be avoided" kind of reply but that response is unfounded. I mean, V5 never had a problem, then came V6 and the trouble started, then all off a sudden it's VMware's fault....

To be completely fair, it's not quite as simple as "...V5 never had a problem, then came V6 and the trouble started...". Veeam is a product that is heavily dependent on the VMware VDDK. This VDDK is installed as part of the B&R install process. Calls to mount drives and return CBT information are simple API calls to the VDDK API. To support vSphere 5 the VDDK had to be update to the lastest version, a version that is also very new and has seen limited deployment. It's very possible that bugs exist in this new VDDK that are triggered in certain environments. Certainly this has occurred many times with previous versions of the VDDK. In many cases you can "workaround" small issues in the VDDK by calling these APIs differently.

Note that I'm not saying the problem is with the VDDK or Veeam, I honestly don't know, but there are several known issues in the VDDK 5.0 release notes that might have an impact on several of the issues that are being seen with Veeam V6. For example, this note from the VDDK 5.0 "Known Issues" list:

Cleanup of HotAdd disks can affect changed block tracking.
During backup or restore, disks may be HotAdded, and subsequently removed after task completion. This cleanup might remove the change tracking (ctk) file, so that changed block tracking (CBT) could fail. This issue should resolve itself automatically. If it does not, the workaround is to power cycle the virtual machine.

So my overall point is simply that the issue is more complicated than it may at first appear. It's quite possible that V6 has a bug, but it's also possible that the underlying VDDK, on which V6 depends, may have a bug that impacts V6. I have every confidence that, no matter where the problem lies, the development team will eventually find a suitable way to workaround the problem.

[chriswu]

Hi guys

Please can someone tell me how to do:

we can try resetting the CBT for the VM
here is an instruction on how you can reset CBT on the guest VM:
1) power off VM
2) follow the steps shown below:

- You need to set "ctkEnabled" value to false
- You need to set "scsi0:x.ctkEnabled" value to false (false should be set for each disk of the VM in question)

3) power on VM
4) power off VM again (that step is needed to update CTK DataBase)
5) power on VM (from that time you may re-run the job and CTK will be automatically enabled).

I'm not sure how to do step 2.....Where is the settings for this?

thanks!!!!!

[Gargold]

Hi guys

Please can someone tell me how to do:

we can try resetting the CBT for the VM
here is an instruction on how you can reset CBT on the guest VM:
1) power off VM
2) follow the steps shown below:

- You need to set "ctkEnabled" value to false
- You need to set "scsi0:x.ctkEnabled" value to false (false should be set for each disk of the VM in question)

3) power on VM
4) power off VM again (that step is needed to update CTK DataBase)
5) power on VM (from that time you may re-run the job and CTK will be automatically enabled).

I'm not sure how to do step 2.....Where is the settings for this?

thanks!!!!!

Hi chriswu,

In vSphere Client, select VM, go to Advanced Settings.

[chriswu]

thanks gargold for the reply, i had a look: my steps are:

right click vm, click edit settings, click options, in the advanced parts below that there is no setting to tick boxes for ctkenabled or scsi0.

Do you have a screen shot or any other further steps? thanks!

[foggy]

Under Advanced click General and click Configuration Parameters.

[osin_1]

I've got multiple clients running Veeam 6 with patch 2 getting this issue with their backups and replicas. Is this an issue that Veeam are looking at fixing, VMware fixing or are we just having to do the recommended fix of shut down the VM, disable CBT, reboot, reboot to fix this?

I'm not really wanting to have to do this at these sites as theres so many VM's experiencing this issue that it would cause major disruption to shut down the VMs multiple times. If theres a fix that Ive missed please let me know!

[Gostev]

The main issue described in this topic cannot be fixed by resetting CBT... people with different CBT issues are also posting here, which I guess makes it confusing. We are looking on how we can address the "main" issue from our side.

[rawtaz]

I just started having the same issue. Add one to the head count

[Oletho]

See this problem at a customer, case #5170930

Actually it was present in Veeam 5 too, hoped to solve it by upgrading but no dice.

[Gostev]

I am hearing some good news from devs, looks like they were able to work around this... the fix is now in testing.

[rawtaz]

I am hearing some good news from devs, looks like they were able to work around this... the fix is now in testing.

Great!

[jonwhiteman]

+1 I get this error everyday too.

Code: Select all

07/02/2012 04:34:35 :: Cannot use CBT: Soap fault. A specified parameter was not correct. . deviceKeyDetail: '<InvalidArgumentFault xmlns="urn:internalvim25" xsi:type="InvalidArgument"><invalidProperty>deviceKey</invalidProperty></InvalidArgumentFault>', endpoint: ''

Most of our VMs have more than one disk and the second disk is where the data is. It's causing the backup to take a very long time....

I have about 30-40 VMs reporting this error so changing a CBT flag and with multiple power cycles is a bit of a pain. Is there a bit of powershell available to do this (powerdown guest, change flag, power on and reboot guest)?

Any news on when a fix might be released?

[incognito]

[merged]

After over a month back and forth with support I was not able to get much help... in fact part of the problem I ended up having to resolve myself.

The history:

I have 2 virtual machines that absolutely refused to use CBT, both are rather sizeable machines so CBT is imperative to the backup strategy/window.

1.) File server - our primary file server. Initially followed the reset CBT directions and got no results submitted a ticket and after nearly 1 month still no results. Decided to take a weekend to resolve the issues and finally after several CBT reset attempts it finally started working < --- solved.

2.) Exchange 2010 server. Same issues as above except no matter what I do ... it will not use CBT (not fully). Scsi disk 0:0 works with CBT but scsi 0:1 and 0:2 will not!. I've done everything, followed the reset CBT directions even did a V2V of the machine to start with new VMDK and still no luck. I'm at my wits end with the machine ... I need CBT working on it.

After all of this I now have 2 more machines that CBT will not work with and I've tried the CBT reset directions.

Does anybody have anything for me to go on? I will not submit another ticket as I do not have another MONTH to mess around with this.

[jsilva]

Any update on this? I'm experiencing the same issue..

[Oletho]

See this problem at a customer, case #5170930

Actually it was present in Veeam 5 too, hoped to solve it by upgrading but no dice.

Received a patch this morning, and now everything runs as wet feet on a slippery bathroom floor.

Thanks Veeam!

[chriswu]

any chance you got the download link for the patch please!

[Oletho]

As it is not public I must advice you to contact Veeam. Sorry!

My experience is that they react very quickly on support requests.

[chriswu]

I was just thinking do you know if there is a patch section for download on veeam's website? similar to microsoft update

[foggy]

Christopher, there's no special section on the web site, all the available patches are listed in the v6 Known Issues and Patches sticky.

[Oletho]

Yes, and the new Patch3 is not listed there as far as I can see.

[foggy]

It's not officially released yet.

[mysidia]

Any information available on when a fix release might be officially available for this? The person responding to the support request we opened back in December marked our case "resolved" with Veeam, stating it was a known issue, and sending a link straight back to this thread. But this issue is still a major problem and not resolved.

+1 I get this error everyday too.
I have about 30-40 VMs reporting this error so changing a CBT flag and with multiple power cycles is a bit of a pain. Is there a bit of powershell available to do this (powerdown guest, change flag, power on and reboot guest)?
Any news on when a fix might be released?

I would test resetting on CBT on 1 vm first; if that doesn't work, don't bother.
The 1st hard disk issue we were having is not corrected by going through a CBT reset.
If you need to turn off CBT on a larger number of VMs, try something like this:
http://pastebin.com/LGkctswx

Basically, the procedure to turn off CBT is...
get-vm your list of VMs; Issue a config task to turn off CBT;
Put the VM through a stun-unstun by creating/removing a snapshot.

After clearing the CBT flag, and doing the stun-unstun to disable CBT,
verify that there are no *ctk* files in the VMs' directories. Issue Power off and on afterwards.
But i'm sure you can come up with the powershell required to do that task on all the VMs.

[lobo519]

is this going to be a patch 3 item?