CBT failing on drives other than Hard Disk 1

[mysidia]

Is there any information on when Hot-Add mode might be fixed?
Direct SAN access is not an option in my case, because the storage array is NFS, not VMFS based storage.

CBT works when the backup proxy is configured for 'Network' mode; however,
when the backup proxy is configured for Network mode, the throughput rate is poor, where a full backup is being performed.

It seems like the backup software just needs to be querying the change block data after taking the snapshot
but before performing the Hot Adds.

[Gostev]

Is there any information on when Hot-Add mode might be fixed?

See just 4 posts above

[rhnb]

Just catching up with this thread after holidays (I originally opened this thread)...

I'm a bit confused as I've got this response from tech support, and I just can't believe what I'm reading. The response implies that basically, CBT is unreliable and should be avoided??? (See the last respose to some questions I'd asked).

------
> ◦Information about the snapshot being compared is accessed using SCSI hot add transport mode
I guess this is a bit of a showstopper! So basically is it saying that if we use virtual appliance mode it can't be guaranteed that cbt will work? Does that imply that if I set it to use Network mode that CBT would always work?

- no, it seems like they say that when using Hot-add then CBT may fail.

From this I'm assuming you're seeing a lot of customers with this CBT problem? If not, the question has to be why and what we are doing differently.

I'm also wondering why this seems to have suddenly started happening. Was it our change from Veeam5 to Veeam6?

- No difference, CBT is on VMware ESX vStorage APIs.

Will the update to VDDK 1.2.1 fix this issue? and while you can't give me a date, are we talking a month, 6 months?

- Don't think so. This question should be addressed to VMware.

I'd just like to have some idea so that I can take some action to alleviate my backup window problem. I'll create more proxies and split the jobs into smaller chunks.

Summing up, CBT is a native feature of VMware vStorage APIs. Different backup applications can use it. CBT may fail. E.g., on Friday one of my Customers (we had Webex for 5h 30 min) had an issue - VM restored from a backup loaded in Recovery Mode. Because disk C: was corrupted by CBT, disks of the source VM were backed up(with CBT On) and corrupted C: backed up .vmdk-file, because .ctk - file was incorrect, most likely it was against an incorrect time point(there were 4 snapshots of VMware on the source, it could confuse .ctk-file). But backup job had no errors or warnings, because ESX's API told Veeam that CBT is ok and Veeam used it. Actually, CBT is just a table of changes against the last snapshot of VMware, sometimes it gives us serious problems(corrupted backup for e.g.). I would disable if in Veeam backup job properties to be sure that my backed up disks are ok(this option is in advanced job properties, 4th tab), it will make backup process longer but more safety.
--------

I'm a tad disappointed that tech support didn't read this thread as they seem to be implying it's a VMware problem and I need to contact them. There are obviously others with this issue.

Something I can confirm, at least with our setup, is that predominantly the same VM's fail each time, but occasionally VM's that didn't work, start working and ones that had always worked start failing. I hope there is a fix for this, and soon. It must be something that changed between v5 and v6 - never had the problem before and nothing else in our environment has changed.

[Gostev]

This response makes no sense to me either... whoever wrote this has very poor knowledge of VMware CBT. My favorite remark is about "CTK file" getting "confused" by multiple snapshots presence (when in reality, its own CTK file is created for each and every snapshot). Recommendation to disable CBT, as well as speculations on CBT unreliability are ridiculous as well (and have nothing in common with reality). I have forwarded this to the support management in case they want to do anything about this.

However, to address your point, the issue is actually confirmed to be not specific to the Veeam version (as it can be reproduced without Veeam present). Reading through my response above, we do have a plan for the fix, but due to the amount of changes required it will not be very quick. Meanwhile, my suggestion would be to use alternative transport modes for the affected VMs.

Thanks.

[hunaid]

having same issue only on one vm every day, however repeating the replication process succeeded !!!!!!!!!!!!!

[fedusia]

+1 I have the same issue on one vm every day.

[stevenrodenburg1]

Same here. On a specific VM that has 2 VMDK's, the first VMDK is done with CBT without a hitch. Disk number 2 often, but not always, failes with CBT, just like the FAQ says.

The server VM in question is idle 99% of the day. The application installed on that second VMDK generates very little I/O. Could not find a pattern why 60% of the jobs end with the CBT warning and 40% of it's jobs, both backup- and replication-jobs, run flawlessly.

All other "multi disk" VM's are fine. It's just this one VM.

I too sensed a "CBT should be avoided" kind of reply but that response is unfounded. I mean, V5 never had a problem, then came V6 and the trouble started, then all off a sudden it's VMware's fault....
One should be carefull with such replies but i just read (above) that Anton already took action internally.

We will have to wait for the fix. I have confidence it will be ok. Veeam always came through.

[tsightler]

I too sensed a "CBT should be avoided" kind of reply but that response is unfounded. I mean, V5 never had a problem, then came V6 and the trouble started, then all off a sudden it's VMware's fault....

To be completely fair, it's not quite as simple as "...V5 never had a problem, then came V6 and the trouble started...". Veeam is a product that is heavily dependent on the VMware VDDK. This VDDK is installed as part of the B&R install process. Calls to mount drives and return CBT information are simple API calls to the VDDK API. To support vSphere 5 the VDDK had to be update to the lastest version, a version that is also very new and has seen limited deployment. It's very possible that bugs exist in this new VDDK that are triggered in certain environments. Certainly this has occurred many times with previous versions of the VDDK. In many cases you can "workaround" small issues in the VDDK by calling these APIs differently.

Note that I'm not saying the problem is with the VDDK or Veeam, I honestly don't know, but there are several known issues in the VDDK 5.0 release notes that might have an impact on several of the issues that are being seen with Veeam V6. For example, this note from the VDDK 5.0 "Known Issues" list:

Cleanup of HotAdd disks can affect changed block tracking.
During backup or restore, disks may be HotAdded, and subsequently removed after task completion. This cleanup might remove the change tracking (ctk) file, so that changed block tracking (CBT) could fail. This issue should resolve itself automatically. If it does not, the workaround is to power cycle the virtual machine.

So my overall point is simply that the issue is more complicated than it may at first appear. It's quite possible that V6 has a bug, but it's also possible that the underlying VDDK, on which V6 depends, may have a bug that impacts V6. I have every confidence that, no matter where the problem lies, the development team will eventually find a suitable way to workaround the problem.

[chriswu]

Hi guys

Please can someone tell me how to do:

we can try resetting the CBT for the VM
here is an instruction on how you can reset CBT on the guest VM:
1) power off VM
2) follow the steps shown below:

- You need to set "ctkEnabled" value to false
- You need to set "scsi0:x.ctkEnabled" value to false (false should be set for each disk of the VM in question)

3) power on VM
4) power off VM again (that step is needed to update CTK DataBase)
5) power on VM (from that time you may re-run the job and CTK will be automatically enabled).

I'm not sure how to do step 2.....Where is the settings for this?

thanks!!!!!

[Gargold]

Hi guys

Please can someone tell me how to do:

we can try resetting the CBT for the VM
here is an instruction on how you can reset CBT on the guest VM:
1) power off VM
2) follow the steps shown below:

- You need to set "ctkEnabled" value to false
- You need to set "scsi0:x.ctkEnabled" value to false (false should be set for each disk of the VM in question)

3) power on VM
4) power off VM again (that step is needed to update CTK DataBase)
5) power on VM (from that time you may re-run the job and CTK will be automatically enabled).

I'm not sure how to do step 2.....Where is the settings for this?

thanks!!!!!

Hi chriswu,

In vSphere Client, select VM, go to Advanced Settings.

[chriswu]

thanks gargold for the reply, i had a look: my steps are:

right click vm, click edit settings, click options, in the advanced parts below that there is no setting to tick boxes for ctkenabled or scsi0.

Do you have a screen shot or any other further steps? thanks!

[foggy]

Under Advanced click General and click Configuration Parameters.

[osin_1]

I've got multiple clients running Veeam 6 with patch 2 getting this issue with their backups and replicas. Is this an issue that Veeam are looking at fixing, VMware fixing or are we just having to do the recommended fix of shut down the VM, disable CBT, reboot, reboot to fix this?

I'm not really wanting to have to do this at these sites as theres so many VM's experiencing this issue that it would cause major disruption to shut down the VMs multiple times. If theres a fix that Ive missed please let me know!

[Gostev]

The main issue described in this topic cannot be fixed by resetting CBT... people with different CBT issues are also posting here, which I guess makes it confusing. We are looking on how we can address the "main" issue from our side.

[rawtaz]

I just started having the same issue. Add one to the head count