Discussions specific to the VMware vSphere hypervisor
mysidia
Influencer
Posts: 14
Liked: 3 times
Joined: Mar 18, 2010 5:07 pm
Full Name: James Hess
Location: Louisiana
Contact:

Re: CBT failing on drives other than Hard Disk 1

Post by mysidia » Dec 30, 2011 4:08 pm

Is there any information on when Hot-Add mode might be fixed?
Direct SAN access is not an option in my case, because the storage array is NFS, not VMFS based storage.

CBT works when the backup proxy is configured for 'Network' mode; however,
when the backup proxy is configured for Network mode, the throughput rate is poor, where a full backup is being performed.

It seems like the backup software just needs to be querying the change block data after taking the snapshot
but before performing the Hot Adds.

Gostev
SVP, Product Management
Posts: 24984
Liked: 3629 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: CBT failing on drives other than Hard Disk 1

Post by Gostev » Dec 30, 2011 4:25 pm

mysidia wrote:Is there any information on when Hot-Add mode might be fixed?
See just 4 posts above ;)

rhnb
Enthusiast
Posts: 81
Liked: 2 times
Joined: Jan 27, 2010 2:25 pm
Full Name: Allan Nelson
Contact:

Re: CBT failing on drives other than Hard Disk 1

Post by rhnb » Jan 03, 2012 7:53 pm

Just catching up with this thread after holidays (I originally opened this thread)...

I'm a bit confused as I've got this response from tech support, and I just can't believe what I'm reading. The response implies that basically, CBT is unreliable and should be avoided??? (See the last respose to some questions I'd asked).

------
> ◦Information about the snapshot being compared is accessed using SCSI hot add transport mode
I guess this is a bit of a showstopper! So basically is it saying that if we use virtual appliance mode it can't be guaranteed that cbt will work? Does that imply that if I set it to use Network mode that CBT would always work?

- no, it seems like they say that when using Hot-add then CBT may fail.

From this I'm assuming you're seeing a lot of customers with this CBT problem? If not, the question has to be why and what we are doing differently.

I'm also wondering why this seems to have suddenly started happening. Was it our change from Veeam5 to Veeam6?

- No difference, CBT is on VMware ESX vStorage APIs.

Will the update to VDDK 1.2.1 fix this issue? and while you can't give me a date, are we talking a month, 6 months?

- Don't think so. This question should be addressed to VMware.

I'd just like to have some idea so that I can take some action to alleviate my backup window problem. I'll create more proxies and split the jobs into smaller chunks.

Summing up, CBT is a native feature of VMware vStorage APIs. Different backup applications can use it. CBT may fail. E.g., on Friday one of my Customers (we had Webex for 5h 30 min) had an issue - VM restored from a backup loaded in Recovery Mode. Because disk C: was corrupted by CBT, disks of the source VM were backed up(with CBT On) and corrupted C: backed up .vmdk-file, because .ctk - file was incorrect, most likely it was against an incorrect time point(there were 4 snapshots of VMware on the source, it could confuse .ctk-file). But backup job had no errors or warnings, because ESX's API told Veeam that CBT is ok and Veeam used it. Actually, CBT is just a table of changes against the last snapshot of VMware, sometimes it gives us serious problems(corrupted backup for e.g.). I would disable if in Veeam backup job properties to be sure that my backed up disks are ok(this option is in advanced job properties, 4th tab), it will make backup process longer but more safety.
--------

I'm a tad disappointed that tech support didn't read this thread as they seem to be implying it's a VMware problem and I need to contact them. There are obviously others with this issue.

Something I can confirm, at least with our setup, is that predominantly the same VM's fail each time, but occasionally VM's that didn't work, start working and ones that had always worked start failing. I hope there is a fix for this, and soon. It must be something that changed between v5 and v6 - never had the problem before and nothing else in our environment has changed.

Gostev
SVP, Product Management
Posts: 24984
Liked: 3629 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: CBT failing on drives other than Hard Disk 1

Post by Gostev » Jan 03, 2012 8:02 pm

This response makes no sense to me either... whoever wrote this has very poor knowledge of VMware CBT. My favorite remark is about "CTK file" getting "confused" by multiple snapshots presence (when in reality, its own CTK file is created for each and every snapshot). Recommendation to disable CBT, as well as speculations on CBT unreliability are ridiculous as well (and have nothing in common with reality). I have forwarded this to the support management in case they want to do anything about this.

However, to address your point, the issue is actually confirmed to be not specific to the Veeam version (as it can be reproduced without Veeam present). Reading through my response above, we do have a plan for the fix, but due to the amount of changes required it will not be very quick. Meanwhile, my suggestion would be to use alternative transport modes for the affected VMs.

Thanks.

hunaid
Lurker
Posts: 1
Liked: never
Joined: Jan 06, 2012 12:08 pm
Full Name: Hunaid Haroon Al Qureshi
Contact:

Re: CBT failing on drives other than Hard Disk 1

Post by hunaid » Jan 06, 2012 12:11 pm

having same issue only on one vm every day, however repeating the replication process succeeded !!!!!!!!!!!!!

fedusia
Lurker
Posts: 1
Liked: never
Joined: Nov 23, 2011 6:12 am
Full Name: Dmitry S. Fedorov
Contact:

Re: CBT failing on drives other than Hard Disk 1

Post by fedusia » Jan 13, 2012 10:45 am

+1 I have the same issue on one vm every day.

stevenrodenburg1
Expert
Posts: 127
Liked: 19 times
Joined: May 31, 2011 9:11 am
Full Name: Steven Rodenburg
Location: Switzerland
Contact:

Re: CBT failing on drives other than Hard Disk 1

Post by stevenrodenburg1 » Jan 14, 2012 9:28 am

Same here. On a specific VM that has 2 VMDK's, the first VMDK is done with CBT without a hitch. Disk number 2 often, but not always, failes with CBT, just like the FAQ says.

The server VM in question is idle 99% of the day. The application installed on that second VMDK generates very little I/O. Could not find a pattern why 60% of the jobs end with the CBT warning and 40% of it's jobs, both backup- and replication-jobs, run flawlessly.

All other "multi disk" VM's are fine. It's just this one VM.

I too sensed a "CBT should be avoided" kind of reply but that response is unfounded. I mean, V5 never had a problem, then came V6 and the trouble started, then all off a sudden it's VMware's fault....
One should be carefull with such replies but i just read (above) that Anton already took action internally.

We will have to wait for the fix. I have confidence it will be ok. Veeam always came through.

tsightler
VP, Product Management
Posts: 5454
Liked: 2272 times
Joined: Jun 05, 2009 12:57 pm
Full Name: Tom Sightler
Contact:

Re: CBT failing on drives other than Hard Disk 1

Post by tsightler » Jan 14, 2012 6:17 pm

stevenrodenburg1 wrote:I too sensed a "CBT should be avoided" kind of reply but that response is unfounded. I mean, V5 never had a problem, then came V6 and the trouble started, then all off a sudden it's VMware's fault....
To be completely fair, it's not quite as simple as "...V5 never had a problem, then came V6 and the trouble started...". Veeam is a product that is heavily dependent on the VMware VDDK. This VDDK is installed as part of the B&R install process. Calls to mount drives and return CBT information are simple API calls to the VDDK API. To support vSphere 5 the VDDK had to be update to the lastest version, a version that is also very new and has seen limited deployment. It's very possible that bugs exist in this new VDDK that are triggered in certain environments. Certainly this has occurred many times with previous versions of the VDDK. In many cases you can "workaround" small issues in the VDDK by calling these APIs differently.

Note that I'm not saying the problem is with the VDDK or Veeam, I honestly don't know, but there are several known issues in the VDDK 5.0 release notes that might have an impact on several of the issues that are being seen with Veeam V6. For example, this note from the VDDK 5.0 "Known Issues" list:
Cleanup of HotAdd disks can affect changed block tracking.
During backup or restore, disks may be HotAdded, and subsequently removed after task completion. This cleanup might remove the change tracking (ctk) file, so that changed block tracking (CBT) could fail. This issue should resolve itself automatically. If it does not, the workaround is to power cycle the virtual machine.
So my overall point is simply that the issue is more complicated than it may at first appear. It's quite possible that V6 has a bug, but it's also possible that the underlying VDDK, on which V6 depends, may have a bug that impacts V6. I have every confidence that, no matter where the problem lies, the development team will eventually find a suitable way to workaround the problem.

chriswu
Novice
Posts: 6
Liked: never
Joined: Jan 20, 2012 9:32 am
Full Name: Christopher Wu
Contact:

Re: CBT failing on drives other than Hard Disk 1

Post by chriswu » Jan 20, 2012 9:54 am

Hi guys

Please can someone tell me how to do:

we can try resetting the CBT for the VM
here is an instruction on how you can reset CBT on the guest VM:
1) power off VM
2) follow the steps shown below:

- You need to set "ctkEnabled" value to false
- You need to set "scsi0:x.ctkEnabled" value to false (false should be set for each disk of the VM in question)

3) power on VM
4) power off VM again (that step is needed to update CTK DataBase)
5) power on VM (from that time you may re-run the job and CTK will be automatically enabled).

I'm not sure how to do step 2.....Where is the settings for this?

thanks!!!!!

Gargold
Lurker
Posts: 1
Liked: never
Joined: Jan 05, 2012 4:47 pm
Contact:

Re: CBT failing on drives other than Hard Disk 1

Post by Gargold » Jan 20, 2012 11:17 am

chriswu wrote:Hi guys

Please can someone tell me how to do:

we can try resetting the CBT for the VM
here is an instruction on how you can reset CBT on the guest VM:
1) power off VM
2) follow the steps shown below:

- You need to set "ctkEnabled" value to false
- You need to set "scsi0:x.ctkEnabled" value to false (false should be set for each disk of the VM in question)

3) power on VM
4) power off VM again (that step is needed to update CTK DataBase)
5) power on VM (from that time you may re-run the job and CTK will be automatically enabled).

I'm not sure how to do step 2.....Where is the settings for this?

thanks!!!!!
Hi chriswu,

In vSphere Client, select VM, go to Advanced Settings.

chriswu
Novice
Posts: 6
Liked: never
Joined: Jan 20, 2012 9:32 am
Full Name: Christopher Wu
Contact:

Re: CBT failing on drives other than Hard Disk 1

Post by chriswu » Jan 20, 2012 11:49 am

thanks gargold for the reply, i had a look: my steps are:

right click vm, click edit settings, click options, in the advanced parts below that there is no setting to tick boxes for ctkenabled or scsi0.

Do you have a screen shot or any other further steps? thanks!

foggy
Veeam Software
Posts: 18390
Liked: 1581 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: CBT failing on drives other than Hard Disk 1

Post by foggy » Jan 20, 2012 12:31 pm

Under Advanced click General and click Configuration Parameters.

osin_1
Service Provider
Posts: 6
Liked: never
Joined: Nov 02, 2009 8:28 pm
Full Name: Bruiser
Contact:

Re: CBT failing on drives other than Hard Disk 1

Post by osin_1 » Jan 25, 2012 6:06 am

I've got multiple clients running Veeam 6 with patch 2 getting this issue with their backups and replicas. Is this an issue that Veeam are looking at fixing, VMware fixing or are we just having to do the recommended fix of shut down the VM, disable CBT, reboot, reboot to fix this?

I'm not really wanting to have to do this at these sites as theres so many VM's experiencing this issue that it would cause major disruption to shut down the VMs multiple times. If theres a fix that Ive missed please let me know!

Gostev
SVP, Product Management
Posts: 24984
Liked: 3629 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: CBT failing on drives other than Hard Disk 1

Post by Gostev » Jan 25, 2012 9:27 am

The main issue described in this topic cannot be fixed by resetting CBT... people with different CBT issues are also posting here, which I guess makes it confusing. We are looking on how we can address the "main" issue from our side.

rawtaz
Enthusiast
Posts: 85
Liked: 15 times
Joined: Jan 27, 2012 4:42 pm
Contact:

Re: CBT failing on drives other than Hard Disk 1

Post by rawtaz » Feb 02, 2012 4:01 pm

I just started having the same issue. Add one to the head count :-)

Oletho
Enthusiast
Posts: 66
Liked: 2 times
Joined: Sep 17, 2010 4:37 am
Full Name: Ole Thomsen
Contact:

Re: CBT failing on drives other than Hard Disk 1

Post by Oletho » Feb 05, 2012 8:17 am

See this problem at a customer, case #5170930

Actually it was present in Veeam 5 too, hoped to solve it by upgrading but no dice.

Gostev
SVP, Product Management
Posts: 24984
Liked: 3629 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: CBT failing on drives other than Hard Disk 1

Post by Gostev » Feb 05, 2012 2:33 pm

I am hearing some good news from devs, looks like they were able to work around this... the fix is now in testing.

rawtaz
Enthusiast
Posts: 85
Liked: 15 times
Joined: Jan 27, 2012 4:42 pm
Contact:

Re: CBT failing on drives other than Hard Disk 1

Post by rawtaz » Feb 05, 2012 2:37 pm

Gostev wrote:I am hearing some good news from devs, looks like they were able to work around this... the fix is now in testing.
Great! :D

jonwhiteman
Lurker
Posts: 2
Liked: never
Joined: Feb 07, 2012 12:12 pm
Full Name: Jon Whiteman
Contact:

Re: CBT failing on drives other than Hard Disk 1

Post by jonwhiteman » Feb 07, 2012 12:36 pm

+1 I get this error everyday too.

Code: Select all

07/02/2012 04:34:35 :: Cannot use CBT: Soap fault. A specified parameter was not correct. . deviceKeyDetail: '<InvalidArgumentFault xmlns="urn:internalvim25" xsi:type="InvalidArgument"><invalidProperty>deviceKey</invalidProperty></InvalidArgumentFault>', endpoint: ''
Most of our VMs have more than one disk and the second disk is where the data is. It's causing the backup to take a very long time....

I have about 30-40 VMs reporting this error so changing a CBT flag and with multiple power cycles is a bit of a pain. Is there a bit of powershell available to do this (powerdown guest, change flag, power on and reboot guest)?

Any news on when a fix might be released?

incognito
Novice
Posts: 5
Liked: never
Joined: Feb 07, 2012 2:47 pm
Full Name: NATE
Contact:

cbt problems - starting to get frustarted

Post by incognito » Feb 07, 2012 2:58 pm

[merged]

After over a month back and forth with support I was not able to get much help... in fact part of the problem I ended up having to resolve myself.

The history:

I have 2 virtual machines that absolutely refused to use CBT, both are rather sizeable machines so CBT is imperative to the backup strategy/window.

1.) File server - our primary file server. Initially followed the reset CBT directions and got no results submitted a ticket and after nearly 1 month still no results. Decided to take a weekend to resolve the issues and finally after several CBT reset attempts it finally started working < --- solved.

2.) Exchange 2010 server. Same issues as above except no matter what I do ... it will not use CBT (not fully). Scsi disk 0:0 works with CBT but scsi 0:1 and 0:2 will not!. I've done everything, followed the reset CBT directions even did a V2V of the machine to start with new VMDK and still no luck. I'm at my wits end with the machine ... I need CBT working on it.

After all of this I now have 2 more machines that CBT will not work with and I've tried the CBT reset directions.

Does anybody have anything for me to go on? I will not submit another ticket as I do not have another MONTH to mess around with this.

jsilva
Lurker
Posts: 2
Liked: never
Joined: Feb 10, 2012 11:33 am
Full Name: Jason Silva
Contact:

Re: CBT failing on drives other than Hard Disk 1

Post by jsilva » Feb 10, 2012 11:52 am

Any update on this? I'm experiencing the same issue..

Oletho
Enthusiast
Posts: 66
Liked: 2 times
Joined: Sep 17, 2010 4:37 am
Full Name: Ole Thomsen
Contact:

Re: CBT failing on drives other than Hard Disk 1 [Solved]

Post by Oletho » Feb 10, 2012 1:36 pm

Oletho wrote:See this problem at a customer, case #5170930

Actually it was present in Veeam 5 too, hoped to solve it by upgrading but no dice.
Received a patch this morning, and now everything runs as wet feet on a slippery bathroom floor.

Thanks Veeam!

chriswu
Novice
Posts: 6
Liked: never
Joined: Jan 20, 2012 9:32 am
Full Name: Christopher Wu
Contact:

Re: CBT failing on drives other than Hard Disk 1

Post by chriswu » Feb 10, 2012 2:09 pm

any chance you got the download link for the patch please!

Oletho
Enthusiast
Posts: 66
Liked: 2 times
Joined: Sep 17, 2010 4:37 am
Full Name: Ole Thomsen
Contact:

Re: CBT failing on drives other than Hard Disk 1

Post by Oletho » Feb 10, 2012 2:25 pm

As it is not public I must advice you to contact Veeam. Sorry!

My experience is that they react very quickly on support requests.

chriswu
Novice
Posts: 6
Liked: never
Joined: Jan 20, 2012 9:32 am
Full Name: Christopher Wu
Contact:

Re: CBT failing on drives other than Hard Disk 1

Post by chriswu » Feb 10, 2012 4:02 pm

I was just thinking do you know if there is a patch section for download on veeam's website? similar to microsoft update

foggy
Veeam Software
Posts: 18390
Liked: 1581 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: CBT failing on drives other than Hard Disk 1

Post by foggy » Feb 10, 2012 4:13 pm

Christopher, there's no special section on the web site, all the available patches are listed in the v6 Known Issues and Patches sticky.

Oletho
Enthusiast
Posts: 66
Liked: 2 times
Joined: Sep 17, 2010 4:37 am
Full Name: Ole Thomsen
Contact:

Re: CBT failing on drives other than Hard Disk 1

Post by Oletho » Feb 10, 2012 5:29 pm

Yes, and the new Patch3 is not listed there as far as I can see.

foggy
Veeam Software
Posts: 18390
Liked: 1581 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: CBT failing on drives other than Hard Disk 1

Post by foggy » Feb 10, 2012 8:39 pm

It's not officially released yet.

mysidia
Influencer
Posts: 14
Liked: 3 times
Joined: Mar 18, 2010 5:07 pm
Full Name: James Hess
Location: Louisiana
Contact:

Re: CBT failing on drives other than Hard Disk 1

Post by mysidia » Feb 13, 2012 6:19 pm

Any information available on when a fix release might be officially available for this? The person responding to the support request we opened back in December marked our case "resolved" with Veeam, stating it was a known issue, and sending a link straight back to this thread. But this issue is still a major problem and not resolved.
jonwhiteman wrote:+1 I get this error everyday too.
I have about 30-40 VMs reporting this error so changing a CBT flag and with multiple power cycles is a bit of a pain. Is there a bit of powershell available to do this (powerdown guest, change flag, power on and reboot guest)?
Any news on when a fix might be released?
I would test resetting on CBT on 1 vm first; if that doesn't work, don't bother.
The 1st hard disk issue we were having is not corrected by going through a CBT reset.
If you need to turn off CBT on a larger number of VMs, try something like this:
http://pastebin.com/LGkctswx

Basically, the procedure to turn off CBT is...
get-vm your list of VMs; Issue a config task to turn off CBT;
Put the VM through a stun-unstun by creating/removing a snapshot.

After clearing the CBT flag, and doing the stun-unstun to disable CBT,
verify that there are no *ctk* files in the VMs' directories. Issue Power off and on afterwards.
But i'm sure you can come up with the powershell required to do that task on all the VMs.

lobo519
Expert
Posts: 300
Liked: 34 times
Joined: Sep 29, 2010 3:37 pm
Contact:

Re: CBT failing on drives other than Hard Disk 1

Post by lobo519 » Feb 13, 2012 7:08 pm

is this going to be a patch 3 item?

Post Reply

Who is online

Users browsing this forum: Yrite~ and 20 guests