Host-based backup of VMware vSphere VMs.
Frosty
Expert
Posts: 200
Liked: 43 times
Joined: Dec 22, 2009 9:00 pm
Full Name: Stephen Frost
Contact:

v6 "Cannot use CBT. Soap fault" ... fixed

Post by Frosty »

I completed my upgrade to v6 yesterday. All went well. Ran our usual backups last night and all completed successfully except for one which had a Warning as follows:

Cannot use CBT: Soap fault. A specified parameter was not correct. . deviceKeyDetail: '<InvalidArgumentFault xmlns="urn:internalvim25" xsi:type="InvalidArgument"><invalidProperty>deviceKey</invalidProperty></InvalidArgumentFault>', endpoint: ''

It just so happens that this is one of two VMs which is normally powered off. It also happens to be a VM with two (2) VMDKs (D: drive used for SQL data).

After checking through the forums here, I tried the following:

* in vCentre took a snaphot
* from Snapshot Manager did a "Delete All" of the snapshots for that VM
* booted VM
* logged in, waiting 1/2 a minute, restarted VM
* waited a couple of minutes, ran the Veeam backup ... Successful completion
* shutdown the VM, ran the Veeam backup again ... Successful completion

Not exactly sure whether it was the "Delete All" of the snapshots, or the boot and reboot which fixed it; main thing is that its OK now.
Gostev
Chief Product Officer
Posts: 31459
Liked: 6648 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: v6 "Cannot use CBT. Soap fault" ... fixed

Post by Gostev »

Yes, CBT may have some issues with powered off VMs.
Frosty
Expert
Posts: 200
Liked: 43 times
Joined: Dec 22, 2009 9:00 pm
Full Name: Stephen Frost
Contact:

Re: v6 "Cannot use CBT. Soap fault" ... fixed

Post by Frosty »

The warning is back again this morning. Wasn't there yesterday. Do I need to lodge a support ticket to get this checked out? Or do I just ignore it, even though this wasn't an issue with v5?
foggy
Veeam Software
Posts: 21069
Liked: 2115 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: v6 "Cannot use CBT. Soap fault" ... fixed

Post by foggy »

Stephen, have you installed patch #3 during the upgrade to v6? There is a known issue regarding this error, please review.
deevie
Influencer
Posts: 11
Liked: 1 time
Joined: Jan 22, 2012 8:58 pm
Contact:

Re: v6 "Cannot use CBT. Soap fault" ... fixed

Post by deevie »

I also have issues with CBT on one certain job which contains more than one vmdk. Someday CBT works fine, the other day I get an error. The day after that it's fine again.
Frosty
Expert
Posts: 200
Liked: 43 times
Joined: Dec 22, 2009 9:00 pm
Full Name: Stephen Frost
Contact:

Re: v6 "Cannot use CBT. Soap fault" ... fixed

Post by Frosty »

Hi foggy,
Yes, I installed Patch 3 right after upgrading from v5 to v6 ...
I might reboot the VBR server today and then see if the problem is still there next week.
Should I re-install Patch 3 (just in case something went astray first time)?
Frosty.
Frosty
Expert
Posts: 200
Liked: 43 times
Joined: Dec 22, 2009 9:00 pm
Full Name: Stephen Frost
Contact:

Re: v6 "Cannot use CBT. Soap fault" ... fixed

Post by Frosty »

Just rebooted the server, ran the job, got the Warning message again:

30/03/2012 11:05:05 AM :: Job finished with warning at 30/03/2012 11:05:05 AM
*** and ***
Cannot use CBT: Soap fault. A specified parameter was not correct. . deviceKeyDetail: '<InvalidArgumentFault xmlns="urn:internalvim25" xsi:type="InvalidArgument"><invalidProperty>deviceKey</invalidProperty></InvalidArgumentFault>', endpoint: ''
Vitaliy S.
VP, Product Management
Posts: 27055
Liked: 2710 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: v6 "Cannot use CBT. Soap fault" ... fixed

Post by Vitaliy S. »

I believe it's better to involve our support team in the investigation.
gcadavid
Novice
Posts: 4
Liked: never
Joined: Mar 30, 2012 8:18 am
Full Name: guillermo cadavid garcia
Contact:

Re: v6 "Cannot use CBT. Soap fault" ... fixed

Post by gcadavid »

Hello,

I have the same problem with all my VMs on a Veeam backup 6. I try to enable the ctkEnabled on the VM settings and now it works fine. I have this in a lab environment but sincerly I don't see the backup copy faster than disable the ctkEnabled parameter.

Regards,
Gostev
Chief Product Officer
Posts: 31459
Liked: 6648 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: v6 "Cannot use CBT. Soap fault" ... fixed

Post by Gostev »

This likely means that your have a really big bottleneck somewhere else in your backup infrastructure... the VM-level bottleneck statistics should clearly show what it is.
stevenrodenburg1
Expert
Posts: 135
Liked: 20 times
Joined: May 31, 2011 9:11 am
Full Name: Steven Rodenburg
Location: Switzerland
Contact:

Re: v6 "Cannot use CBT. Soap fault" ... fixed

Post by stevenrodenburg1 » 1 person likes this post

Just sharing:


If a vSphere Host (ESXi) crashes, all of it's VM's will have CBT problems, causing Veeam to finish their jobs succesfully but with (cbt)warnings. One job later however, all is fine again.

We had a complete powerloss over here, causing the entire datacenter (storage and vSphere) to blackout.

After everything was up and running again, the backup jobs that night **all** ended with warning complaining about CBT. No VM was without CBT problems.

The next night, all the jobs were OK again. I guess CBT repairs itself (or is repaired by vStorage API aware software like Veeam).
Gostev
Chief Product Officer
Posts: 31459
Liked: 6648 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: v6 "Cannot use CBT. Soap fault" ... fixed

Post by Gostev »

That's right.

Nothing is "repaired", just new consistent CBT point of reference is created. Because CBT data corrupts during ESXi crash, the first run after crash detects CBT data corruption, and uses legacy "full scan" incremental (physically reads every disk block of source VM, comparing each block to what is stored in the backup). This type of run will of course take much longer than CBT run, but still less time than a full backup. After that, once we make sure what is on disk is exactly what we have in the backup, we start using CBT to track the changes again (so the following increments use CBT).
richardw
Enthusiast
Posts: 29
Liked: never
Joined: Jan 27, 2010 1:03 pm
Full Name: Richard Wallis
Contact:

Re: v6 "Cannot use CBT. Soap fault" ... fixed

Post by richardw »

Hi there,

Have you tried re-creating the job again?
Fox54
Enthusiast
Posts: 29
Liked: never
Joined: May 12, 2009 1:47 am
Contact:

Error messages Cannot use cBT

Post by Fox54 »

[merged]

Hi,

We just updated to Veeam 6.0 (running 6.0.0.181) patch3 applied

I have 3 Vmware servers (2 Vsphere 4.1 and 1 Vsphere5). All the Replication or backup jobs for the VM running on the Vsphere 5 server end with this warning

Cannot use CBT: Soap fault. A specified parameter was not correct. . deviceKeyDetail: '<InvalidArgumentFault xmlns="urn:internalvim25" xsi:type="InvalidArgument"><invalidProperty>deviceKey</invalidProperty></InvalidArgumentFault>', endpoint: ''

All the other jobs for the VM on the others vSphere 4.1 servers run fine. All the jobs are not Legacy jobs. We recreate all the jobs from scratch after the 6.x update

I looked into the forums but nothing seems to apply to my situation

Any idea ?
ddockter
Enthusiast
Posts: 60
Liked: never
Joined: Sep 25, 2010 2:09 pm
Full Name: Doug Dockter

Cannot use CBT: Soap fault

Post by ddockter »

[merged]

I'm running v6.0 with patch 3 applied. Four times over the last four months all of my backup jobs have received the below message for all the VMs in the backup. The previous day all backups ran OK and the following day all backups ran OK (to be determined if that is true tonight). Any idea why this would occur?

6/5/2012 7:00:52 PM :: Hot add is not supported for this disk, failing over to network mode...
6/5/2012 7:00:56 PM :: Cannot use CBT: Soap fault. A specified parameter was not correct. . deviceKeyDetail: '<InvalidArgumentFault xmlns="urn:internalvim25" xsi:type="InvalidArgument"><invalidProperty>deviceKey</invalidProperty></InvalidArgumentFault>', endpoint: ''
marius roma
Veteran
Posts: 459
Liked: 5 times
Joined: Feb 01, 2012 12:04 pm
Full Name: Mario
Contact:

Cannot use CBT: Soap fault. A specified parameter ...

Post by marius roma »

[merged]

After replicating some VMs from one site to another I attempted to backup the replicas before powering them on and I got a warning saying that "Cannot use CBT: Soap fault. A specified parameter was not correct. . deviceKeyDetail: '<InvalidArgumentFault xmlns="urn:internalvim25" xsi:type="InvalidArgument"><invalidProperty>deviceKey</invalidProperty></InvalidArgumentFault>', endpoint: ''"
The problem looks very similar to what described on KB1113 (see http://www.veeam.com/KB1113 ) so I presume I already have the solution :D
I still have some troubles:
- Given I have the same VM in the source site with the same parameters, why the warning occurs only in the destination site?
- Can changing the "ctkEnabled" value to false produce any kind of problems in the VMware environment?
- Given it is the only warning in the backup job, can I trust the backup to be usable in case of need despite the warning?
- As I see the "ctkEnabled" value in the configuration parameters but no "scsi0:x:ctkEnabled" parameter, shoud I add it and set it to false?
Regards
marius
Vitaliy S.
VP, Product Management
Posts: 27055
Liked: 2710 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: v6 "Cannot use CBT. Soap fault" ... fixed

Post by Vitaliy S. »

marius roma wrote:- Given I have the same VM in the source site with the same parameters, why the warning occurs only in the destination site?
Please take a look at this topic for more info: Veeam Backup of Veeam Replication VMs
marius roma wrote:Can changing the "ctkEnabled" value to false produce any kind of problems in the VMware environment?
No.
marius roma wrote:Given it is the only warning in the backup job, can I trust the backup to be usable in case of need despite the warning?
Yes.
cffit
Veteran
Posts: 338
Liked: 35 times
Joined: Jan 20, 2012 2:36 pm
Full Name: Christensen Farms
Contact:

Re: v6 "Cannot use CBT. Soap fault" ... fixed

Post by cffit »

I am on v6.1 and working great, but I notice that in the following scenario I get this error too:

1. Shut down VM
2. Take Snapshot of VM
3. Boot VM
4. Eventually revert back to Snapshot I took while powered down
5. Power on VM
6. Delete snapshot files

The first night I'll get a failed backup with this error, but then the next night it runs ok. I do the above steps all in one day, so there is no snapshot overnight when the backups run. I found this to be the case with v6 as well, not jsut v6.1.
foggy
Veeam Software
Posts: 21069
Liked: 2115 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: v6 "Cannot use CBT. Soap fault" ... fixed

Post by foggy »

Looks like the described manipulations result in CBT data corruption so it cannot be used for reference to the changed blocks. Then, after the first job run, the consistent CBT data is collected. The strange thing is that you're saying that the backup has failed - is it really so? This message should be logged as a warning resulting in a longer but still successful backup.
cffit
Veteran
Posts: 338
Liked: 35 times
Joined: Jan 20, 2012 2:36 pm
Full Name: Christensen Farms
Contact:

Re: v6 "Cannot use CBT. Soap fault" ... fixed

Post by cffit »

I apologize, it is a warning, but I assumed by the wording that the job fails since it can't use CBT.
foggy
Veeam Software
Posts: 21069
Liked: 2115 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: v6 "Cannot use CBT. Soap fault" ... fixed

Post by foggy »

The job does not fail in case it can't use VMware CBT. Our own proprietary change tracking mechanism is used in this case (it is much longer as needs to compare each block of the source VM to the one stored in the backup).
PK_GAA
Influencer
Posts: 11
Liked: never
Joined: Jun 14, 2012 1:51 pm
Full Name: Peter K.
Contact:

Re: v6 "Cannot use CBT. Soap fault" ... fixed

Post by PK_GAA »

We are suffering the same error for a few days now. I cannot tell, why it started occuring, but the fact, that it occurs is a big problem. The problem occurs when we backup our storage server, which is about 800GB. The backup job contains 7 VMs and normaly the whole job would be done in about 5-7 hours. Now with the disabled CBT the job is running for more than 17 hours just to backup the storage VM. so all the other VMs don't get backuped in that time. Plus the backup keeps the replikation job from replkating the storage server, because the VM is in use. The waiting replikation job keeps active and waiting, so no new replikation job (the whole job contains 5 VMs) can be startet, which means the other VMs won't be replikated. Can someone help with this topic? Support is informed.

i have one idea left: about a week ago we had to enlarge the virtual disk of the storage server. is it possible, this causes the CBT probem?

Some information about our system:
* 3 esxi hosts. 2 of them in HA-mode, the third is the backup host directly connected the the iscsi-backup-storage.
* on the production-system we run 10 VMs (including 1 storage-server, 1 sql-server, 1 vcenter client (which is also the veeam b&r server), 1 domain controller)
* Productive and backup hardware is staged in two buildings, connected via straight direct link wireless LAN (connection is monitored and 99.9% of time working)
* backup proxys are: the VBR server itself, the backup vcenter client, 1 VM at backup-side, 1 VM at prodictive-side, 1 physical machine. (would it be better to have more physical machines as proxys?)
Vitaliy S.
VP, Product Management
Posts: 27055
Liked: 2710 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: v6 "Cannot use CBT. Soap fault" ... fixed

Post by Vitaliy S. »

PK_GAA wrote:i have one idea left: about a week ago we had to enlarge the virtual disk of the storage server. is it possible, this causes the CBT probem?
Yes, looks like it might be the root cause of your CBT problem. Could you please tell me if you've started to see CBT failures right after extending virtual disk size? Have you tried resetting VMware CBT manually after that?
PK_GAA
Influencer
Posts: 11
Liked: never
Joined: Jun 14, 2012 1:51 pm
Full Name: Peter K.
Contact:

Re: v6 "Cannot use CBT. Soap fault" ... fixed

Post by PK_GAA »

Vitaliy S. wrote:Could you please tell me if you've started to see CBT failures right after extending virtual disk size?
yes and no.
Replikation detected the changed disk-size at Aug 8.:

Code: Select all

02.08.2012 13:15:40 :: VM disk size was changed since last sync, deleting all restore points
the next backup at aug 8, 8pm worked out normal. no errors.

the next error at a backup happened the next day when the job wrote a full backup:

Code: Select all

03.08.2012 23:26:40 :: Hard Disk 1 (40,0 GB)
03.08.2012 23:44:06 :: Hard Disk 2 (250,0 GB)
04.08.2012 03:02:22 :: Removing snapshot
04.08.2012 03:04:49 :: Swap file blocks skipped: 4,0 GB
04.08.2012 03:04:54 :: Error: Client error: Eine Ausnahme vom Typ "Veeam.Backup.AgentProvider.AgentClosedException" wurde ausgelöst.
04.08.2012 03:04:54 :: Busy: Source 6% > Proxy 12% > Network 96% > Target 10%
04.08.2012 03:04:54 :: Primary bottleneck: Network
04.08.2012 03:04:54 :: Processing finished with errors at 04.08.2012 03:04:54
first retry finished with the following errors:

Code: Select all

04.08.2012 07:04:22 :: Hard Disk 1 (40,0 GB)
04.08.2012 07:28:55 :: Hard Disk 2 (250,0 GB)
04.08.2012 13:39:53 :: Hard Disk 3 (251,0 GB)
05.08.2012 00:01:35 :: Removing snapshot
05.08.2012 00:05:55 :: Swap file blocks skipped: 4,0 GB
05.08.2012 00:06:04 :: Error: Client error: End of file
Unable to retrieve next block transmission command. Number of already processed blocks: [564883].
05.08.2012 00:06:04 :: Busy: Source 3% > Proxy 17% > Network 90% > Target 28%
05.08.2012 00:06:04 :: Primary bottleneck: Network
05.08.2012 00:06:05 :: Processing finished with errors at 05.08.2012 00:06:04
the second retry was taking almost a day and it also failed.
since then the CBT warning keeps occuring.

my next step is write new full backup. maybe this will fix the problem.
Vitaliy S. wrote:Have you tried resetting VMware CBT manually after that?
no, not yet.
satheesh
Enthusiast
Posts: 38
Liked: never
Joined: Jun 06, 2009 8:12 pm
Full Name: Satheesh Varadharajan
Contact:

Re: v6 "Cannot use CBT. Soap fault" ... fixed

Post by satheesh »

I've found this to be quite helpful and solved my problem -> How to reset CBT.

:D
Satheesh.net
wahrheit2004
Influencer
Posts: 10
Liked: 2 times
Joined: Feb 27, 2012 7:33 am
Full Name: DW
Contact:

Re: v6 "Cannot use CBT. Soap fault" ... fixed

Post by wahrheit2004 » 2 people like this post

Hello everyone,

I just had the same issue running 2x ESXi 5.0 (latest patch), 2x iSCSI SAN (each Multipath / Active/Active Controller), vCenter 5.1.0a and Veeam B&R 6.5.
My message is:

01.11.2012 08:21:19 :: Cannot use CBT: Soap fault. Von Datei /vmfs/volumes/50912cdd-881ab72c-b9ec-001e67259774/debmirror_Squeeze/SUGARCRM_Squeeze.vmdk verursachter Fehler.Detail: '', endpoint: ''

The reason was that I moved the VM from SAN A -> SAN B.
After 2nd backup run it workt without this error message.
rsargeant
Influencer
Posts: 11
Liked: never
Joined: Feb 16, 2012 2:03 pm
Full Name: Richard Sargeant
Contact:

Re: v6 "Cannot use CBT. Soap fault" ... fixed

Post by rsargeant »

Hi Guys,

I'm trying to backup _replica VMs and getting this same problem with v6.0.181

Having done some testing, the first time you backup a _replica VM, CBT seems to 'be happy' and the backup runs
through quite quick. You then re-run the same job and CBT 'has broken'. This is a real pain because the backup then
takes 4 times as long!

Really need a fix or temporary workaround for this (the manual CBT reset fix doesn't seem to work BTW).

Thanks,

Richard.
Vitaliy S.
VP, Product Management
Posts: 27055
Liked: 2710 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: v6 "Cannot use CBT. Soap fault" ... fixed

Post by Vitaliy S. »

Hi Richard,

VMware CBT will not be used for VM replicas, because CBT cannot be enabled on the VM that has already at least one snapshot in its configuration, besides VMware CBT doesn't seem to work (seen a couple of issues reported on these forums) for powered off VMs.

Thanks.
rsargeant
Influencer
Posts: 11
Liked: never
Joined: Feb 16, 2012 2:03 pm
Full Name: Richard Sargeant
Contact:

Re: v6 "Cannot use CBT. Soap fault" ... fixed

Post by rsargeant »

OK, so VMware CBT doesn't work with Powered Off VMs, fair enough.

It is my understanding therefore, that Veeam's 'Proprietary CBT equivalent' should then kick in? If this is the case it doesn't seem to work because backing up the same _replica server everyday takes 4-5 hours, it's as if it does a Full backup everyday??

This can't be right??

Cheers,

Richard.
Vitaliy S.
VP, Product Management
Posts: 27055
Liked: 2710 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: v6 "Cannot use CBT. Soap fault" ... fixed

Post by Vitaliy S. »

rsargeant wrote:It is my understanding therefore, that Veeam's 'Proprietary CBT equivalent' should then kick in?
Absolutely correct, but our proprietary CBT engine basically scans the entire source VM image, calculates hashes for all blocks, and then verifies what blocks have been changed on your VM compared to the VM image stored in the backup file. Usually it takes the same amount of time as it is required to perform full VM backup, but in the case of using our CBT engine only changed blocks are transferred to the target repository.
Post Reply

Who is online

Users browsing this forum: tsmith and 96 guests