Host-based backup of VMware vSphere VMs.
rsargeant
Influencer
Posts: 11
Liked: never
Joined: Feb 16, 2012 2:03 pm
Full Name: Richard Sargeant
Contact:

Re: v6 "Cannot use CBT. Soap fault" ... fixed

Post by rsargeant »

Ok, so are you saying it shouldn't be effectively doing a Full Backup evey time? Or are you saying it
should only be backing up the delta changes?

Lastly, is there anyway to verify / monitor that the 'Veeam CBT' is actually working or not?

Many thanks.

Richard.
foggy
Veeam Software
Posts: 21069
Liked: 2115 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: v6 "Cannot use CBT. Soap fault" ... fixed

Post by foggy »

rsargeant wrote:Ok, so are you saying it shouldn't be effectively doing a Full Backup evey time? Or are you saying it
should only be backing up the delta changes?
It should scan the whole VM image, but back up the changes only.
rsargeant wrote:Lastly, is there anyway to verify / monitor that the 'Veeam CBT' is actually working or not?
In this case you should see the "Default tracker will be used" line in the job log.
rsargeant
Influencer
Posts: 11
Liked: never
Joined: Feb 16, 2012 2:03 pm
Full Name: Richard Sargeant
Contact:

Re: v6 "Cannot use CBT. Soap fault" ... fixed

Post by rsargeant »

Not seeing "Default tracker will be used" in the job logs, just the "Cannot use CBT. Soap fault...." error.

Also, tried specifically disabling VMware CBT in the job, but could only see 'Changed block tracking is disabled'

Any ideas?

Thanks

Richard.
foggy
Veeam Software
Posts: 21069
Liked: 2115 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: v6 "Cannot use CBT. Soap fault" ... fixed

Post by foggy »

Sorry, it's in in the source agent log (something like "Agent.<BackupJobName>.Source.log"). Anyway, if you do not see the "CBT" tag in the job session next to the virtual disk, then our proprietary algorithm is used for it.
damocue
Veeam ProPartner
Posts: 45
Liked: never
Joined: Feb 16, 2011 9:10 am
Full Name: David Molina
Location: Valencia, Spain
Contact:

[MERGED] Possible CBT warning causes

Post by damocue »

Hi all,

On several customers we get CBT warnings on random executions, and most of the time on same VMs, and each time we reset CBT.
We are investigating on possible causes for it, like storage IOPS for SAN datastore, network connection, storage connectivity, VMs OS, etc.
Has someone any idea about possible causes for CBT warnings on Veeam B&R?

Thanks in advance,
foggy
Veeam Software
Posts: 21069
Liked: 2115 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Possible CBT warning causes

Post by foggy »

David, what kind of warning do you get exactly? Were there probably any configuration changes performed over these VMs that could result in such warnings (e.g., virtual disk size extension)?
damocue
Veeam ProPartner
Posts: 45
Liked: never
Joined: Feb 16, 2011 9:10 am
Full Name: David Molina
Location: Valencia, Spain
Contact:

Re: Possible CBT warning causes

Post by damocue »

Hi,

This is the warning we get:
Cannot use CBT: Soap fault. Error caused by file /vmfs/volumes/<device hash>/<Virtual-Machine>/<Virtual-Machine>.vmdkDetail: '', endpoint: ''
No, it hasn't been any configuration change on these VMs for several months, and we get it on differents customers. It started to appear one day on each one, without any change made, and it repeats random times. The most of them share the same configuration: SAN storage with 10k (or 15k) rpm SAS disks, direct SAS connectivity between servers and storage, 2 hosts with VMware ESXi 5.0 Update 1 (but with 4.1 and 5.0 we got it too, with 5.1 we haven't done any installation by now), over 10-15 VMs. But it occurs also on customers with SAN storage with 10k-15k rpm FC disks (an HP EVA, if you know them), fiber channel connectivity with switches between servers and storage, and a cluster of 4-6 hosts with 30-40 VMs.

Our main goal (for my comrades and I) is to get the best configuration for only get green mails from Veeam B&R, at least for 2 months, hehe. No, seriously, this is a behavior which hasn't got any cause, at least for us, so we're looking for any possible cause. Any answer will be welcome.

We opened cases on Veeam support, and the answer for our question was that Veeam couldn't access to ctk file (<disk name>-ctk.vmdk), with the following IDs (for support team :) ):
- 5217888
- 5219073
- 5214448 & 5215205 (these 2 were put together by Veeam support team)
- 5202648

On all we applied Veeam KB 1113 for reset CBT (https://www.veeam.com/kb1113), but days after the warning appears on same VMs.

Regards,
linux4guru
Enthusiast
Posts: 40
Liked: never
Joined: Aug 24, 2011 8:26 pm
Full Name: Thomas Lam
Contact:

Re: v6 "Cannot use CBT. Soap fault" ... fixed

Post by linux4guru »

Just curious... when you get the "cannot use CBT. Soap fault" - did you lose connection to the VM during the backup?

I am seeing that in my backup environment randomly... I am thinking this is caused by the previous CBT reading before the lost of connection so when it re-established the connection; it tries to make the request to read the CBT but can't because of the previous staled process.
linux4guru
Enthusiast
Posts: 40
Liked: never
Joined: Aug 24, 2011 8:26 pm
Full Name: Thomas Lam
Contact:

Re: v6 "Cannot use CBT. Soap fault" ... fixed

Post by linux4guru »

Question - What is the different of this setting...

Changed block tracking....

Checked box - Use changed block tracking data (recommended)

Checked box - Enable changed block tracking for all processed VMs.

*What happen if I disabled the Enable changed block tracking for all processed VMs and leave the checked box for Use changed block tracking data? Will this get rip of the CBT error that I am seeing at random backup jobs?
Vitaliy S.
VP, Product Management
Posts: 27055
Liked: 2710 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: v6 "Cannot use CBT. Soap fault" ... fixed

Post by Vitaliy S. »

"Enable changed block tracking for all processed VMs" turns on VMware CBT for all VMs that currently have this option disabled. Once this option is enabled, VMware starts tracking changes for VM virtual disks.

"Use changed block tracking data (recommended)" option instructs Veeam backup/replication jobs to query VMware CBT data while backing up VMs with CBT option enabled (by the first option).
linux4guru wrote:*What happen if I disabled the Enable changed block tracking for all processed VMs and leave the checked box for Use changed block tracking data? Will this get rip of the CBT error that I am seeing at random backup jobs?
I doubt that, as once VMware CBT has been enabled for processed VMs, the first option is no longer needed.
kamalkant
Influencer
Posts: 24
Liked: 1 time
Joined: Feb 04, 2013 10:45 am
Full Name: kamalkant singhal
Contact:

Re: v6 "Cannot use CBT. Soap fault" ... fixed

Post by kamalkant »

Vitaliy wrote:
VMware CBT will not be used for VM replicas, because CBT cannot be enabled on the VM that has already at least one snapshot in its configuration, besides VMware CBT doesn't seem to work (seen a couple of issues reported on these forums) for powered off VMs. "

I have replication job which is running successfully sometimes and sometime this error occurs:" Cannot use CBT: Soap fault. Error caused by file" (change block tracking is enabled) ..
"CBT wont work with Power off VM."..What will happen if I disable the NIC in VM rather than put my VM off. will then CBT would work? I need to do this " either Power off or disconnect "so that users cant access the server during my final replication. and I want this time as less as possible.
foggy
Veeam Software
Posts: 21069
Liked: 2115 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: v6 "Cannot use CBT. Soap fault" ... fixed

Post by foggy »

kamalkant wrote:"CBT wont work with Power off VM."..What will happen if I disable the NIC in VM rather than put my VM off. will then CBT would work? I need to do this " either Power off or disconnect "so that users cant access the server during my final replication. and I want this time as less as possible.
May I ask you, what is the purpose of this action? Do you probably need to migrate the VM to other host/datastore?
kamalkant
Influencer
Posts: 24
Liked: 1 time
Joined: Feb 04, 2013 10:45 am
Full Name: kamalkant singhal
Contact:

Re: v6 "Cannot use CBT. Soap fault" ... fixed

Post by kamalkant »

Hi Foggy ...I am replicating my VM to different virtual data center in different location, what happened last night, we asked our user to log off from this DB server, so that no changes are made at the time of final replication. CBT did not work so my 270 GB data took around 5-6 hour to replicate the final changes to replica server.server was out of business for 5 hour. this could have been minimized if CBT would work with Power-off VM. I would definitely welcome any other way to perform my migration.
veremin
Product Manager
Posts: 20270
Liked: 2252 times
Joined: Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin
Contact:

Re: v6 "Cannot use CBT. Soap fault" ... fixed

Post by veremin »

What will happen if I disable the NIC in VM rather than put my VM off. Will then CBT would work?
In fact, CBT functionality doesn’t care about network accessibility at all.
CBT wont work with Power off VM.
You might have slightly misunderstood this.

If you had enabled CBT for the given VM firstly, worked with this VM for a while, and only after that shut it down, then, there shouldn’t have been any issues with CBT work. In contrast, CBT wouldn't work for the VM that stays constantly in the powered off state, since there is nothing to track.

Just out of curiosity – why is it crucial for you to have users being unable to access the server during replication job?

Hope this helps.
Thanks.
Vitaliy S.
VP, Product Management
Posts: 27055
Liked: 2710 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: v6 "Cannot use CBT. Soap fault" ... fixed

Post by Vitaliy S. »

kamalkant wrote:I would definitely welcome any other way to perform my migration.
Why just not to simply use Quick Migration jobs then? See our User Guide (page 61) for a detailed explanation of how these jobs work.
kamalkant
Influencer
Posts: 24
Liked: 1 time
Joined: Feb 04, 2013 10:45 am
Full Name: kamalkant singhal
Contact:

Re: v6 "Cannot use CBT. Soap fault" ... fixed

Post by kamalkant »

Hi v.Eremin

Yes I misunderstood it. reason of my warning was --> existing snapshot not the powe off VM. I deleted the snapshot and it worked fine. Thanks All, issue resolved now
Borsh
Influencer
Posts: 20
Liked: 1 time
Joined: May 06, 2013 5:35 am
Full Name: Igor Bonev
Contact:

[MERGED] CBT warning

Post by Borsh »

Hello!
During backing up the passive node Exchange server, a warning appears:
Cannot use CBT: Soap fault. A specified parameter was not correct. . deviceKeyDetail: '<InvalidArgumentFault xmlns="urn:internalvim25" xsi:type="InvalidArgument"><invalidProperty>deviceKey</invalidProperty></InvalidArgumentFault>', endpoint: ''

http://www.veeam.com/kb1113 - doesn't help us and warning remains.
Does anyone know of other ways to solve this problem?

Thanks
foggy
Veeam Software
Posts: 21069
Liked: 2115 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: v6 "Cannot use CBT. Soap fault" ... fixed

Post by foggy »

Igor, you've been merged into existing discussion regarding this specific error. Please review the considerations given above and feel free to ask any additional questions. Thanks.
Borsh
Influencer
Posts: 20
Liked: 1 time
Joined: May 06, 2013 5:35 am
Full Name: Igor Bonev
Contact:

Re: v6 "Cannot use CBT. Soap fault" ... fixed

Post by Borsh »

Alexander, I have watched this topic and have not found the solution. :(
In addition about kb1113 (4 step): "Open the source folder and remove any -CTK.VMDK files." - There are not any CTK.VMDK files and any snapshots, which may prevent CBT.
foggy
Veeam Software
Posts: 21069
Liked: 2115 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: v6 "Cannot use CBT. Soap fault" ... fixed

Post by foggy »

Then I suggest contacting support so they could determine the reason of CBT failure in your particular case by reviewing the job log files.
veremin
Product Manager
Posts: 20270
Liked: 2252 times
Joined: Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin
Contact:

Re: v6 "Cannot use CBT. Soap fault" ... fixed

Post by veremin »

Additionally, what happens if you manually enable CBT on a given VM via vSphere client? Thanks.
Waddy
Novice
Posts: 7
Liked: never
Joined: Feb 02, 2010 3:17 am
Full Name: Waddy Williams
Contact:

Re: v6 "Cannot use CBT. Soap fault" ... fixed

Post by Waddy »

Recently had 2- different sites power off their VM Ware 5.1 Host servers. Not sure why. Accidents maybe. After these events the soap cbt errors were present. I remebered having a similar issue (cbt soap error or other error related to cbt). Previous (a few years ago on veeam 5) solution was to create a vmware snapshot and then delete. Of course, you cannot have any existing snapshots with CBT anyway so these would be new ss..
I performed this on all the problematic VM's and the next veeam bu with cbt was successful - on about 15 individual VM jobs.
jamerson
Veteran
Posts: 366
Liked: 24 times
Joined: May 01, 2013 9:54 pm
Full Name: Julien
Contact:

[MERGED] Backup Error

Post by jamerson »

Dear All,
i am still on 6.5, after the back-up report is sent i get this error !

Code: Select all

Cannot use CBT: Soap fault. Error caused by file /vmfs/volumes/51ee8ec7-8ff76db9-b3c5-6805ca195370/DC/DC.vmdkDetail: '', endpoint: ''
i know its End point Symantec Error, but it shows up just with one server and others not that has ENDPOINT installed!
any suggestions please?
foggy
Veeam Software
Posts: 21069
Liked: 2115 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: [MERGED] Backup Error

Post by foggy »

jamerson wrote:i know its End point Symantec Error, but it shows up just with one server and others not that has ENDPOINT installed!
Julien, actually, this error has nothing to do with Symantec Endpoint Protection. Please review this thread for the tips that might help and feel free to contact support for investigation in case it does not help.
jamerson
Veteran
Posts: 366
Liked: 24 times
Joined: May 01, 2013 9:54 pm
Full Name: Julien
Contact:

Re: [MERGED] Backup Error

Post by jamerson »

foggy wrote: Julien, actually, this error has nothing to do with Symantec Endpoint Protection. Please review this thread for the tips that might help and feel free to contact support for investigation in case it does not help.
Thank you foggy,
what i am supposed to check for in this forum ?
none has the same error !
foggy
Veeam Software
Posts: 21069
Liked: 2115 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: v6 "Cannot use CBT. Soap fault" ... fixed

Post by foggy »

Exactly the same error is mentioned in this thread at least a couple of times. You could check, for example, whether the VM in action has existing snapshots and try to delete them.
jamerson
Veteran
Posts: 366
Liked: 24 times
Joined: May 01, 2013 9:54 pm
Full Name: Julien
Contact:

Re: v6 "Cannot use CBT. Soap fault" ... fixed

Post by jamerson »

foggy wrote:Exactly the same error is mentioned in this thread at least a couple of times. You could check, for example, whether the VM in action has existing snapshots and try to delete them.
i've deleted the snashot and disk that were created by the backup jobs, and run the job again and voila went smootly,
waiting for the schedule backup tonight to confirm either it fixed or not,
foggy
Veeam Software
Posts: 21069
Liked: 2115 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: v6 "Cannot use CBT. Soap fault" ... fixed

Post by foggy »

Glad it helped, at least for the manual job run. Please keep us informed on how it is going.
Noiden
Influencer
Posts: 10
Liked: never
Joined: Mar 28, 2013 2:21 pm
Contact:

Re: v6 "Cannot use CBT. Soap fault" ... fixed

Post by Noiden »

jamerson wrote: i've deleted the snashot and disk that were created by the backup jobs, and run the job again and voila went smootly,
waiting for the schedule backup tonight to confirm either it fixed or not,
Just wondering, did you delete the snapshot through vSphere client? Or from veeam?
Because I have this warning and the snapshot manager has a "VEEAM BACKUP TEMPORARY SNAPSHOT" but it also says "Please do not delete this snapshot. It is being used by Veeam Backup".
But it's safe to delete that one and what is the files that I should remove from the disk?
foggy
Veeam Software
Posts: 21069
Liked: 2115 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: v6 "Cannot use CBT. Soap fault" ... fixed

Post by foggy »

Noiden wrote:Because I have this warning and the snapshot manager has a "VEEAM BACKUP TEMPORARY SNAPSHOT" but it also says "Please do not delete this snapshot. It is being used by Veeam Backup". But it's safe to delete that one and what is the files that I should remove from the disk?
That's actually another issue, please review this thread for details.
Post Reply

Who is online

Users browsing this forum: No registered users and 64 guests