v6 "Cannot use CBT. Soap fault" ... fixed

[rsargeant]

Ok, so are you saying it shouldn't be effectively doing a Full Backup evey time? Or are you saying it
should only be backing up the delta changes?

Lastly, is there anyway to verify / monitor that the 'Veeam CBT' is actually working or not?

Many thanks.

Richard.

[foggy]

Ok, so are you saying it shouldn't be effectively doing a Full Backup evey time? Or are you saying it
should only be backing up the delta changes?

It should scan the whole VM image, but back up the changes only.

Lastly, is there anyway to verify / monitor that the 'Veeam CBT' is actually working or not?

In this case you should see the "Default tracker will be used" line in the job log.

[rsargeant]

Not seeing "Default tracker will be used" in the job logs, just the "Cannot use CBT. Soap fault...." error.

Also, tried specifically disabling VMware CBT in the job, but could only see 'Changed block tracking is disabled'

Any ideas?

Thanks

Richard.

[foggy]

Sorry, it's in in the source agent log (something like "Agent.<BackupJobName>.Source.log"). Anyway, if you do not see the "CBT" tag in the job session next to the virtual disk, then our proprietary algorithm is used for it.

[damocue]

Hi all,

On several customers we get CBT warnings on random executions, and most of the time on same VMs, and each time we reset CBT.
We are investigating on possible causes for it, like storage IOPS for SAN datastore, network connection, storage connectivity, VMs OS, etc.
Has someone any idea about possible causes for CBT warnings on Veeam B&R?

Thanks in advance,

[foggy]

David, what kind of warning do you get exactly? Were there probably any configuration changes performed over these VMs that could result in such warnings (e.g., virtual disk size extension)?

[damocue]

Hi,

This is the warning we get:

Cannot use CBT: Soap fault. Error caused by file /vmfs/volumes/<device hash>/<Virtual-Machine>/<Virtual-Machine>.vmdkDetail: '', endpoint: ''

No, it hasn't been any configuration change on these VMs for several months, and we get it on differents customers. It started to appear one day on each one, without any change made, and it repeats random times. The most of them share the same configuration: SAN storage with 10k (or 15k) rpm SAS disks, direct SAS connectivity between servers and storage, 2 hosts with VMware ESXi 5.0 Update 1 (but with 4.1 and 5.0 we got it too, with 5.1 we haven't done any installation by now), over 10-15 VMs. But it occurs also on customers with SAN storage with 10k-15k rpm FC disks (an HP EVA, if you know them), fiber channel connectivity with switches between servers and storage, and a cluster of 4-6 hosts with 30-40 VMs.

Our main goal (for my comrades and I) is to get the best configuration for only get green mails from Veeam B&R, at least for 2 months, hehe. No, seriously, this is a behavior which hasn't got any cause, at least for us, so we're looking for any possible cause. Any answer will be welcome.

We opened cases on Veeam support, and the answer for our question was that Veeam couldn't access to ctk file (<disk name>-ctk.vmdk), with the following IDs (for support team ):
- 5217888
- 5219073
- 5214448 & 5215205 (these 2 were put together by Veeam support team)
- 5202648

On all we applied Veeam KB 1113 for reset CBT (https://www.veeam.com/kb1113), but days after the warning appears on same VMs.

Regards,

[linux4guru]

Just curious... when you get the "cannot use CBT. Soap fault" - did you lose connection to the VM during the backup?

I am seeing that in my backup environment randomly... I am thinking this is caused by the previous CBT reading before the lost of connection so when it re-established the connection; it tries to make the request to read the CBT but can't because of the previous staled process.

[linux4guru]

Question - What is the different of this setting...

Changed block tracking....

Checked box - Use changed block tracking data (recommended)

Checked box - Enable changed block tracking for all processed VMs.

*What happen if I disabled the Enable changed block tracking for all processed VMs and leave the checked box for Use changed block tracking data? Will this get rip of the CBT error that I am seeing at random backup jobs?

[Vitaliy S.]

"Enable changed block tracking for all processed VMs" turns on VMware CBT for all VMs that currently have this option disabled. Once this option is enabled, VMware starts tracking changes for VM virtual disks.

"Use changed block tracking data (recommended)" option instructs Veeam backup/replication jobs to query VMware CBT data while backing up VMs with CBT option enabled (by the first option).

*What happen if I disabled the Enable changed block tracking for all processed VMs and leave the checked box for Use changed block tracking data? Will this get rip of the CBT error that I am seeing at random backup jobs?

I doubt that, as once VMware CBT has been enabled for processed VMs, the first option is no longer needed.

[kamalkant]

Vitaliy wrote:
VMware CBT will not be used for VM replicas, because CBT cannot be enabled on the VM that has already at least one snapshot in its configuration, besides VMware CBT doesn't seem to work (seen a couple of issues reported on these forums) for powered off VMs. "

I have replication job which is running successfully sometimes and sometime this error occurs:" Cannot use CBT: Soap fault. Error caused by file" (change block tracking is enabled) ..
"CBT wont work with Power off VM."..What will happen if I disable the NIC in VM rather than put my VM off. will then CBT would work? I need to do this " either Power off or disconnect "so that users cant access the server during my final replication. and I want this time as less as possible.

[foggy]

"CBT wont work with Power off VM."..What will happen if I disable the NIC in VM rather than put my VM off. will then CBT would work? I need to do this " either Power off or disconnect "so that users cant access the server during my final replication. and I want this time as less as possible.

May I ask you, what is the purpose of this action? Do you probably need to migrate the VM to other host/datastore?

[kamalkant]

Hi Foggy ...I am replicating my VM to different virtual data center in different location, what happened last night, we asked our user to log off from this DB server, so that no changes are made at the time of final replication. CBT did not work so my 270 GB data took around 5-6 hour to replicate the final changes to replica server.server was out of business for 5 hour. this could have been minimized if CBT would work with Power-off VM. I would definitely welcome any other way to perform my migration.

[veremin]

What will happen if I disable the NIC in VM rather than put my VM off. Will then CBT would work?

In fact, CBT functionality doesn’t care about network accessibility at all.

CBT wont work with Power off VM.

You might have slightly misunderstood this.

If you had enabled CBT for the given VM firstly, worked with this VM for a while, and only after that shut it down, then, there shouldn’t have been any issues with CBT work. In contrast, CBT wouldn't work for the VM that stays constantly in the powered off state, since there is nothing to track.

Just out of curiosity – why is it crucial for you to have users being unable to access the server during replication job?

Hope this helps.
Thanks.

[Vitaliy S.]

I would definitely welcome any other way to perform my migration.

Why just not to simply use Quick Migration jobs then? See our User Guide (page 61) for a detailed explanation of how these jobs work.

[kamalkant]

Hi v.Eremin

Yes I misunderstood it. reason of my warning was --> existing snapshot not the powe off VM. I deleted the snapshot and it worked fine. Thanks All, issue resolved now

[Borsh]

Hello!
During backing up the passive node Exchange server, a warning appears:
Cannot use CBT: Soap fault. A specified parameter was not correct. . deviceKeyDetail: '<InvalidArgumentFault xmlns="urn:internalvim25" xsi:type="InvalidArgument"><invalidProperty>deviceKey</invalidProperty></InvalidArgumentFault>', endpoint: ''

http://www.veeam.com/kb1113 - doesn't help us and warning remains.
Does anyone know of other ways to solve this problem?

Thanks

[foggy]

Igor, you've been merged into existing discussion regarding this specific error. Please review the considerations given above and feel free to ask any additional questions. Thanks.

[Borsh]

Alexander, I have watched this topic and have not found the solution.
In addition about kb1113 (4 step): "Open the source folder and remove any -CTK.VMDK files." - There are not any CTK.VMDK files and any snapshots, which may prevent CBT.

[foggy]

Then I suggest contacting support so they could determine the reason of CBT failure in your particular case by reviewing the job log files.

[veremin]

Additionally, what happens if you manually enable CBT on a given VM via vSphere client? Thanks.

[Waddy]

Recently had 2- different sites power off their VM Ware 5.1 Host servers. Not sure why. Accidents maybe. After these events the soap cbt errors were present. I remebered having a similar issue (cbt soap error or other error related to cbt). Previous (a few years ago on veeam 5) solution was to create a vmware snapshot and then delete. Of course, you cannot have any existing snapshots with CBT anyway so these would be new ss..
I performed this on all the problematic VM's and the next veeam bu with cbt was successful - on about 15 individual VM jobs.

[jamerson]

Dear All,
i am still on 6.5, after the back-up report is sent i get this error !

Code: Select all

Cannot use CBT: Soap fault. Error caused by file /vmfs/volumes/51ee8ec7-8ff76db9-b3c5-6805ca195370/DC/DC.vmdkDetail: '', endpoint: ''

i know its End point Symantec Error, but it shows up just with one server and others not that has ENDPOINT installed!
any suggestions please?

[foggy]

i know its End point Symantec Error, but it shows up just with one server and others not that has ENDPOINT installed!

Julien, actually, this error has nothing to do with Symantec Endpoint Protection. Please review this thread for the tips that might help and feel free to contact support for investigation in case it does not help.

[jamerson]

Julien, actually, this error has nothing to do with Symantec Endpoint Protection. Please review this thread for the tips that might help and feel free to contact support for investigation in case it does not help.

Thank you foggy,
what i am supposed to check for in this forum ?
none has the same error !

[foggy]

Exactly the same error is mentioned in this thread at least a couple of times. You could check, for example, whether the VM in action has existing snapshots and try to delete them.

[jamerson]

Exactly the same error is mentioned in this thread at least a couple of times. You could check, for example, whether the VM in action has existing snapshots and try to delete them.

i've deleted the snashot and disk that were created by the backup jobs, and run the job again and voila went smootly,
waiting for the schedule backup tonight to confirm either it fixed or not,

[foggy]

Glad it helped, at least for the manual job run. Please keep us informed on how it is going.

[Noiden]

i've deleted the snashot and disk that were created by the backup jobs, and run the job again and voila went smootly,
waiting for the schedule backup tonight to confirm either it fixed or not,

Just wondering, did you delete the snapshot through vSphere client? Or from veeam?
Because I have this warning and the snapshot manager has a "VEEAM BACKUP TEMPORARY SNAPSHOT" but it also says "Please do not delete this snapshot. It is being used by Veeam Backup".
But it's safe to delete that one and what is the files that I should remove from the disk?

[foggy]

Because I have this warning and the snapshot manager has a "VEEAM BACKUP TEMPORARY SNAPSHOT" but it also says "Please do not delete this snapshot. It is being used by Veeam Backup". But it's safe to delete that one and what is the files that I should remove from the disk?

That's actually another issue, please review this thread for details.