CBT/Soap Faults after v8 update 2

VMware specific discussions

CBT/Soap Faults after v8 update 2

Veeam Logoby PaulNSW » Fri May 08, 2015 9:18 am

I have an open case with a bunch of logs uploaded to 00909520

After updating to the latest version of B&R 8, a lot of our VM backups are failing or erroring during backup.
Right after the update, quite a few VMs in different jobs were showing the same errors

Code: Select all
05.05.2015 21:05:01 :: Processing xxxxx Error: Soap fault. A specified parameter was not correct.
startOffsetDetail: '<InvalidArgumentFault xmlns="urn:internalvim25" xsi:type="InvalidArgument"><invalidProperty>startOffset</invalidProperty></InvalidArgumentFault>', endpoint: ''
Failed to upload disk.
Agent failed to process method {DataTransfer.SyncDisk}.
Exception from server: Soap fault. A specified parameter was not correct.
startOffsetDetail: '<InvalidArgumentFault xmlns="urn:internalvim25" xsi:type="InvalidArgument">

05.05.2015 19:01:49 :: Processing YYYYYY Error: Soap fault. A specified parameter was not correct.
startOffsetDetail: '<InvalidArgumentFault xmlns="urn:internalvim25" xsi:type="InvalidArgument"><invalidProperty>startOffset</invalidProperty></InvalidArgumentFault>', endpoint: ''
Failed to upload disk.
Agent failed to process method {DataTransfer.SyncDisk}.
Exception from server: Soap fault. A specified parameter was not correct.
startOffsetDetail: '<InvalidArgumentFault xmlns="urn:internalvim25" xsi:type="InvalidArgument">


For some VMs, the backup completes successfully on the 3rd auto-retry, but others haven't completed a successful backup in a week! Some VMshave fixed themselves; they had failed, but then worked on the 3rd attempt, but now they complete on the first attempt.

I'm looking to get our service level up to premium now, as one email response a day is not getting the problem solved!!

I was advised to reset CBT, which I did using the powershell script. I even tried creating a completely new job with a VM that was having trouble. Still getting the same errors.

Has anyone else had this trouble?
PaulNSW
Enthusiast
 
Posts: 48
Liked: 8 times
Joined: Mon May 14, 2012 10:48 am

Re: CBT/Soap Faults after v8 update 2

Veeam Logoby foggy » Fri May 08, 2015 1:07 pm

Seems you're the first one to report this. There's an existing thread discussing quite similar issues, not sure it is applicable in your case though. Please continue investigating with support.
foggy
Veeam Software
 
Posts: 14744
Liked: 1082 times
Joined: Mon Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson

Re: CBT/Soap Faults after v8 update 2

Veeam Logoby PaulNSW » Mon May 11, 2015 8:09 am

So no matter how many times support ask me to reset CBT, it's not helping. I even tried to create a new backup job for one of the problematic VM's but it also fails.

I did notice however that the disk sizes showing the Action log are completely wrong - all disks are being reported as the same size, in this case, 900GB

Image

Image

Then I looked through all the Activity logs, and for the majority of VMs with mulitple disks, they are *all* showing the same disk size; usually the same size as the first disk to be hot-added! I was then advised to try refreshing/recalculating the disk size in the job settings, but it still did not help.

I then set up a new Veeam v8 server, but kept it at the base version, backed up the same VM, to the same destination (different folder) and it went through without problems.
PaulNSW
Enthusiast
 
Posts: 48
Liked: 8 times
Joined: Mon May 14, 2012 10:48 am

Re: CBT/Soap Faults after v8 update 2

Veeam Logoby foggy » Mon May 11, 2015 10:36 am

I recommend escalating the case to a higher tier unless you've already done that, since those basic guesses suggested so far do not work.
foggy
Veeam Software
 
Posts: 14744
Liked: 1082 times
Joined: Mon Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson

Re: CBT/Soap Faults after v8 update 2

Veeam Logoby PaulNSW » Mon May 11, 2015 10:40 am

2 is as high as support will let me go as it's only effecting 2 particular jobs, and not the entire server :(
PaulNSW
Enthusiast
 
Posts: 48
Liked: 8 times
Joined: Mon May 14, 2012 10:48 am

Re: CBT/Soap Faults after v8 update 2

Veeam Logoby foggy » Mon May 11, 2015 10:43 am

2d tier is ok, these guys should be able to investigate this.
foggy
Veeam Software
 
Posts: 14744
Liked: 1082 times
Joined: Mon Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson

Re: CBT/Soap Faults after v8 update 2

Veeam Logoby max2015 » Mon May 11, 2015 11:32 pm

We had a similar error message on backup and replication job with one of our Exchange server after updating to 8.0 update 2. Issue only occurs to this VM. I opened a ticket 00892985 and support got me to change Target and Source proxy setting on the replication job to use Network Transport Mode. Once replication job runs on network transport, it has been successful for last 3 days. For backup, I tried the CBT reset script, run Active Full, change proxy to network transport and the backup job would still failling. After I recreated the job, then it started working.. Ticket is still open for further investigation, but at least now I have successful backup on this Exchange server.

Error message from backup
5/3/2015 9:34:39 PM :: Processing #### Error: Soap fault. A specified parameter was not correct.
startOffsetDetail: '<InvalidArgumentFault xmlns="urn:internalvim25" xsi:type="InvalidArgument"><invalidProperty>startOffset</invalidProperty></InvalidArgumentFault>', endpoint: ''
Failed to upload disk.
Agent failed to process method {DataTransfer.SyncDisk}.
Exception from server: Soap fault. A specified parameter was not correct.
startOffsetDetail: '<InvalidArgumentFault xmlns="urn:internalvim25" xsi:type="InvalidArgument"

error message from replication
4/29/2015 4:32:22 AM :: Processing Error: vddk_disk_t::Resize: Not supported
Failed to download disk.
Reconnectable protocol device was closed.
Failed to upload disk.
Agent failed to process method {DataTransfer.SyncDisk}.
max2015
Novice
 
Posts: 3
Liked: never
Joined: Thu Nov 27, 2014 11:12 pm
Full Name: Min

Re: CBT/Soap Faults after v8 update 2

Veeam Logoby PaulNSW » Tue Jun 02, 2015 6:09 am 2 people like this post

In case someone else has the same issue, support finally got to the bottom of this. It seems VDDK6 addresses the disks differently than VDDK5.5
VDDK 6.0 doesn't take into account the SCSI adapters the VMDKs are attached to. It only uses bus-target-lun-diskname. So if you have multiple disks with the same VMDK name, it gets a little confused.

Workarounds were any of the following: rename the VMDKs within the VM so they are unique, or put all disks on the same SCSI adapter, or put the disks on different slots on different adapters (from 0:0, 1:0, 2:0 to 0:0, 1:1, 2:2)
That of course needs down-time on each VM.

The alternative is to replace the VDDK6 with VDDK5.5. Go to C:\Program Files (x86)\Veeam\Backup Transport\x64\vddk_6_0 on the proxies, and replace it with the contents of vddk_5_5 folder

This will be addressed in a future Veeam update.
PaulNSW
Enthusiast
 
Posts: 48
Liked: 8 times
Joined: Mon May 14, 2012 10:48 am

Re: CBT/Soap Faults after v8 update 2

Veeam Logoby PaulNSW » Mon Jun 15, 2015 7:03 am 1 person likes this post

KB on this issue - although it doesn't mention reverting to VDDK5.5
http://www.veeam.com/kb2042

Didn't realise it could cause unrecoverable backups though!! Guess it's time to give SureBackup a try...
PaulNSW
Enthusiast
 
Posts: 48
Liked: 8 times
Joined: Mon May 14, 2012 10:48 am


Return to VMware vSphere



Who is online

Users browsing this forum: Yahoo [Bot] and 22 guests