CBT/Soap Faults after v8 update 2

[PaulNSW]

I have an open case with a bunch of logs uploaded to 00909520

After updating to the latest version of B&R 8, a lot of our VM backups are failing or erroring during backup.
Right after the update, quite a few VMs in different jobs were showing the same errors

Code: Select all

05.05.2015 21:05:01 :: Processing xxxxx Error: Soap fault. A specified parameter was not correct. 
startOffsetDetail: '<InvalidArgumentFault xmlns="urn:internalvim25" xsi:type="InvalidArgument"><invalidProperty>startOffset</invalidProperty></InvalidArgumentFault>', endpoint: ''
Failed to upload disk.
Agent failed to process method {DataTransfer.SyncDisk}.
Exception from server: Soap fault. A specified parameter was not correct. 
startOffsetDetail: '<InvalidArgumentFault xmlns="urn:internalvim25" xsi:type="InvalidArgument"> 

05.05.2015 19:01:49 :: Processing YYYYYY Error: Soap fault. A specified parameter was not correct. 
startOffsetDetail: '<InvalidArgumentFault xmlns="urn:internalvim25" xsi:type="InvalidArgument"><invalidProperty>startOffset</invalidProperty></InvalidArgumentFault>', endpoint: ''
Failed to upload disk.
Agent failed to process method {DataTransfer.SyncDisk}.
Exception from server: Soap fault. A specified parameter was not correct. 
startOffsetDetail: '<InvalidArgumentFault xmlns="urn:internalvim25" xsi:type="InvalidArgument"> 

For some VMs, the backup completes successfully on the 3rd auto-retry, but others haven't completed a successful backup in a week! Some VMshave fixed themselves; they had failed, but then worked on the 3rd attempt, but now they complete on the first attempt.

I'm looking to get our service level up to premium now, as one email response a day is not getting the problem solved!!

I was advised to reset CBT, which I did using the powershell script. I even tried creating a completely new job with a VM that was having trouble. Still getting the same errors.

Has anyone else had this trouble?

[foggy]

Seems you're the first one to report this. There's an existing thread discussing quite similar issues, not sure it is applicable in your case though. Please continue investigating with support.

[PaulNSW]

So no matter how many times support ask me to reset CBT, it's not helping. I even tried to create a new backup job for one of the problematic VM's but it also fails.

I did notice however that the disk sizes showing the Action log are completely wrong - all disks are being reported as the same size, in this case, 900GB





Then I looked through all the Activity logs, and for the majority of VMs with mulitple disks, they are *all* showing the same disk size; usually the same size as the first disk to be hot-added! I was then advised to try refreshing/recalculating the disk size in the job settings, but it still did not help.

I then set up a new Veeam v8 server, but kept it at the base version, backed up the same VM, to the same destination (different folder) and it went through without problems.

[foggy]

I recommend escalating the case to a higher tier unless you've already done that, since those basic guesses suggested so far do not work.

[PaulNSW]

2 is as high as support will let me go as it's only effecting 2 particular jobs, and not the entire server

[foggy]

2d tier is ok, these guys should be able to investigate this.

[max2015]

We had a similar error message on backup and replication job with one of our Exchange server after updating to 8.0 update 2. Issue only occurs to this VM. I opened a ticket 00892985 and support got me to change Target and Source proxy setting on the replication job to use Network Transport Mode. Once replication job runs on network transport, it has been successful for last 3 days. For backup, I tried the CBT reset script, run Active Full, change proxy to network transport and the backup job would still failling. After I recreated the job, then it started working.. Ticket is still open for further investigation, but at least now I have successful backup on this Exchange server.

Error message from backup
5/3/2015 9:34:39 PM :: Processing #### Error: Soap fault. A specified parameter was not correct.
startOffsetDetail: '<InvalidArgumentFault xmlns="urn:internalvim25" xsi:type="InvalidArgument"><invalidProperty>startOffset</invalidProperty></InvalidArgumentFault>', endpoint: ''
Failed to upload disk.
Agent failed to process method {DataTransfer.SyncDisk}.
Exception from server: Soap fault. A specified parameter was not correct.
startOffsetDetail: '<InvalidArgumentFault xmlns="urn:internalvim25" xsi:type="InvalidArgument"

error message from replication
4/29/2015 4:32:22 AM :: Processing Error: vddk_disk_t::Resize: Not supported
Failed to download disk.
Reconnectable protocol device was closed.
Failed to upload disk.
Agent failed to process method {DataTransfer.SyncDisk}.

[PaulNSW]

In case someone else has the same issue, support finally got to the bottom of this. It seems VDDK6 addresses the disks differently than VDDK5.5
VDDK 6.0 doesn't take into account the SCSI adapters the VMDKs are attached to. It only uses bus-target-lun-diskname. So if you have multiple disks with the same VMDK name, it gets a little confused.

Workarounds were any of the following: rename the VMDKs within the VM so they are unique, or put all disks on the same SCSI adapter, or put the disks on different slots on different adapters (from 0:0, 1:0, 2:0 to 0:0, 1:1, 2:2)
That of course needs down-time on each VM.

The alternative is to replace the VDDK6 with VDDK5.5. Go to C:\Program Files (x86)\Veeam\Backup Transport\x64\vddk_6_0 on the proxies, and replace it with the contents of vddk_5_5 folder

This will be addressed in a future Veeam update.

[PaulNSW]

KB on this issue - although it doesn't mention reverting to VDDK5.5
http://www.veeam.com/kb2042

Didn't realise it could cause unrecoverable backups though!! Guess it's time to give SureBackup a try...