V9: random soap fault- vm backup failure

VMware specific discussions

V9: random soap fault- vm backup failure

Veeam Logoby kjstech » Tue Jan 19, 2016 11:55 am

Hi, running v9 since last week and today I experienced my first set of failed backups. There are a small number of VM's that completely failed backup (including the retries) with messages like this:

Code: Select all
Error: Soap fault. fault.InvalidArgument.summaryDetail: '<InvalidArgumentFault xmlns="urn:internalvim25" xsi:type="InvalidArgument"><invalidProperty>deviceKey</invalidProperty></InvalidArgumentFault>', endpoint: '' Failed to upload disk. Agent failed to process method {DataTransfer.SyncDisk}.


Any idea what this means? Thanks!
kjstech
Expert
 
Posts: 142
Liked: 14 times
Joined: Fri Jan 17, 2014 4:12 pm
Full Name: Keith S

Re: V9: random soap fault- vm backup failure

Veeam Logoby PTide » Tue Jan 19, 2016 11:58 am

Hi,

A closer look to logs is needed. Please open a case with support team and post your case ID here. Do those VMs differ somehow from others? Also, this KB is worth checking.

Thank you.
PTide
Veeam Software
 
Posts: 3019
Liked: 246 times
Joined: Tue May 19, 2015 1:46 pm

Re: V9: random soap fault- vm backup failure

Veeam Logoby kjstech » Tue Jan 19, 2016 3:07 pm

Thanks 01671181 created.
kjstech
Expert
 
Posts: 142
Liked: 14 times
Joined: Fri Jan 17, 2014 4:12 pm
Full Name: Keith S

Re: V9: random soap fault- vm backup failure

Veeam Logoby kjstech » Tue Jan 26, 2016 1:54 pm 1 person likes this post

While not detailed in the daily job log summaries, an examination of full logs to support yielded this gem:
Code: Select all
[18.01.2016 20:43:42] <  2824>      >>  |--tr:Failed to enumerate changed areas of the disk using CTK. Device key: [2001], size: [1319413952512]. VM ref: [vm-5231]. Change ID: [52 e2 b4 c9 f2 dc ad bf-a6 38 24 10 b0 ff ce e9/504]


Since its referencing CTK, we decided to try that vmware powershell script from Veeam KB1113 on this machine (and the few others). Last night the job completed fine for the machine in question.

It still is odd that it began to happen out of the blue when there were no updates to vSphere or ESXi.
kjstech
Expert
 
Posts: 142
Liked: 14 times
Joined: Fri Jan 17, 2014 4:12 pm
Full Name: Keith S

Re: V9: random soap fault- vm backup failure

Veeam Logoby PTide » Tue Jan 26, 2016 2:08 pm

Thank you for feedback! As it was mentioned in release notes, it's considered to be a best practice to reset CBT after new installs/updates. Please see this thread and this KB for more info.
PTide
Veeam Software
 
Posts: 3019
Liked: 246 times
Joined: Tue May 19, 2015 1:46 pm

Re: V9: random soap fault- vm backup failure

Veeam Logoby kjstech » Wed Jan 27, 2016 3:19 am

It happened again tonight. Just got the failure again. Here's what the email says:

Code: Select all
Error: Soap fault. fault.InvalidArgument.summaryDetail: '<InvalidArgumentFault xmlns="urn:internalvim25" xsi:type="InvalidArgument"><invalidProperty>deviceKey</invalidProperty></InvalidArgumentFault>', endpoint: '' Failed to upload disk. Agent failed to process method {DataTransfer.SyncDisk}. Exception from server: Soap fault. fault.InvalidArgument.summaryDetail: '<InvalidArgumentFault xmlns="urn:internalvim25" xsi:type="InvalidArgument"><invalidProperty>deviceKey</invalidProperty></InvalidArgumentFault>


I will update the case tomorrow.

I don't know, do we have to reset CBT every night?
kjstech
Expert
 
Posts: 142
Liked: 14 times
Joined: Fri Jan 17, 2014 4:12 pm
Full Name: Keith S

Re: V9: random soap fault- vm backup failure

Veeam Logoby PTide » Wed Jan 27, 2016 9:58 am

I don't know, do we have to reset CBT every night?
Of course not. That's an unexpected behaviour so please keep working with support team.

Also, did the failure happened to the very same VMs? If so please check if there are any snapshots present.

Thank you.
PTide
Veeam Software
 
Posts: 3019
Liked: 246 times
Joined: Tue May 19, 2015 1:46 pm

V9: random soap fault- vm backup failure

Veeam Logoby kjstech » Wed Jan 27, 2016 10:24 am

Another job had showed two more VMs that failed with this same error

Code: Select all
Error: Soap fault. fault.InvalidArgument.summaryDetail: '<InvalidArgumentFault xmlns="urn:internalvim25" xsi:type="InvalidArgument"><invalidProperty>deviceKey</invalidProperty></InvalidArgumentFault>', endpoint: '' Failed to upload disk. Agent failed to process method {DataTransfer.SyncDisk}


There's no way they had snapshots as the are very active machines and vcenter sends us alert emails when the snapshot delta disk exceeds 5gb. That only takes a few hours to achieve and we didn't get a single email about that yesterday.

I also have two newer VM's who've never been backed up. First it was that veeam couldn't connect to them, but I opened up ping and smb in their respective firewalls and tested successful access of their c$ share from the Veeam server:


Code: Select all
Error: Error code: 0x800706ba Failed to invoke func [RegisterIndexJob]: The RPC server is unavailable.. RPC function call failed. Function name: [BlobCall]. Target machine: [10.1.1.39]. RPC error:The RPC server is unavailable. Code: 1722 Error code: 0x800706ba Failed to invoke func [RegisterIndexJob]: The RPC server is unavailable.. RPC function call failed. Function name: [BlobCall]. Target machine: [10.1.1.39]. RPC error:The RPC server is unavailable. Code: 1722



I forwarded the job summary emails to support and will update the logs when I get in to work.
kjstech
Expert
 
Posts: 142
Liked: 14 times
Joined: Fri Jan 17, 2014 4:12 pm
Full Name: Keith S

Re: V9: random soap fault- vm backup failure

Veeam Logoby PTide » Wed Jan 27, 2016 10:42 am

Should you feel like you're not satisfied with a service you get from tech team please use the "Talk to Manager" button to escalate the case. Looking forward to hear good news from you.

Thank you.
PTide
Veeam Software
 
Posts: 3019
Liked: 246 times
Joined: Tue May 19, 2015 1:46 pm

Re: V9: random soap fault- vm backup failure

Veeam Logoby kjstech » Wed Jan 27, 2016 1:05 pm

Thank you, all the appropriate logs were updated to the case. I generated 16 days worth of logs so they can go back and look at a time when the backups just worked. I have three backup jobs. In job 01 4 VM's failed out of 34. In job 02 3 VM's failed out of 13, but in the retry operation 2 of those failures were successful. What's weird is the retry operation indicated the 3 failures were all the same soap error urn:internalvim25. But two were able to recover and one wasn't. Then in job 03 that soap error happen on 2 out of 11 VM's, however during the first retry operation they were able to be backed up successfully. If CBT was messed up, I doubt a retry operation would have been able to make a difference. However I reset CBT Monday anyway, all backups on Monday evening worked, then its Tuesday into Wednesday where we are seeing the failures creep up again.
kjstech
Expert
 
Posts: 142
Liked: 14 times
Joined: Fri Jan 17, 2014 4:12 pm
Full Name: Keith S

Re: V9: random soap fault- vm backup failure

Veeam Logoby davecla » Wed Feb 03, 2016 9:51 pm

Did you get a fix for this?

I have the same error.
Since upgrading to v9 I get SOAP auth errors.

In my case I have a job with 10 servers in it. Some servers will fail in each run. The servers that fail change in each run.
I'm also seeing many, many more CBT errors since upgrading to v9.

I have a call logged with veeam support, but my experience to date with veeam support is "sub optimal". I'm sure my TZ doesn't help, but still....
davecla
Influencer
 
Posts: 20
Liked: 2 times
Joined: Wed Feb 03, 2016 9:40 pm
Full Name: Dave Clarke

Re: V9: random soap fault- vm backup failure

Veeam Logoby foggy » Thu Feb 04, 2016 11:57 am

As far as I can see from the OP's case, it is still under investigation. I recommend to work with support within your case, should you be not satisfied with how it is going, you can always use the Talk to Manager button at the support portal.
foggy
Veeam Software
 
Posts: 14742
Liked: 1080 times
Joined: Mon Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson

Re: V9: random soap fault- vm backup failure

Veeam Logoby kjstech » Fri Feb 05, 2016 8:37 pm

So far I have been getting successful backups if Veeam's parallel processing is unchecked. In Veeam backup and replication, click the top left icon (sometimes referred to as the hamburger button). Click Options and then uncheck "Enable parallel processing".

Now its only been two days for me, but since then all the backups were running great. I only have two warnings on SQL servers, which is being looked at. But no more SOAP errors. I thought we would really take a performance hit from disabling parallel processing, however that's not the case. Our three nightly backup jobs are still completing well within our backup window. I'll be interested how long this weekend's synthetic full jobs will take. However we are using a Veeam accelerated data mover in an Exagrid appliance that handles a lot of that synthetic full data in itself.
kjstech
Expert
 
Posts: 142
Liked: 14 times
Joined: Fri Jan 17, 2014 4:12 pm
Full Name: Keith S

Re: V9: random soap fault- vm backup failure

Veeam Logoby Alex » Mon Feb 08, 2016 9:56 pm

Hi,

I have the same error(I think):
Code: Select all
02/08/2016 10:32:25 PM :: Error: Soap fault. A specified parameter was not correct.
startOffsetDetail: '<InvalidArgumentFault xmlns="urn:internalvim25" xsi:type="InvalidArgument"><invalidProperty>startOffset</invalidProperty></InvalidArgumentFault>', endpoint: ''
Failed to upload disk.
Agent failed to process method {DataTransfer.SyncDisk}


This started to happen when upgraded to V9

Will try the Parallel setting to Disable, just to get working backups again.

It only happens with Incremental backups, the full backups don't have that problem.
Alex
Novice
 
Posts: 4
Liked: never
Joined: Wed Jun 23, 2010 11:12 am
Full Name: Alex Bos

Re: V9: random soap fault- vm backup failure

Veeam Logoby Alex » Mon Feb 08, 2016 10:16 pm

Disabling Paralell is not the solution, just tried it. Also tried https://www.veeam.com/kb1113 , but it din't help.

it seems to only happen with the Large Disks(2+ TB) in a VM.
Alex
Novice
 
Posts: 4
Liked: never
Joined: Wed Jun 23, 2010 11:12 am
Full Name: Alex Bos

Next

Return to VMware vSphere



Who is online

Users browsing this forum: No registered users and 11 guests