V9: random soap fault- vm backup failure

VMware specific discussions

Re: V9: random soap fault- vm backup failure

Veeam Logoby PTide » Tue Feb 09, 2016 10:55 am

Hi Alex,

Kindly open a case with our support team and post your case ID here.

Thank you.
PTide
Veeam Software
 
Posts: 3019
Liked: 246 times
Joined: Tue May 19, 2015 1:46 pm

Re: V9: random soap fault- vm backup failure

Veeam Logoby kjstech » Wed Feb 10, 2016 1:15 pm 2 people like this post

Ok the next thing support had me test is to re-enable parallel processing, but on one job choose a specific Veeam proxy. Then in the backup infrastructure on that veeam proxy specify the network transport mode.

So I did that as a test and last night this job completed sucessfully (about 45 minutes FASTER than usual too).

This got me thinking... We are an NFS shop here, why not try the new direct storage access? Both of my Veeam proxies have a second nic on that storage network and can ping our EMC VNX5200 NFS interfaces. So I went into our EMC VNX5200 storage array configuration and gave both Veeam proxies IP addresses as full access to our NFS file systems. I am testing a job right now and its sucessfully backedup 11 VM's in 15 minutes 17 seconds. Now the backup already ran a few hours ago so there was only 12.8 GB of changed data out of the 745.9 GB processed and 28.5 GB read.

Tonight I will be excited to see how NFS direct access handles the jobs, and also correlate if our alert-bot text message does not go out about our website not being accessible. See with NFS and hotadd, if a machine is not on the same ESXi host as the Veeam proxy (and it won't always be since we have 2 proxies but 6 hosts), when the hotadd disk is released it stuns the VM and it looses pings / nework connectivity. So I'm hoping that this mode will help according to https://www.veeam.com/kb1681

I know that is deriving a little off topic, however maybe you can try the different transport modes. For me network worked, and nfs appeared to work as well on one job I tested it with.
kjstech
Expert
 
Posts: 142
Liked: 14 times
Joined: Fri Jan 17, 2014 4:12 pm
Full Name: Keith S

Re: V9: random soap fault- vm backup failure

Veeam Logoby kjstech » Thu Feb 11, 2016 5:39 pm 2 people like this post

Wow Direct NFS access is great. We didn't get any text message notifications of machines loosing ping during the backup window. We also halved our backup window. 10 gig port now is showing close to 10 gig going to the exagrid appliance. Previously it was a little less than half that.

The only issue is that both NFS and NBD failover could not access a VM on the same NFS datastore of other successful backups. The path would be vnxfs1\SolarWinds Log & Event Manager\SolarWinds Log +Jg-Event Manager.vmdk.

Logs sent on support ticket.

No CBT or SOAP errors at all in this transport mode.
kjstech
Expert
 
Posts: 142
Liked: 14 times
Joined: Fri Jan 17, 2014 4:12 pm
Full Name: Keith S

Re: V9: random soap fault- vm backup failure

Veeam Logoby davecla » Sun Feb 14, 2016 9:43 pm

So in my case the SOAP Auth errors just stopped after about a week.
As far as I can tell nothing changed in the ESX or Veeam environments over that time which could have impacted on the backup process.

Strange...
davecla
Influencer
 
Posts: 20
Liked: 2 times
Joined: Wed Feb 03, 2016 9:40 pm
Full Name: Dave Clarke

Re: V9: random soap fault- vm backup failure

Veeam Logoby Gostev » Mon Feb 15, 2016 10:40 pm

These issues are currently suspected to be caused by intermittent failures in vCenter SSL certificate validation process, so the root cause of the issue is likely outside of our code. We're currently trying to confirm that in one of the affected environments by running a temp hot fix that disables certificate validation completely. I asked devs to keep me posted on their findings.
Gostev
Veeam Software
 
Posts: 21390
Liked: 2349 times
Joined: Sun Jan 01, 2006 1:01 am
Location: Baar, Switzerland

Re: V9: random soap fault- vm backup failure

Veeam Logoby kjstech » Wed Feb 17, 2016 4:52 pm 2 people like this post

Thanks Gostev,

We were able to close the case today. Since forcing transport to Direct Storage we haven't had a single SOAP error.
Were an NFS shop so we've also reaped huge benefits from not having disruptive VM STUN times during backup completion due to the hot add transport mode. We've also halved our backup window as the Direct Storage transport has proven to be almost twice as fast in throughput.

We had one issue with a VM having an ampersand (&) in the name, but the fix was to rename it in vSphere, then vmotion it to another filesystem. Storage vMotion takes care of renaming all the file paths. In Veeam we moved this particular VM to another job that is tied to that filesystem and it was sucessful.

So in our case changing the transport mode worked. Initially disabling parallel processing helped, but support had us do some tests with it re-enabled but using a different transport method. I came across the support for NFS direct transport in V9 and gave it a shot. THANK YOU!!!

For reference we are on the following vmware builds
ESXi 5.0.0, 2312428
vCenter Server 5.0.0, 2656067

Storage is NFS on an EMC VNX5200.
Backup is to an Exagrid appliance using the Veeam Accelerated Data Mover.
kjstech
Expert
 
Posts: 142
Liked: 14 times
Joined: Fri Jan 17, 2014 4:12 pm
Full Name: Keith S

Previous

Return to VMware vSphere



Who is online

Users browsing this forum: Bing [Bot] and 35 guests