Backup Job Failed due to closed underlying connection

VMware specific discussions

Backup Job Failed due to closed underlying connection

Veeam Logoby rurouni » Tue Mar 03, 2015 6:05 pm

Hi all,
We're suffering with our backups for weeks since some backups Jobs fail with this error:

"Error: The underlying connection was closed: A connection that was expected to be kept alive was closed by the server."
Job has failed. See logs for details.


Case #00743334 has been opened on february and i must say that we're still completely stuck. We planned to give access to our servers to support team but we could never arrange a meeting with the support engineer.

We're struggling to find the root cause of these backup failure and it has a huge impact on our backup reliability, especially during Full backups in the week end.

On the other hand some other side effects occurs like these:
- full backups are triggered on monday though they re scheduled on Saturday only.(Nothing On Sunday)
- backup windows are not respected.

I can understand that it is related to vcenter connectivity but when we check at vcenter logs and its database, we can't find any service interruption during backup hours.

Can someone provide further help on this case?

Thank you.
rurouni
Enthusiast
 
Posts: 44
Liked: 6 times
Joined: Wed Jul 24, 2013 7:21 pm

Re: Backup Job Failed due to closed underlying connection

Veeam Logoby ivordillen » Wed Mar 04, 2015 3:42 pm

Hello,

Do you use storage integration?

Ivor
ivordillen
Enthusiast
 
Posts: 61
Liked: never
Joined: Thu Nov 03, 2011 2:55 pm
Full Name: Ivor Dillen

Re: Backup Job Failed due to closed underlying connection

Veeam Logoby Vitaliy S. » Wed Mar 04, 2015 5:02 pm

Can you confirm the same behavior when standalone hosts are added to the Veeam backup console? As a troubleshooting step, you can add one of the hosts by IP address and then configure a backup job through this connection. If the issue still persists, then vCenter Server is out of the scope in this problem.
Vitaliy S.
Veeam Software
 
Posts: 19558
Liked: 1102 times
Joined: Mon Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov

Re: Backup Job Failed due to closed underlying connection

Veeam Logoby rurouni » Wed Mar 04, 2015 5:20 pm

Hi,

@Ivor: Yes, we re using storage Integration on 3par 7400 Array.

@Vitaliy: We'll try that on impacted vms in one job. after registering a standalone host, do we have to re-register the vms in the impacted job to point them on the standalone Host? If so, Will it reset Retention due to consideration of those vms being new one?

Thank you.
rurouni
Enthusiast
 
Posts: 44
Liked: 6 times
Joined: Wed Jul 24, 2013 7:21 pm

Re: Backup Job Failed due to closed underlying connection

Veeam Logoby ivordillen » Wed Mar 04, 2015 6:34 pm

I also have 2 3par 7400 arrays .

The vcenter server is not in the job and the vcenter vm is on local storage (sql is on other vlun/datastore on other esxi host) and I had the issue. (the vcenter server is on a host that has some vm's snapshotted)...

For the moment I think it is the length of the first step of the job (61 vm's) all snapshotting, renewing.... that there is a threshold that is passed. For a couple of days the job goes wel but I have done nothing. I have other jobs with fewer vm's and I never ever had issues with them. It is this one job.

Our setup is vcenter 5.0.0 (last version).

Hope we can figure it out here :-)

kind regards,

Ivor

OK, the job failed ....
ivordillen
Enthusiast
 
Posts: 61
Liked: never
Joined: Thu Nov 03, 2011 2:55 pm
Full Name: Ivor Dillen

Re: Backup Job Failed due to closed underlying connection

Veeam Logoby Vitaliy S. » Wed Mar 04, 2015 9:17 pm

rurouni wrote:@Vitaliy: We'll try that on impacted vms in one job. after registering a standalone host, do we have to re-register the vms in the impacted job to point them on the standalone Host? If so, Will it reset Retention due to consideration of those vms being new one?

I would suggest just to use a separate job in order not to mess up the original job with its settings. If you backup the impacted VMs via standalone host connection, they will be treated as new VMs (due to VMware moref change) and full backup will be made.
Vitaliy S.
Veeam Software
 
Posts: 19558
Liked: 1102 times
Joined: Mon Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov

Re: Backup Job Failed due to closed underlying connection

Veeam Logoby rurouni » Sat Mar 07, 2015 7:37 pm

I ll try to create the same job as the one failing but bases on host sélection, not vcenter and i Will update the résulte. The job totally failed This week end during full backups.

We re in vsphere 5.1 last update with veeam 8 patch 1. I remember having This kind of issues in v7 patch 4 also.
rurouni
Enthusiast
 
Posts: 44
Liked: 6 times
Joined: Wed Jul 24, 2013 7:21 pm

Re: Backup Job Failed due to closed underlying connection

Veeam Logoby rurouni » Sun Mar 08, 2015 9:50 am

While Inspecting the Backup Failures during the Week end i noticed that lots of jobs starting between 20:00 and Midnight are failing with the same error. Even on retry 10 min later.

Looking at the logs i found that the jobs fail with this error:

[EDIT] Logs have been removed by moderator

Which leads me to ask myself a question. Is there any Number of Soap connections limitation we can use at the same time? Else i can't explain why some jobs are running, some other don't. After 8PM that s probably the most critical time for our backups since many jobs start or run at the same time.

Is there a way i can check availability of soap connections?

I'm still investigating. Hope these logs will help you help me.
rurouni
Enthusiast
 
Posts: 44
Liked: 6 times
Joined: Wed Jul 24, 2013 7:21 pm

Re: Backup Job Failed due to closed underlying connection

Veeam Logoby Vitaliy S. » Sun Mar 08, 2015 10:09 am

Please avoid posting log snippets as it is requested by our forum rules. As to the number of SOAP connections at the same time, then as far as I know there is such limitation. Let me double check with our dev team tomorrow, but it seems like you've nailed it.
Vitaliy S.
Veeam Software
 
Posts: 19558
Liked: 1102 times
Joined: Mon Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov

Re: Backup Job Failed due to closed underlying connection

Veeam Logoby 20100 » Sun Mar 08, 2015 8:37 pm

Hi

We have been experiencing a similar issue.
We are also using V8 SP1. We are new to Veeam, so this is a new setup.
We also use 3PAR storage and ESX 5.5
Our weekly backup contains about 40VM. last week-end and this week-end, a couple of VM experienced the same problem. Different VM every time, so this is not limited to a specific one. Also nothing changed from one week to the other, but the VM that had issues last week was fine this week-end!

I cannot figure out what it could be. The only thing I can observe from the 2 backups is that the VM that had problems are either the 1st, 2nd or 3rd to be backed-up. Not sure if it is a coincidence.

I would be interrested to hear about Soap connections too and if there is a fix/tweak

Cheers
20100
Enthusiast
 
Posts: 69
Liked: 5 times
Joined: Thu Feb 16, 2012 9:23 pm
Full Name: qq

Re: Backup Job Failed due to closed underlying connection

Veeam Logoby ivordillen » Sun Mar 08, 2015 9:47 pm

I gave support the case number of rurouni and they confirm that the issue is the same. R&D is investigating.
I give some more info when I get some.
Ivor
ivordillen
Enthusiast
 
Posts: 61
Liked: never
Joined: Thu Nov 03, 2011 2:55 pm
Full Name: Ivor Dillen

Re: Backup Job Failed due to closed underlying connection

Veeam Logoby rurouni » Mon Mar 09, 2015 7:54 am

Sorry About the logs extract but i wanted to point something here since i can't get to go further with support team.

Reading the comments makes me realize that thing we have in common here is a 3par Storeserv 7400. Ours has been updated recently to FW: 3.2.1(MU1).
We're using vsphere 5.1 Build 2306356
veeam B&R 8 Patch 1.

Meanwhile i can't find any issue about storage array connectivity in the job logs so i guess the point is not here.
rurouni
Enthusiast
 
Posts: 44
Liked: 6 times
Joined: Wed Jul 24, 2013 7:21 pm

Re: Backup Job Failed due to closed underlying connection

Veeam Logoby Vitaliy S. » Mon Mar 09, 2015 10:34 am

Thanks Ivor for the update.

@rurouni these kind of issues are hard to troubleshoot over the forums, so let's continue to work with our support team.
Vitaliy S.
Veeam Software
 
Posts: 19558
Liked: 1102 times
Joined: Mon Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov

Re: Backup Job Failed due to closed underlying connection

Veeam Logoby rurouni » Mon Mar 09, 2015 11:29 am

I also noted that the First error to come out on multiple failing jobs is from this type:
"<17> Error [ViJobSanSource] Exception occured while preparing vm source for task 'VM_NAME'"

Do Ivor and 20100 have the same issue in their logs?
rurouni
Enthusiast
 
Posts: 44
Liked: 6 times
Joined: Wed Jul 24, 2013 7:21 pm

Re: Backup Job Failed due to closed underlying connection

Veeam Logoby 20100 » Mon Mar 09, 2015 8:17 pm

Hi

Rurouni, which logs are you looking at? I went through many in C:\programData\veeam\backup and could not even find where it is logged with the original error!

Vitaliy, I have not logged a case with support for this yet. Should I? and should I mention Rurouni's case too?
20100
Enthusiast
 
Posts: 69
Liked: 5 times
Joined: Thu Feb 16, 2012 9:23 pm
Full Name: qq

Next

Return to VMware vSphere



Who is online

Users browsing this forum: Google [Bot], Majestic-12 [Bot] and 3 guests