Discussions specific to the VMware vSphere hypervisor
rurouni
Enthusiast
Posts: 56
Liked: 6 times
Joined: Jul 24, 2013 7:21 pm
Contact:

Backup Job Failed due to closed underlying connection

Post by rurouni » Mar 03, 2015 6:05 pm

Hi all,
We're suffering with our backups for weeks since some backups Jobs fail with this error:

"Error: The underlying connection was closed: A connection that was expected to be kept alive was closed by the server."
Job has failed. See logs for details.


Case #00743334 has been opened on february and i must say that we're still completely stuck. We planned to give access to our servers to support team but we could never arrange a meeting with the support engineer.

We're struggling to find the root cause of these backup failure and it has a huge impact on our backup reliability, especially during Full backups in the week end.

On the other hand some other side effects occurs like these:
- full backups are triggered on monday though they re scheduled on Saturday only.(Nothing On Sunday)
- backup windows are not respected.

I can understand that it is related to vcenter connectivity but when we check at vcenter logs and its database, we can't find any service interruption during backup hours.

Can someone provide further help on this case?

Thank you.

ivordillen
Enthusiast
Posts: 62
Liked: never
Joined: Nov 03, 2011 2:55 pm
Full Name: Ivor Dillen
Contact:

Re: Backup Job Failed due to closed underlying connection

Post by ivordillen » Mar 04, 2015 3:42 pm

Hello,

Do you use storage integration?

Ivor

Vitaliy S.
Product Manager
Posts: 22278
Liked: 1410 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: Backup Job Failed due to closed underlying connection

Post by Vitaliy S. » Mar 04, 2015 5:02 pm

Can you confirm the same behavior when standalone hosts are added to the Veeam backup console? As a troubleshooting step, you can add one of the hosts by IP address and then configure a backup job through this connection. If the issue still persists, then vCenter Server is out of the scope in this problem.

rurouni
Enthusiast
Posts: 56
Liked: 6 times
Joined: Jul 24, 2013 7:21 pm
Contact:

Re: Backup Job Failed due to closed underlying connection

Post by rurouni » Mar 04, 2015 5:20 pm

Hi,

@Ivor: Yes, we re using storage Integration on 3par 7400 Array.

@Vitaliy: We'll try that on impacted vms in one job. after registering a standalone host, do we have to re-register the vms in the impacted job to point them on the standalone Host? If so, Will it reset Retention due to consideration of those vms being new one?

Thank you.

ivordillen
Enthusiast
Posts: 62
Liked: never
Joined: Nov 03, 2011 2:55 pm
Full Name: Ivor Dillen
Contact:

Re: Backup Job Failed due to closed underlying connection

Post by ivordillen » Mar 04, 2015 6:34 pm

I also have 2 3par 7400 arrays .

The vcenter server is not in the job and the vcenter vm is on local storage (sql is on other vlun/datastore on other esxi host) and I had the issue. (the vcenter server is on a host that has some vm's snapshotted)...

For the moment I think it is the length of the first step of the job (61 vm's) all snapshotting, renewing.... that there is a threshold that is passed. For a couple of days the job goes wel but I have done nothing. I have other jobs with fewer vm's and I never ever had issues with them. It is this one job.

Our setup is vcenter 5.0.0 (last version).

Hope we can figure it out here :-)

kind regards,

Ivor

OK, the job failed ....

Vitaliy S.
Product Manager
Posts: 22278
Liked: 1410 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: Backup Job Failed due to closed underlying connection

Post by Vitaliy S. » Mar 04, 2015 9:17 pm

rurouni wrote:@Vitaliy: We'll try that on impacted vms in one job. after registering a standalone host, do we have to re-register the vms in the impacted job to point them on the standalone Host? If so, Will it reset Retention due to consideration of those vms being new one?
I would suggest just to use a separate job in order not to mess up the original job with its settings. If you backup the impacted VMs via standalone host connection, they will be treated as new VMs (due to VMware moref change) and full backup will be made.

rurouni
Enthusiast
Posts: 56
Liked: 6 times
Joined: Jul 24, 2013 7:21 pm
Contact:

Re: Backup Job Failed due to closed underlying connection

Post by rurouni » Mar 07, 2015 7:37 pm

I ll try to create the same job as the one failing but bases on host sélection, not vcenter and i Will update the résulte. The job totally failed This week end during full backups.

We re in vsphere 5.1 last update with veeam 8 patch 1. I remember having This kind of issues in v7 patch 4 also.

rurouni
Enthusiast
Posts: 56
Liked: 6 times
Joined: Jul 24, 2013 7:21 pm
Contact:

Re: Backup Job Failed due to closed underlying connection

Post by rurouni » Mar 08, 2015 9:50 am

While Inspecting the Backup Failures during the Week end i noticed that lots of jobs starting between 20:00 and Midnight are failing with the same error. Even on retry 10 min later.

Looking at the logs i found that the jobs fail with this error:

[EDIT] Logs have been removed by moderator

Which leads me to ask myself a question. Is there any Number of Soap connections limitation we can use at the same time? Else i can't explain why some jobs are running, some other don't. After 8PM that s probably the most critical time for our backups since many jobs start or run at the same time.

Is there a way i can check availability of soap connections?

I'm still investigating. Hope these logs will help you help me.

Vitaliy S.
Product Manager
Posts: 22278
Liked: 1410 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: Backup Job Failed due to closed underlying connection

Post by Vitaliy S. » Mar 08, 2015 10:09 am

Please avoid posting log snippets as it is requested by our forum rules. As to the number of SOAP connections at the same time, then as far as I know there is such limitation. Let me double check with our dev team tomorrow, but it seems like you've nailed it.

20100
Enthusiast
Posts: 72
Liked: 6 times
Joined: Feb 16, 2012 9:23 pm
Full Name: qq
Contact:

Re: Backup Job Failed due to closed underlying connection

Post by 20100 » Mar 08, 2015 8:37 pm

Hi

We have been experiencing a similar issue.
We are also using V8 SP1. We are new to Veeam, so this is a new setup.
We also use 3PAR storage and ESX 5.5
Our weekly backup contains about 40VM. last week-end and this week-end, a couple of VM experienced the same problem. Different VM every time, so this is not limited to a specific one. Also nothing changed from one week to the other, but the VM that had issues last week was fine this week-end!

I cannot figure out what it could be. The only thing I can observe from the 2 backups is that the VM that had problems are either the 1st, 2nd or 3rd to be backed-up. Not sure if it is a coincidence.

I would be interrested to hear about Soap connections too and if there is a fix/tweak

Cheers

ivordillen
Enthusiast
Posts: 62
Liked: never
Joined: Nov 03, 2011 2:55 pm
Full Name: Ivor Dillen
Contact:

Re: Backup Job Failed due to closed underlying connection

Post by ivordillen » Mar 08, 2015 9:47 pm

I gave support the case number of rurouni and they confirm that the issue is the same. R&D is investigating.
I give some more info when I get some.
Ivor

rurouni
Enthusiast
Posts: 56
Liked: 6 times
Joined: Jul 24, 2013 7:21 pm
Contact:

Re: Backup Job Failed due to closed underlying connection

Post by rurouni » Mar 09, 2015 7:54 am

Sorry About the logs extract but i wanted to point something here since i can't get to go further with support team.

Reading the comments makes me realize that thing we have in common here is a 3par Storeserv 7400. Ours has been updated recently to FW: 3.2.1(MU1).
We're using vsphere 5.1 Build 2306356
veeam B&R 8 Patch 1.

Meanwhile i can't find any issue about storage array connectivity in the job logs so i guess the point is not here.

Vitaliy S.
Product Manager
Posts: 22278
Liked: 1410 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: Backup Job Failed due to closed underlying connection

Post by Vitaliy S. » Mar 09, 2015 10:34 am

Thanks Ivor for the update.

@rurouni these kind of issues are hard to troubleshoot over the forums, so let's continue to work with our support team.

rurouni
Enthusiast
Posts: 56
Liked: 6 times
Joined: Jul 24, 2013 7:21 pm
Contact:

Re: Backup Job Failed due to closed underlying connection

Post by rurouni » Mar 09, 2015 11:29 am

I also noted that the First error to come out on multiple failing jobs is from this type:
"<17> Error [ViJobSanSource] Exception occured while preparing vm source for task 'VM_NAME'"

Do Ivor and 20100 have the same issue in their logs?

20100
Enthusiast
Posts: 72
Liked: 6 times
Joined: Feb 16, 2012 9:23 pm
Full Name: qq
Contact:

Re: Backup Job Failed due to closed underlying connection

Post by 20100 » Mar 09, 2015 8:17 pm

Hi

Rurouni, which logs are you looking at? I went through many in C:\programData\veeam\backup and could not even find where it is logged with the original error!

Vitaliy, I have not logged a case with support for this yet. Should I? and should I mention Rurouni's case too?

Post Reply

Who is online

Users browsing this forum: csydas, restore-helper and 21 guests