Host-based backup of VMware vSphere VMs.
rurouni
Enthusiast
Posts: 68
Liked: 6 times
Joined: Jul 24, 2013 7:21 pm
Contact:

Backup Job Failed due to closed underlying connection

Post by rurouni »

Hi all,
We're suffering with our backups for weeks since some backups Jobs fail with this error:

"Error: The underlying connection was closed: A connection that was expected to be kept alive was closed by the server."
Job has failed. See logs for details.


Case #00743334 has been opened on february and i must say that we're still completely stuck. We planned to give access to our servers to support team but we could never arrange a meeting with the support engineer.

We're struggling to find the root cause of these backup failure and it has a huge impact on our backup reliability, especially during Full backups in the week end.

On the other hand some other side effects occurs like these:
- full backups are triggered on monday though they re scheduled on Saturday only.(Nothing On Sunday)
- backup windows are not respected.

I can understand that it is related to vcenter connectivity but when we check at vcenter logs and its database, we can't find any service interruption during backup hours.

Can someone provide further help on this case?

Thank you.
ivordillen
Enthusiast
Posts: 62
Liked: never
Joined: Nov 03, 2011 2:55 pm
Full Name: Ivor Dillen
Contact:

Re: Backup Job Failed due to closed underlying connection

Post by ivordillen »

Hello,

Do you use storage integration?

Ivor
Vitaliy S.
VP, Product Management
Posts: 27055
Liked: 2710 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: Backup Job Failed due to closed underlying connection

Post by Vitaliy S. »

Can you confirm the same behavior when standalone hosts are added to the Veeam backup console? As a troubleshooting step, you can add one of the hosts by IP address and then configure a backup job through this connection. If the issue still persists, then vCenter Server is out of the scope in this problem.
rurouni
Enthusiast
Posts: 68
Liked: 6 times
Joined: Jul 24, 2013 7:21 pm
Contact:

Re: Backup Job Failed due to closed underlying connection

Post by rurouni »

Hi,

@Ivor: Yes, we re using storage Integration on 3par 7400 Array.

@Vitaliy: We'll try that on impacted vms in one job. after registering a standalone host, do we have to re-register the vms in the impacted job to point them on the standalone Host? If so, Will it reset Retention due to consideration of those vms being new one?

Thank you.
ivordillen
Enthusiast
Posts: 62
Liked: never
Joined: Nov 03, 2011 2:55 pm
Full Name: Ivor Dillen
Contact:

Re: Backup Job Failed due to closed underlying connection

Post by ivordillen »

I also have 2 3par 7400 arrays .

The vcenter server is not in the job and the vcenter vm is on local storage (sql is on other vlun/datastore on other esxi host) and I had the issue. (the vcenter server is on a host that has some vm's snapshotted)...

For the moment I think it is the length of the first step of the job (61 vm's) all snapshotting, renewing.... that there is a threshold that is passed. For a couple of days the job goes wel but I have done nothing. I have other jobs with fewer vm's and I never ever had issues with them. It is this one job.

Our setup is vcenter 5.0.0 (last version).

Hope we can figure it out here :-)

kind regards,

Ivor

OK, the job failed ....
Vitaliy S.
VP, Product Management
Posts: 27055
Liked: 2710 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: Backup Job Failed due to closed underlying connection

Post by Vitaliy S. »

rurouni wrote:@Vitaliy: We'll try that on impacted vms in one job. after registering a standalone host, do we have to re-register the vms in the impacted job to point them on the standalone Host? If so, Will it reset Retention due to consideration of those vms being new one?
I would suggest just to use a separate job in order not to mess up the original job with its settings. If you backup the impacted VMs via standalone host connection, they will be treated as new VMs (due to VMware moref change) and full backup will be made.
rurouni
Enthusiast
Posts: 68
Liked: 6 times
Joined: Jul 24, 2013 7:21 pm
Contact:

Re: Backup Job Failed due to closed underlying connection

Post by rurouni »

I ll try to create the same job as the one failing but bases on host sélection, not vcenter and i Will update the résulte. The job totally failed This week end during full backups.

We re in vsphere 5.1 last update with veeam 8 patch 1. I remember having This kind of issues in v7 patch 4 also.
rurouni
Enthusiast
Posts: 68
Liked: 6 times
Joined: Jul 24, 2013 7:21 pm
Contact:

Re: Backup Job Failed due to closed underlying connection

Post by rurouni »

While Inspecting the Backup Failures during the Week end i noticed that lots of jobs starting between 20:00 and Midnight are failing with the same error. Even on retry 10 min later.

Looking at the logs i found that the jobs fail with this error:

[EDIT] Logs have been removed by moderator

Which leads me to ask myself a question. Is there any Number of Soap connections limitation we can use at the same time? Else i can't explain why some jobs are running, some other don't. After 8PM that s probably the most critical time for our backups since many jobs start or run at the same time.

Is there a way i can check availability of soap connections?

I'm still investigating. Hope these logs will help you help me.
Vitaliy S.
VP, Product Management
Posts: 27055
Liked: 2710 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: Backup Job Failed due to closed underlying connection

Post by Vitaliy S. »

Please avoid posting log snippets as it is requested by our forum rules. As to the number of SOAP connections at the same time, then as far as I know there is such limitation. Let me double check with our dev team tomorrow, but it seems like you've nailed it.
20100
Enthusiast
Posts: 72
Liked: 6 times
Joined: Feb 16, 2012 9:23 pm
Full Name: qq
Contact:

Re: Backup Job Failed due to closed underlying connection

Post by 20100 »

Hi

We have been experiencing a similar issue.
We are also using V8 SP1. We are new to Veeam, so this is a new setup.
We also use 3PAR storage and ESX 5.5
Our weekly backup contains about 40VM. last week-end and this week-end, a couple of VM experienced the same problem. Different VM every time, so this is not limited to a specific one. Also nothing changed from one week to the other, but the VM that had issues last week was fine this week-end!

I cannot figure out what it could be. The only thing I can observe from the 2 backups is that the VM that had problems are either the 1st, 2nd or 3rd to be backed-up. Not sure if it is a coincidence.

I would be interrested to hear about Soap connections too and if there is a fix/tweak

Cheers
ivordillen
Enthusiast
Posts: 62
Liked: never
Joined: Nov 03, 2011 2:55 pm
Full Name: Ivor Dillen
Contact:

Re: Backup Job Failed due to closed underlying connection

Post by ivordillen »

I gave support the case number of rurouni and they confirm that the issue is the same. R&D is investigating.
I give some more info when I get some.
Ivor
rurouni
Enthusiast
Posts: 68
Liked: 6 times
Joined: Jul 24, 2013 7:21 pm
Contact:

Re: Backup Job Failed due to closed underlying connection

Post by rurouni »

Sorry About the logs extract but i wanted to point something here since i can't get to go further with support team.

Reading the comments makes me realize that thing we have in common here is a 3par Storeserv 7400. Ours has been updated recently to FW: 3.2.1(MU1).
We're using vsphere 5.1 Build 2306356
veeam B&R 8 Patch 1.

Meanwhile i can't find any issue about storage array connectivity in the job logs so i guess the point is not here.
Vitaliy S.
VP, Product Management
Posts: 27055
Liked: 2710 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: Backup Job Failed due to closed underlying connection

Post by Vitaliy S. »

Thanks Ivor for the update.

@rurouni these kind of issues are hard to troubleshoot over the forums, so let's continue to work with our support team.
rurouni
Enthusiast
Posts: 68
Liked: 6 times
Joined: Jul 24, 2013 7:21 pm
Contact:

Re: Backup Job Failed due to closed underlying connection

Post by rurouni »

I also noted that the First error to come out on multiple failing jobs is from this type:
"<17> Error [ViJobSanSource] Exception occured while preparing vm source for task 'VM_NAME'"

Do Ivor and 20100 have the same issue in their logs?
20100
Enthusiast
Posts: 72
Liked: 6 times
Joined: Feb 16, 2012 9:23 pm
Full Name: qq
Contact:

Re: Backup Job Failed due to closed underlying connection

Post by 20100 »

Hi

Rurouni, which logs are you looking at? I went through many in C:\programData\veeam\backup and could not even find where it is logged with the original error!

Vitaliy, I have not logged a case with support for this yet. Should I? and should I mention Rurouni's case too?
rurouni
Enthusiast
Posts: 68
Liked: 6 times
Joined: Jul 24, 2013 7:21 pm
Contact:

Re: Backup Job Failed due to closed underlying connection

Post by rurouni »

20100,
I'm comparing the vpxd.log file from the vcenter log folder and the Veeam Backup Job log folder. Inside you should see a Log Folder with the name of your failing job.Inside you can search the file "Job.Name.Backup.log"
Look at the errors like "underlying connection" was closed. or the latest one i posted Error [ViJobSanSource].

Regards
veremin
Product Manager
Posts: 20270
Liked: 2252 times
Joined: Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin
Contact:

Re: Backup Job Failed due to closed underlying connection

Post by veremin »

Vitaliy, I have not logged a case with support for this yet. Should I?
Please, do.
Should I mention Rurouni's case too?
Yep, might give support some directions.

Thanks.
20100
Enthusiast
Posts: 72
Liked: 6 times
Joined: Feb 16, 2012 9:23 pm
Full Name: qq
Contact:

Re: Backup Job Failed due to closed underlying connection

Post by 20100 » 1 person likes this post

OK. I got case 00828216 for it.
veremin
Product Manager
Posts: 20270
Liked: 2252 times
Joined: Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin
Contact:

Re: Backup Job Failed due to closed underlying connection

Post by veremin »

Thank you for providing the ticket number. Let's wait and see what support says, once logs are analyzed.
20100
Enthusiast
Posts: 72
Liked: 6 times
Joined: Feb 16, 2012 9:23 pm
Full Name: qq
Contact:

Re: Backup Job Failed due to closed underlying connection

Post by 20100 »

Hi

Not much from support yet. I get one email every couple of days, usually at night time. I usually reply straigh away but have to wait another day or 2 to have a reply!).
Anyway so far, I have been asked: "restart Vcenter management center", "do you have antivirus or/and firewall on the VM?", etc.

The issue is intermittent and not a huge issue. the job does not fail, it comes only with a warning. Once it happened when I manually ran the backup and I was in front the screen. It looked to me like the snapshot from Vmware to 3Par back to Vmware was the part that failed, and instead it only did a Vmware snapshot. However I could be wrong in what I observed.

Rurouni, do you had more chance with your case?

Thanks
rurouni
Enthusiast
Posts: 68
Liked: 6 times
Joined: Jul 24, 2013 7:21 pm
Contact:

Re: Backup Job Failed due to closed underlying connection

Post by rurouni » 1 person likes this post

Hi, for me solution has been provided by support using a registry key which limits the number of simultaneous snapshot performed on the vcenter until devs provide a new patch for v8
20100
Enthusiast
Posts: 72
Liked: 6 times
Joined: Feb 16, 2012 9:23 pm
Full Name: qq
Contact:

Re: Backup Job Failed due to closed underlying connection

Post by 20100 »

Thanks Rurouni

I will chase our case support. I did not hear from them about this fix...
veremin
Product Manager
Posts: 20270
Liked: 2252 times
Joined: Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin
Contact:

Re: Backup Job Failed due to closed underlying connection

Post by veremin »

The rurouni's case number is 00743334. So, you can make a reference to it via your ticket and let support engineer compare the behaviours and see whether the said fix applies to your situation. Thanks.
20100
Enthusiast
Posts: 72
Liked: 6 times
Joined: Feb 16, 2012 9:23 pm
Full Name: qq
Contact:

Re: Backup Job Failed due to closed underlying connection

Post by 20100 »

The registry fix has been applied for a week now and the problem has disappeared from all backups.
Thanks again for your help
veremin
Product Manager
Posts: 20270
Liked: 2252 times
Joined: Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin
Contact:

Re: Backup Job Failed due to closed underlying connection

Post by veremin »

You're welcome. Let's know, should other help be needed.
ivordillen
Enthusiast
Posts: 62
Liked: never
Joined: Nov 03, 2011 2:55 pm
Full Name: Ivor Dillen
Contact:

Re: Backup Job Failed due to closed underlying connection

Post by ivordillen »

I got an update for the case that they are not going to fix it but I have to wait until the new patch end of april
ivordillen
Enthusiast
Posts: 62
Liked: never
Joined: Nov 03, 2011 2:55 pm
Full Name: Ivor Dillen
Contact:

Re: Backup Job Failed due to closed underlying connection

Post by ivordillen »

the new update has not fiexed my issue.
foggy
Veeam Software
Posts: 21069
Liked: 2115 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Backup Job Failed due to closed underlying connection

Post by foggy »

Ivor, did you let your engineer know about that? What is your support case ID number?
alando
Novice
Posts: 4
Liked: 1 time
Joined: Sep 29, 2015 11:50 am
Contact:

Re: Backup Job Failed due to closed underlying connection

Post by alando »

Any news on this issue? This is driving me nuts.

Neither upgrading vCenter to 5.5u3 nor applying patch 2 (Veeam B&R 8.0.0.2030) brought any help on this topic.

Since we switched onto a 3par all flash array we have this issue, nobody seems to know what causes it.
foggy
Veeam Software
Posts: 21069
Liked: 2115 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Backup Job Failed due to closed underlying connection

Post by foggy »

Have you contacted Veeam tech support with this issue?
Post Reply

Who is online

Users browsing this forum: Lewpy, nielsengelen and 88 guests