Backup Job Failed due to closed underlying connection

[rurouni]

Hi all,
We're suffering with our backups for weeks since some backups Jobs fail with this error:

"Error: The underlying connection was closed: A connection that was expected to be kept alive was closed by the server."
Job has failed. See logs for details.


Case #00743334 has been opened on february and i must say that we're still completely stuck. We planned to give access to our servers to support team but we could never arrange a meeting with the support engineer.

We're struggling to find the root cause of these backup failure and it has a huge impact on our backup reliability, especially during Full backups in the week end.

On the other hand some other side effects occurs like these:
- full backups are triggered on monday though they re scheduled on Saturday only.(Nothing On Sunday)
- backup windows are not respected.

I can understand that it is related to vcenter connectivity but when we check at vcenter logs and its database, we can't find any service interruption during backup hours.

Can someone provide further help on this case?

Thank you.

[ivordillen]

Hello,

Do you use storage integration?

Ivor

[Vitaliy S.]

Can you confirm the same behavior when standalone hosts are added to the Veeam backup console? As a troubleshooting step, you can add one of the hosts by IP address and then configure a backup job through this connection. If the issue still persists, then vCenter Server is out of the scope in this problem.

[rurouni]

Hi,

@Ivor: Yes, we re using storage Integration on 3par 7400 Array.

@Vitaliy: We'll try that on impacted vms in one job. after registering a standalone host, do we have to re-register the vms in the impacted job to point them on the standalone Host? If so, Will it reset Retention due to consideration of those vms being new one?

Thank you.

[ivordillen]

I also have 2 3par 7400 arrays .

The vcenter server is not in the job and the vcenter vm is on local storage (sql is on other vlun/datastore on other esxi host) and I had the issue. (the vcenter server is on a host that has some vm's snapshotted)...

For the moment I think it is the length of the first step of the job (61 vm's) all snapshotting, renewing.... that there is a threshold that is passed. For a couple of days the job goes wel but I have done nothing. I have other jobs with fewer vm's and I never ever had issues with them. It is this one job.

Our setup is vcenter 5.0.0 (last version).

Hope we can figure it out here

kind regards,

Ivor

OK, the job failed ....

[Vitaliy S.]

@Vitaliy: We'll try that on impacted vms in one job. after registering a standalone host, do we have to re-register the vms in the impacted job to point them on the standalone Host? If so, Will it reset Retention due to consideration of those vms being new one?

I would suggest just to use a separate job in order not to mess up the original job with its settings. If you backup the impacted VMs via standalone host connection, they will be treated as new VMs (due to VMware moref change) and full backup will be made.

[rurouni]

I ll try to create the same job as the one failing but bases on host sélection, not vcenter and i Will update the résulte. The job totally failed This week end during full backups.

We re in vsphere 5.1 last update with veeam 8 patch 1. I remember having This kind of issues in v7 patch 4 also.

[rurouni]

While Inspecting the Backup Failures during the Week end i noticed that lots of jobs starting between 20:00 and Midnight are failing with the same error. Even on retry 10 min later.

Looking at the logs i found that the jobs fail with this error:

[EDIT] Logs have been removed by moderator

Which leads me to ask myself a question. Is there any Number of Soap connections limitation we can use at the same time? Else i can't explain why some jobs are running, some other don't. After 8PM that s probably the most critical time for our backups since many jobs start or run at the same time.

Is there a way i can check availability of soap connections?

I'm still investigating. Hope these logs will help you help me.

[Vitaliy S.]

Please avoid posting log snippets as it is requested by our forum rules. As to the number of SOAP connections at the same time, then as far as I know there is such limitation. Let me double check with our dev team tomorrow, but it seems like you've nailed it.

[20100]

Hi

We have been experiencing a similar issue.
We are also using V8 SP1. We are new to Veeam, so this is a new setup.
We also use 3PAR storage and ESX 5.5
Our weekly backup contains about 40VM. last week-end and this week-end, a couple of VM experienced the same problem. Different VM every time, so this is not limited to a specific one. Also nothing changed from one week to the other, but the VM that had issues last week was fine this week-end!

I cannot figure out what it could be. The only thing I can observe from the 2 backups is that the VM that had problems are either the 1st, 2nd or 3rd to be backed-up. Not sure if it is a coincidence.

I would be interrested to hear about Soap connections too and if there is a fix/tweak

Cheers

[ivordillen]

I gave support the case number of rurouni and they confirm that the issue is the same. R&D is investigating.
I give some more info when I get some.
Ivor

[rurouni]

Sorry About the logs extract but i wanted to point something here since i can't get to go further with support team.

Reading the comments makes me realize that thing we have in common here is a 3par Storeserv 7400. Ours has been updated recently to FW: 3.2.1(MU1).
We're using vsphere 5.1 Build 2306356
veeam B&R 8 Patch 1.

Meanwhile i can't find any issue about storage array connectivity in the job logs so i guess the point is not here.

[Vitaliy S.]

Thanks Ivor for the update.

@rurouni these kind of issues are hard to troubleshoot over the forums, so let's continue to work with our support team.

[rurouni]

I also noted that the First error to come out on multiple failing jobs is from this type:
"<17> Error [ViJobSanSource] Exception occured while preparing vm source for task 'VM_NAME'"

Do Ivor and 20100 have the same issue in their logs?

[20100]

Hi

Rurouni, which logs are you looking at? I went through many in C:\programData\veeam\backup and could not even find where it is logged with the original error!

Vitaliy, I have not logged a case with support for this yet. Should I? and should I mention Rurouni's case too?

[rurouni]

20100,
I'm comparing the vpxd.log file from the vcenter log folder and the Veeam Backup Job log folder. Inside you should see a Log Folder with the name of your failing job.Inside you can search the file "Job.Name.Backup.log"
Look at the errors like "underlying connection" was closed. or the latest one i posted Error [ViJobSanSource].

Regards

[veremin]

Vitaliy, I have not logged a case with support for this yet. Should I?

Please, do.

Should I mention Rurouni's case too?

Yep, might give support some directions.

Thanks.

[20100]

OK. I got case 00828216 for it.

[veremin]

Thank you for providing the ticket number. Let's wait and see what support says, once logs are analyzed.

[20100]

Hi

Not much from support yet. I get one email every couple of days, usually at night time. I usually reply straigh away but have to wait another day or 2 to have a reply!).
Anyway so far, I have been asked: "restart Vcenter management center", "do you have antivirus or/and firewall on the VM?", etc.

The issue is intermittent and not a huge issue. the job does not fail, it comes only with a warning. Once it happened when I manually ran the backup and I was in front the screen. It looked to me like the snapshot from Vmware to 3Par back to Vmware was the part that failed, and instead it only did a Vmware snapshot. However I could be wrong in what I observed.

Rurouni, do you had more chance with your case?

Thanks

[rurouni]

Hi, for me solution has been provided by support using a registry key which limits the number of simultaneous snapshot performed on the vcenter until devs provide a new patch for v8

[20100]

Thanks Rurouni

I will chase our case support. I did not hear from them about this fix...

[veremin]

The rurouni's case number is 00743334. So, you can make a reference to it via your ticket and let support engineer compare the behaviours and see whether the said fix applies to your situation. Thanks.

[20100]

The registry fix has been applied for a week now and the problem has disappeared from all backups.
Thanks again for your help

[veremin]

You're welcome. Let's know, should other help be needed.

[ivordillen]

I got an update for the case that they are not going to fix it but I have to wait until the new patch end of april

[ivordillen]

the new update has not fiexed my issue.

[foggy]

Ivor, did you let your engineer know about that? What is your support case ID number?

[alando]

Any news on this issue? This is driving me nuts.

Neither upgrading vCenter to 5.5u3 nor applying patch 2 (Veeam B&R 8.0.0.2030) brought any help on this topic.

Since we switched onto a 3par all flash array we have this issue, nobody seems to know what causes it.

[foggy]

Have you contacted Veeam tech support with this issue?