Transmission pipeline hanged

[perjonsson1960]

Folks,

Last week we "sucked in" our vCenter server into VMware. Before that it was a physical Windows server. After this change, every other day or so, we are getting this error during backup:

"Transmission pipeline hanged, aborting process."

We have a large backup job that backs up 215 VMs, and sometimes one of the VMs (but not the same VM every time) gets this error. And B&R does a retry of that VM, and the retry succeeds.

This error never occured when we had the physical Windows server as the vCenter server. Does anybody know what this is all about?

PJ

[Mike Resseler]

Hi Per,

That message could mean a few things I think. In this case, I would look in the direction of the VM not being reachable or talking back. Maybe because the bigger issue here is the communication with VCenter where there is an issue or so. Difficult to say with only this one message.

I would suggest to open a support call (with the regular post the support case ID here and the outcome after that) so the engineers can look at the messages before and after this in the logs. They might (no promises as this typical message is very generic) see where the communication is lost or hanging

Cheers
Mike

[HannesK]

Hello,
I suggest contacting support for that. I just searched a little bit internally and there seem to be various reasons for that from corrupted VMs to network and resource issues. So it might be a coincidence between P2V of your VCenter.

Please post the case number here for reference.

Best regards,
Hannes

[trevorgov]

Good day,

Please can someone assist, for the last 3 weeks now we are not able to backup our exchange server which is a vm on VMWare ESXi. The backup starts fine and runs till about 20% and then just hangs and eventually fails after 5 hours or so with the error, 'Transmission pipeline hanged, aborting process." We have 2 other jobs running on the same Veeam management PC and they run fine, I have tried using different proxies, different repositories, even created another management PC all return the same result. However if I do an agent backup from within Exchange 2016, that goes through fine.I have reported the case to Veeam support but I am just concerned that we are sitting ducks without a proper backup for the last 3 weeks.

Any suggestions would be much appreciated.

Case number : #03723972

[foggy]

Hi Trevor, if it is a production VM, I'd increase the case severity and also ask to escalate it to a higher tier, looks like there's not much progress after almost two weeks of investigation.

[trevorgov]

Thanks Foggy, the case severity is on 4 at the moment. How would I escalate the issue?

[foggy]

Severity 4 is the lowest one. To escalate the case, please use the 'Talk to a Manager' option on the support portal. Thanks!

[trevorgov]

Perfect, thanks Foggy. They requested a whole load of logs but it has been escalated.

[rodextreme]

I'm facing the same problem, case opened - 03782873

[KOPFteam]

I had this problem solved by Veeam support yesterday (case 03838384).

They first suspected a problem with CBT. I disabled CBT for this VM + the used drives and deleted a few files in the datastore as specified in the VMware KB article. Didn't help. Still "Transmission pipline hanged".

Then they asked me to activate nbdssl = "Enable host to proxy traffic encyption in Network mode (NBDSSL)" in the properties of the used proxy (in the Transport Mode section). And this did it! No more "Transmission pipline hanged"!

Problem solved

[bdufour]

before we reinvent the wheel, try vmotioning exchange to another host, and see if that solves the issue or not.

[perjonsson1960]

The "Enable host to proxy traffic encyption in Network mode" didn't solve it for me. I still get "Transmission pipline hanged, aborting process" once in a while. Maybe once a week, and not with the same VM every time, but all the affected VMs reside in the same Esxi host. Maybe there is a problem with that host. And sometimes I also get "Error: VSSControl: Failed to freeze guest over network, wait timeout" with VMs on that Esxi host.

[veremin]

Have you tried to resolve the issue via support team? They should be able to identify the root cause, based on the infrastructure and debug log investigation. Thanks!

[perjonsson1960]

I tried to submit a case, but when it had been uploading the 2 GB zip file with logs for about an hour, and the page refreshed, and the "Next" button was grayed out, I got irritated and skipped the upload. But when I continued to submit the case, it failed, and no case was even created. Perhaps I will try again at another time.

[perjonsson1960]

Case #03860915.

[perjonsson1960]

The solution from support was to split our large backup job with 215 VMs into two or three separate jobs, and I did that yesterday. As long as we had a physical vCenter server the large job worked well, but since we "sucked in" the vCenter server into our VxRail environment, the pipeline errors started. The original job now has 78 VMs, and the two new jobs have 67 and 70 VMs respectively.

The original job (incremental backups) went well. The second new job (full backups) got two "pipeline errors" and two "VDDK async operation errors". And the third new job (full backups) also went well. When the first retry was performed on the four affected VMs, it succeeded. The three jobs were run after one another, and no other jobs were running at the same time.

New case #03865617.

[timgowen]

I have noted this issue on an Agent backup. I back up folders on a file server - all fine - but then the system backup - which backs up everything, fails with 20/04/2021 08:35:30 :: Transmission pipeline hanged, aborting process.

[foggy]

Hi Tim, please investigate with the help of our support team. Thanks!