Host-based backup of VMware vSphere VMs.
Post Reply
perjonsson1960
Veteran
Posts: 443
Liked: 44 times
Joined: Jun 06, 2018 5:41 am
Full Name: Per Jonsson
Location: Sweden
Contact:

Transmission pipeline hanged

Post by perjonsson1960 »

Folks,

Last week we "sucked in" our vCenter server into VMware. Before that it was a physical Windows server. After this change, every other day or so, we are getting this error during backup:

"Transmission pipeline hanged, aborting process."

We have a large backup job that backs up 215 VMs, and sometimes one of the VMs (but not the same VM every time) gets this error. And B&R does a retry of that VM, and the retry succeeds.

This error never occured when we had the physical Windows server as the vCenter server. Does anybody know what this is all about?

PJ
Mike Resseler
Product Manager
Posts: 8044
Liked: 1263 times
Joined: Feb 08, 2013 3:08 pm
Full Name: Mike Resseler
Location: Belgium
Contact:

Re: Transmission pipeline hanged

Post by Mike Resseler »

Hi Per,

That message could mean a few things I think. In this case, I would look in the direction of the VM not being reachable or talking back. Maybe because the bigger issue here is the communication with VCenter where there is an issue or so. Difficult to say with only this one message.

I would suggest to open a support call (with the regular post the support case ID here and the outcome after that) so the engineers can look at the messages before and after this in the logs. They might (no promises as this typical message is very generic) see where the communication is lost or hanging

Cheers
Mike
HannesK
Product Manager
Posts: 14287
Liked: 2877 times
Joined: Sep 01, 2014 11:46 am
Full Name: Hannes Kasparick
Location: Austria
Contact:

Re: Transmission pipeline hanged

Post by HannesK »

Hello,
I suggest contacting support for that. I just searched a little bit internally and there seem to be various reasons for that from corrupted VMs to network and resource issues. So it might be a coincidence between P2V of your VCenter.

Please post the case number here for reference.

Best regards,
Hannes
trevorgov
Novice
Posts: 3
Liked: 1 time
Joined: Sep 04, 2019 9:26 am
Full Name: Trevor Govender
Contact:

[MERGED] Error: Transmission pipeline hanged, aborting process. When backing up exchange 2016 server

Post by trevorgov »

Good day,

Please can someone assist, for the last 3 weeks now we are not able to backup our exchange server which is a vm on VMWare ESXi. The backup starts fine and runs till about 20% and then just hangs and eventually fails after 5 hours or so with the error, 'Transmission pipeline hanged, aborting process." We have 2 other jobs running on the same Veeam management PC and they run fine, I have tried using different proxies, different repositories, even created another management PC all return the same result. However if I do an agent backup from within Exchange 2016, that goes through fine.I have reported the case to Veeam support but I am just concerned that we are sitting ducks without a proper backup for the last 3 weeks.

Any suggestions would be much appreciated.

Case number : #03723972
foggy
Veeam Software
Posts: 21069
Liked: 2115 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Transmission pipeline hanged

Post by foggy »

Hi Trevor, if it is a production VM, I'd increase the case severity and also ask to escalate it to a higher tier, looks like there's not much progress after almost two weeks of investigation.
trevorgov
Novice
Posts: 3
Liked: 1 time
Joined: Sep 04, 2019 9:26 am
Full Name: Trevor Govender
Contact:

Re: Transmission pipeline hanged

Post by trevorgov »

Thanks Foggy, the case severity is on 4 at the moment. How would I escalate the issue?
foggy
Veeam Software
Posts: 21069
Liked: 2115 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Transmission pipeline hanged

Post by foggy »

Severity 4 is the lowest one. To escalate the case, please use the 'Talk to a Manager' option on the support portal. Thanks!
trevorgov
Novice
Posts: 3
Liked: 1 time
Joined: Sep 04, 2019 9:26 am
Full Name: Trevor Govender
Contact:

Re: Transmission pipeline hanged

Post by trevorgov » 1 person likes this post

Perfect, thanks Foggy. They requested a whole load of logs but it has been escalated.
rodextreme
Lurker
Posts: 1
Liked: never
Joined: May 08, 2019 4:18 pm
Full Name: Rodrigo Dantas
Contact:

Re: Transmission pipeline hanged

Post by rodextreme »

I'm facing the same problem, case opened - 03782873
KOPFteam
Enthusiast
Posts: 53
Liked: 7 times
Joined: Sep 30, 2011 6:59 pm
Full Name: Florian Pürner
Contact:

Re: Transmission pipeline hanged

Post by KOPFteam »

I had this problem solved by Veeam support yesterday (case 03838384).

They first suspected a problem with CBT. I disabled CBT for this VM + the used drives and deleted a few files in the datastore as specified in the VMware KB article. Didn't help. Still "Transmission pipline hanged".

Then they asked me to activate nbdssl = "Enable host to proxy traffic encyption in Network mode (NBDSSL)" in the properties of the used proxy (in the Transport Mode section). And this did it! No more "Transmission pipline hanged"!

Problem solved :-)
Loving (and fighting with) Veeam for over a decade!
bdufour
Expert
Posts: 206
Liked: 41 times
Joined: Nov 01, 2017 8:52 pm
Full Name: blake dufour
Contact:

Re: Transmission pipeline hanged

Post by bdufour »

before we reinvent the wheel, try vmotioning exchange to another host, and see if that solves the issue or not.
perjonsson1960
Veteran
Posts: 443
Liked: 44 times
Joined: Jun 06, 2018 5:41 am
Full Name: Per Jonsson
Location: Sweden
Contact:

Re: Transmission pipeline hanged

Post by perjonsson1960 »

The "Enable host to proxy traffic encyption in Network mode" didn't solve it for me. I still get "Transmission pipline hanged, aborting process" once in a while. Maybe once a week, and not with the same VM every time, but all the affected VMs reside in the same Esxi host. Maybe there is a problem with that host. And sometimes I also get "Error: VSSControl: Failed to freeze guest over network, wait timeout" with VMs on that Esxi host.
veremin
Product Manager
Posts: 20270
Liked: 2252 times
Joined: Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin
Contact:

Re: Transmission pipeline hanged

Post by veremin »

Have you tried to resolve the issue via support team? They should be able to identify the root cause, based on the infrastructure and debug log investigation. Thanks!
perjonsson1960
Veteran
Posts: 443
Liked: 44 times
Joined: Jun 06, 2018 5:41 am
Full Name: Per Jonsson
Location: Sweden
Contact:

Re: Transmission pipeline hanged

Post by perjonsson1960 »

I tried to submit a case, but when it had been uploading the 2 GB zip file with logs for about an hour, and the page refreshed, and the "Next" button was grayed out, I got irritated and skipped the upload. But when I continued to submit the case, it failed, and no case was even created. Perhaps I will try again at another time.
perjonsson1960
Veteran
Posts: 443
Liked: 44 times
Joined: Jun 06, 2018 5:41 am
Full Name: Per Jonsson
Location: Sweden
Contact:

Re: Transmission pipeline hanged

Post by perjonsson1960 »

Case #03860915.
perjonsson1960
Veteran
Posts: 443
Liked: 44 times
Joined: Jun 06, 2018 5:41 am
Full Name: Per Jonsson
Location: Sweden
Contact:

Re: Transmission pipeline hanged

Post by perjonsson1960 »

The solution from support was to split our large backup job with 215 VMs into two or three separate jobs, and I did that yesterday. As long as we had a physical vCenter server the large job worked well, but since we "sucked in" the vCenter server into our VxRail environment, the pipeline errors started. The original job now has 78 VMs, and the two new jobs have 67 and 70 VMs respectively.

The original job (incremental backups) went well. The second new job (full backups) got two "pipeline errors" and two "VDDK async operation errors". And the third new job (full backups) also went well. When the first retry was performed on the four affected VMs, it succeeded. The three jobs were run after one another, and no other jobs were running at the same time.

New case #03865617.
timgowen
Influencer
Posts: 20
Liked: never
Joined: Aug 22, 2017 8:34 am
Full Name: Tim Gowen
Contact:

Re: Transmission pipeline hanged

Post by timgowen »

I have noted this issue on an Agent backup. I back up folders on a file server - all fine - but then the system backup - which backs up everything, fails with 20/04/2021 08:35:30 :: Transmission pipeline hanged, aborting process.
foggy
Veeam Software
Posts: 21069
Liked: 2115 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Transmission pipeline hanged

Post by foggy »

Hi Tim, please investigate with the help of our support team. Thanks!
Post Reply

Who is online

Users browsing this forum: Google [Bot] and 65 guests