[ID #03954027] Transmission pipeline hanged

KamSandhu44 · Post by **KamSandhu44** » Jan 15, 2020 8:54 am this post

Hi all,

I am having issue's backing up two of my HyperV hosted VMs. I have the latest version of Veeam Backup & Replication (the community edition).

Two Vms are failing with Transmission pipeline hanged, aborting process. (Veeam.Backup.Common.CCppComponentException).

Snipet of the log: -

Code: Select all

[14.01.2020 22:18:08] <13> Info                   [AP] (8ff9) output: --asyncNtf:Pipeline timeout: it seems that pipeline hanged.
[14.01.2020 22:28:27] <65> Info     [AP] Waiting for completion. Id: [0x2c70e5a], Time: 00:30:00.0000108
[14.01.2020 22:46:24] <52> Info                   [AP] (8ff9) output: --asyncNtf:Pipeline timeout: it seems that pipeline hanged.
[14.01.2020 22:46:24] <83> Info                   [AP] (fe9f) output: --asyncNtf:Pipeline timeout: it seems that pipeline hanged.
[14.01.2020 22:46:24] <47> Info                   [AP] (1e3f) output: --asyncNtf:Pipeline timeout: it seems that pipeline hanged.
[14.01.2020 22:46:24] <36> Info                   [AP] (b53c) output: --asyncNtf:Pipeline timeout: it seems that pipeline hanged.
[14.01.2020 22:48:08] <36> Info                   [AP] (8ff9) output: --asyncNtf:Pipeline timeout: it seems that pipeline hanged.
[14.01.2020 22:56:23] <23> Info                   [AP] (1e3f) output: --asyncNtf:--busy: Target backup repository is overloaded.
[14.01.2020 22:56:23] <27> Info                   [AP] (fe9f) output: --asyncNtf:--busy: Target backup repository is overloaded.
[14.01.2020 22:56:23] <47> Info                   [AP] (b53c) output: --asyncNtf:--busy: Target backup repository is overloaded.
[14.01.2020 22:58:27] <65> Info     [AP] Waiting for completion. Id: [0x2c70e5a], Time: 01:00:00.0156287
[14.01.2020 23:16:24] <81> Info                   [AP] (8ff9) output: --asyncNtf:Pipeline timeout: it seems that pipeline hanged.
[14.01.2020 23:16:24] <23> Info                   [AP] (fe9f) output: --asyncNtf:Pipeline timeout: it seems that pipeline hanged.
[14.01.2020 23:16:24] <59> Info                   [AP] (1e3f) output: --asyncNtf:Pipeline timeout: it seems that pipeline hanged.
[14.01.2020 23:16:24] <72> Info                   [AP] (b53c) output: --asyncNtf:Pipeline timeout: it seems that pipeline hanged.
[14.01.2020 23:18:08] <66> Info                   [AP] (8ff9) output: --asyncNtf:Pipeline timeout: it seems that pipeline hanged.
[14.01.2020 23:28:27] <65> Info     [AP] Waiting for completion. Id: [0x2c70e5a], Time: 01:30:00.0312506
[14.01.2020 23:36:23] <56> Info                   [AP] (1e3f) output: --asyncNtf:--busy: Target backup repository is overloaded.
[14.01.2020 23:36:23] <81> Info                   [AP] (fe9f) output: --asyncNtf:--busy: Target backup repository is overloaded.
[14.01.2020 23:36:23] <36> Info                   [AP] (b53c) output: --asyncNtf:--busy: Target backup repository is overloaded.
[14.01.2020 23:46:24] <81> Info                   [AP] (8ff9) output: --asyncNtf:Pipeline timeout: it seems that pipeline hanged.
[14.01.2020 23:46:24] <46> Info                   [AP] (fe9f) output: --asyncNtf:Pipeline timeout: it seems that pipeline hanged.
[14.01.2020 23:46:24] <36> Info                   [AP] (b53c) output: --asyncNtf:Pipeline timeout: it seems that pipeline hanged.
[14.01.2020 23:46:24] <56> Info                   [AP] (1e3f) output: --asyncNtf:Pipeline timeout: it seems that pipeline hanged.
[14.01.2020 23:48:08] <83> Info                   [AP] (8ff9) output: --asyncNtf:Pipeline timeout: it seems that pipeline hanged.
[14.01.2020 23:58:27] <65> Info     [AP] Waiting for completion. Id: [0x2c70e5a], Time: 02:00:00.0468774
[15.01.2020 00:16:23] <21> Info                   [AP] (fe9f) output: --asyncNtf:--busy: Target backup repository is overloaded.
[15.01.2020 00:16:23] <83> Info                   [AP] (1e3f) output: --asyncNtf:--busy: Target backup repository is overloaded.
[15.01.2020 00:16:23] <47> Info                   [AP] (b53c) output: --asyncNtf:--busy: Target backup repository is overloaded.
[15.01.2020 00:16:24] <72> Info                   [AP] (8ff9) output: --asyncNtf:Pipeline timeout: it seems that pipeline hanged.
[15.01.2020 00:16:24] <46> Info                   [AP] (b53c) output: --asyncNtf:Pipeline timeout: it seems that pipeline hanged.
[15.01.2020 00:16:24] <78> Info                   [AP] (fe9f) output: --asyncNtf:Pipeline timeout: it seems that pipeline hanged.
[15.01.2020 00:16:24] <82> Info                   [AP] (1e3f) output: --asyncNtf:Pipeline timeout: it seems that pipeline hanged.
[15.01.2020 00:18:08] <46> Info                   [AP] (8ff9) output: --asyncNtf:Pipeline timeout: it seems that pipeline hanged.
[15.01.2020 00:28:28] <65> Info     [AP] Waiting for completion. Id: [0x2c70e5a], Time: 02:30:00.0625003
[15.01.2020 00:46:24] <52> Info                   [AP] (8ff9) output: --asyncNtf:Pipeline timeout: it seems that pipeline hanged.
[15.01.2020 00:46:24] <23> Info                   [AP] (b53c) output: --asyncNtf:Pipeline timeout: it seems that pipeline hanged.
[15.01.2020 00:46:24] <46> Info                   [AP] (1e3f) output: --asyncNtf:Pipeline timeout: it seems that pipeline hanged.
[15.01.2020 00:46:24] <27> Info                   [AP] (fe9f) output: --asyncNtf:Pipeline timeout: it seems that pipeline hanged.
[15.01.2020 00:48:08] <36> Info                   [AP] (8ff9) output: --asyncNtf:Pipeline timeout: it seems that pipeline hanged.
[15.01.2020 00:56:23] <55> Info                   [AP] (1e3f) output: --asyncNtf:--busy: Target backup repository is overloaded.
[15.01.2020 00:56:23] <56> Info                   [AP] (b53c) output: --asyncNtf:--busy: Target backup repository is overloaded.
[15.01.2020 00:56:23] <46> Info                   [AP] (fe9f) output: --asyncNtf:--busy: Target backup repository is overloaded.
[15.01.2020 00:58:28] <65> Info     [AP] Waiting for completion. Id: [0x2c70e5a], Time: 03:00:00.0781248
[15.01.2020 01:16:24] <46> Info                   [AP] (8ff9) output: --asyncNtf:Pipeline timeout: it seems that pipeline hanged.
[15.01.2020 01:16:24] <47> Info                   [AP] (1e3f) output: --asyncNtf:Pipeline timeout: it seems that pipeline hanged.
[15.01.2020 01:16:24] <17> Info                   [AP] (fe9f) output: --asyncNtf:Pipeline timeout: it seems that pipeline hanged.
[15.01.2020 01:16:24] <46> Info                   [AP] (b53c) output: --asyncNtf:Pipeline timeout: it seems that pipeline hanged.
[15.01.2020 01:18:08] <47> Info                   [AP] (8ff9) output: --asyncNtf:Pipeline timeout: it seems that pipeline hanged.
[15.01.2020 01:28:28] <65> Info     [AP] Waiting for completion. Id: [0x2c70e5a], Time: 03:30:00.0937501
[15.01.2020 01:36:23] <16> Info                   [AP] (1e3f) output: --asyncNtf:--busy: Target backup repository is overloaded.
[15.01.2020 01:36:23] <47> Info                   [AP] (fe9f) output: --asyncNtf:--busy: Target backup repository is overloaded.
[15.01.2020 01:36:23] <66> Info                   [AP] (b53c) output: --asyncNtf:--busy: Target backup repository is overloaded.
[15.01.2020 01:46:24] <83> Info                   [AP] (8ff9) output: --asyncNtf:Pipeline timeout: it seems that pipeline hanged.
[15.01.2020 01:46:24] <16> Info                   [AP] (1e3f) output: --asyncNtf:Pipeline timeout: it seems that pipeline hanged.
[15.01.2020 01:46:24] <55> Info                   [AP] (fe9f) output: --asyncNtf:Pipeline timeout: it seems that pipeline hanged.
[15.01.2020 01:46:24] <46> Info                   [AP] (b53c) output: --asyncNtf:Pipeline timeout: it seems that pipeline hanged.
[15.01.2020 01:48:08] <66> Info                   [AP] (8ff9) output: --asyncNtf:Pipeline timeout: it seems that pipeline hanged.
[15.01.2020 01:58:28] <65> Info     [AP] Waiting for completion. Id: [0x2c70e5a], Time: 04:00:00.0937545
[15.01.2020 02:16:23] <83> Info                   [AP] (fe9f) output: --asyncNtf:--busy: Target backup repository is overloaded.
[15.01.2020 02:16:23] <21> Info                   [AP] (1e3f) output: --asyncNtf:--busy: Target backup repository is overloaded.
[15.01.2020 02:16:23] <66> Info                   [AP] (b53c) output: --asyncNtf:--busy: Target backup repository is overloaded.
[15.01.2020 02:16:24] <59> Info                   [AP] (8ff9) output: --asyncNtf:Pipeline timeout: it seems that pipeline hanged.
[15.01.2020 02:16:24] <66> Info                   [AP] (1e3f) output: --asyncNtf:Pipeline timeout: it seems that pipeline hanged.
[15.01.2020 02:16:24] <56> Info                   [AP] (fe9f) output: --asyncNtf:Pipeline timeout: it seems that pipeline hanged.
[15.01.2020 02:16:24] <21> Info                   [AP] (b53c) output: --asyncNtf:Pipeline timeout: it seems that pipeline hanged.
[15.01.2020 02:28:28] <65> Info     [AP] Waiting for completion. Id: [0x2c70e5a], Time: 04:30:00.1093792
[15.01.2020 02:46:24] <82> Error                  [AP] (8ff9) error: Transmission pipeline hanged, aborting process.
[15.01.2020 02:46:24] <79> Error                  [AP] (b53c) error: Transmission pipeline hanged, aborting process.
[15.01.2020 02:46:24] <21> Error                  [AP] (fe9f) error: Transmission pipeline hanged, aborting process.
[15.01.2020 02:46:24] <71> Error                  [AP] (1e3f) error: Transmission pipeline hanged, aborting process.
[15.01.2020 02:46:24] <65> Error    Transmission pipeline hanged, aborting process. (Veeam.Backup.Common.CCppComponentException)
[15.01.2020 02:46:24] <65> Error       at Veeam.Backup.AgentProvider.CClientAgentProtocol.WaitUntilWorking(Int32 timeout)
[15.01.2020 02:46:24] <65> Error       at Veeam.Backup.AgentProvider.CClientAgentProtocol.DoAgentCommand(String command, String logCommand, Boolean doWait, Nullable`1 outputLogLevel)
[15.01.2020 02:46:24] <65> Error       at Veeam.Backup.AgentProvider.CClientAgentProtocol.Invoke(String command, CVcpCommandArgs inArgs, Boolean noLog, Boolean agentNoLog)
[15.01.2020 02:46:24] <65> Error       at Veeam.Backup.AgentProvider.CBackupClient.Invoke(String command, CVcpCommandArgs inArgs, Boolean noLog)
[15.01.2020 02:46:24] <65> Error    --- End of stack trace from previous location where exception was thrown ---
[15.01.2020 02:46:24] <65> Error       at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
[15.01.2020 02:46:24] <65> Error       at Veeam.Backup.Common.ExceptionFactory.ThrowNecessaryAggregateException(IEnumerable`1 exceptionsCollection)
[15.01.2020 02:46:24] <65> Error       at Veeam.Backup.AgentProvider.CBackupClientImpl.ConstructException(Exception exception, String error)
[15.01.2020 02:46:24] <65> Error       at Veeam.Backup.AgentProvider.CBackupClient.ConstructException(Exception ex, String error, Object[] args)
[15.01.2020 02:46:24] <65> Error       at Veeam.Backup.AgentProvider.CBackupClient.OnInvokeError(Exception e, String command, CVcpCommandArgs inArgs)
[15.01.2020 02:46:24] <65> Error       at Veeam.Backup.AgentProvider.CBackupClient.Invoke(String command, CVcpCommandArgs inArgs, Boolean noLog)
[15.01.2020 02:46:24] <65> Error       at Veeam.Backup.AgentProvider.CBackupClient.DataTransferSyncDisk(ISourceDiskSpec source, ITargetDiskSpec target, Boolean serverHandlesSource, String operationId, Boolean treatProcessedLikeReaded)
[15.01.2020 02:46:24] <65> Error       at Veeam.Backup.Core.CAgentSyncDiskTransport.<>c__DisplayClass2.<SyncDisk>b__0()
[15.01.2020 02:46:24] <65> Error       at Veeam.Backup.Core.CAgentSyncDiskTransport.ProcessDisk(IDiskTaskSource diskTaskSource, IDiskTaskTarget diskTaskTarget, CDiskDigestHook sourceHook)

I am logging to a local NAS, my other seven VMs backup fine.

Any guidance would be great.

Kam

Post by **PetrM** » Jan 15, 2020 6:18 pm this post

Hello!

Generally, this message means that data transmission between Data Mover components is stopped by some reason but connection still exists.

According to log snippet, there is an issue with writing data to a NAS which seems to be overloaded:
- May be you run too many tasks in parallel to the NAS and it would make sense to reschedule backup of 2 problematic VMs
and backup these VMs in a separate job or to decrease concurrent tasks number at the level of repository settings
- There is a hardware issue with NAS itself which affects data write process

Anyway, let's wait for results of troubleshooting from our support team.

Thanks!

R&D Forums

[ID #03954027] Transmission pipeline hanged

Re: [ID #03954027] Transmission pipeline hanged

Who is online