Veeam Backup Copy Jobs got error in the application---Case #05133397

[apolloxm]

Hello Veeam R&D Team,

---Case #05133397

Recently, Our Veeam Backup Copy Jobs got error said that "error in the application". below just the detail error information in the veeam log.

Code: Select all

Error Message:
[15.11.2021 11:00:29] <29> Info                   [AP] (a662) output: --pex:47;1822096883712;1818359758848;0;1818359758848;984704885987;96;16;30;97;16;51;132814727949050000
[15.11.2021 11:00:29] <29> Info                   [AP] (a662) output: --pex:47;1822676746240;1818939621376;0;1818939621376;985099845192;96;16;30;97;16;51;132814728049060000
[15.11.2021 11:00:29] <29> Info                   [AP] (a662) output: --pex:47;1823171674112;1819434549248;0;1819434549248;985481531447;96;16;30;97;16;51;132814728150150000
[15.11.2021 11:00:29] <29> Info                   [AP] (a662) output: --pex:47;1823644581888;1819907457024;0;1819907457024;985830700362;96;16;30;97;16;51;132814728252180000
[15.11.2021 11:01:18] <37> Info                   [AP] (a662) output: --pex:47;1823854297088;1820117172224;0;1820117172224;985981870051;96;16;30;97;16;51;132814728352180000
[15.11.2021 11:01:18] <51> Error        [ReconnectableSocket][PacketAsyncReceiver] Exception on [<guid>].
[15.11.2021 11:01:18] <51> Error        An existing connection was forcibly closed by the remote host (System.Net.Sockets.SocketException)
[15.11.2021 11:01:18] <51> Error           at Veeam.Backup.Common.Reconnect.CPacketAsyncReceiver.PushData(CSocketAsyncEventArgsEx e)
[15.11.2021 11:01:18] <10> Error                  (4549) error: ChannelError: ConnectionReset

[15.11.2021 11:01:19] <36> Error        Failed to connect to agent's endpoint '<IP>'. Host: '<hostname>'.
[15.11.2021 11:01:19] <36> Error        No connection could be made because the target machine actively refused it <IP>:<port> (System.Net.Sockets.SocketException)
[15.11.2021 11:01:19] <36> Error           at System.Net.Sockets.Socket.DoConnect(EndPoint endPointSnapshot, SocketAddress socketAddress)
[15.11.2021 11:01:19] <36> Error           at System.Net.Sockets.Socket.Connect(EndPoint remoteEP)
[15.11.2021 11:01:19] <36> Error           at Veeam.Backup.Common.CNetSocket.Connect(IPEndPoint remoteEp)
[15.11.2021 11:01:19] <36> Error           at Veeam.Backup.AgentProvider.CAgentEndpointConnecter.ConnectToAgentEndpoint(ISocket socket, IAgentEndPoint endPoint)

[15.11.2021 11:01:50] <28> Info         Collecting storage statistic, file '<filename>.vbk'
[15.11.2021 11:01:50] <28> Info                 [AP] (e3ae) command: '<file>.vbk\n'
[15.11.2021 11:01:50] <28> Error        Failed to send command. (System.Exception)
[15.11.2021 11:01:50] <28> Error           at Veeam.Backup.AgentProvider.CSocketAgentService.Send(String cmd)

[15.11.2021 11:01:50] <28> Error        Socket has been forcibly closed. Failed to recover connection. (Veeam.Backup.Common.CAppException)
[15.11.2021 11:01:50] <28> Error           at Veeam.Backup.Common.Reconnect.CInterlockedReconnectEngine.CheckFailed()
[15.11.2021 11:01:50] <28> Error           at Veeam.Backup.Common.Reconnect.CPacketSender.SendPacketWithReconnect(CPacket packet)
[15.11.2021 11:01:50] <28> Error           at Veeam.Backup.Common.Reconnect.CPacketSender.SendData(Byte[] buffer)
[15.11.2021 11:01:50] <28> Error           at Veeam.Backup.AgentProvider.CSocketAgentService.Send(String cmd)
[15.11.2021 11:01:50] <28> Error        Error in the application. (Veeam.Backup.Common.ConnectionException)


[15.11.2021 11:00:43.844] < 21356> cli      |       - 47%, workload rps: 97/16/51, client: {<guid>}
[15.11.2021 11:00:50.016] < 24032> alg      | ERR |A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond
[15.11.2021 11:00:50.016] < 24032> alg      | >>  |--tr:Cannot read data from the socket. Requested data size: [13].
[15.11.2021 11:00:50.016] < 24032> alg      | >>  |An exception was thrown from thread [24032].

[15.11.2021 11:01:18.282] < 21356> alg      | ERR |Transmission pipe error.
[15.11.2021 11:01:18.282] < 21356> alg      | >>  |Application is shutting down.
[15.11.2021 11:01:18.282] < 21356> alg      | >>  |An exception was thrown from thread [29004].

Your Support Team feeback to us as following, but We don't think so this is root cause. Is there anyone from R&D Team can help with this case?Thanks!

Transmission pipe error is likely a networking issue. This means that our connection was simply stopped or blocked. In your case, it could be possible that your target repository is getting overloaded, thus causing connection issues. It seems too many tasks are saturating (Check concurrent task limit set on the repository - Try reducing this) and causing the job to hang/fail.

I would appreciate if you could perform the below troubleshooting:

-If there is any, perform AV exclusion as per https://www.veeam.com/kb1999

We don't install the AV in our server

-Increase the timeout value for Veeam Transport Agent to start and few other repository connection timeouts. Please add the below registry keys on your VBR server.

We don't think so icrease the timeout value for the Veeam Transport Agent and few other repository connection timeouts that would be helpful

Code: Select all

Path: HKEY_LOCAL_MACHINE\SOFTWARE\Veeam\Veeam Backup and Replication\
 
MaxRepositoryConnectionsRetryTimeoutSec
Type: REG_DWORD
Value: 600 (seconds)
 
PipelineTimeoutSecs
Type: REG_DWORD
Value: 3600 (seconds)
 
RepoBusyTimeoutSec
Type: REG_DWORD
Value: 600 (seconds)
 
AgentStartTimeoutSec
Type: REG_DWORD
Value: 600
 

Once the registry keys are applied, please reboot the Veeam Server or wait for 15-20 minutes before running the job.

[Natalia Lupacheva]

Hi Apollo,

Please remove all sensitive information from the logs you post on the public forums. Also, I've formatted your message a little bit.

I see you have an active discussion with Support and already checked the logs with the extended log level. I would suggest to keep working with Support engineer, because it seems the logs analysis is required in your case. If you disagree with your Support engineer or don't want to add the registry key for some reasons, kindly inform Support about that. If you keep Support 100% informed of your case, it would simplify the process.

Thanks!