connection try has failed

[omfk]

Hello,

since some days I notices that my VAW needs several retries before it finishes its job. The error is always

Code: Select all

15.07.2017 23:48:41 :: Error: Ein Verbindungsversuch ist fehlgeschlagen, da die Gegenstelle nach einer bestimmten Zeitspanne nicht richtig reagiert hat, oder die hergestellte Verbindung war fehlerhaft, da der verbundene Host nicht reagiert hat.

A connection try has failed, as the other side hasn't responded after a certain time span or the established connection was faulty, as the connected host didn't reacted.

A ping from source to Veeam server works and I can also "see" the repository when entering the configuration dialog.

What can I further to check to nail down the specific cause?

The connection runs over a WAN-VPN-connection.

BR
Frank

[Shestakov]

Hello Frank,
it`s hard to say since the error message is in German. Could you provide the support case number?
Thanks!

[Technogod]

Examine log file of specific job that is failing. Logs are located in C:\ProgramData\Veeam\Backup

[omfk]

This error is related to the following case: 02223477
In this specific case the error appeared not in the beginning but in the merging phase.

Some lines around the error from the log file:

Code: Select all

[16.07.2017 23:30:08] <10> Info             [RemoteBackupService] Remote session has been started, id: '75e606a8-14c3-4483-bef6-cdc3aed563f5'
[16.07.2017 23:30:09] <10> Info     Local protocol version: 5, remote protocol version: 5
[16.07.2017 23:30:09] <10> Info     Protocol version check returned - Compatible
[16.07.2017 23:30:09] <10> Info     [RemoteBackupService] Creating LeaseKeeper for session [75e606a8-14c3-4483-bef6-cdc3aed563f5] and lease [79eb9d55-a86f-4303-90c4-485ca11db061]
[16.07.2017 23:30:09] <10> Info     [LeaseKeeper] Created, ttl 1800sec
[16.07.2017 23:30:31] <18> Info     Networking error: [Ein Aufrufziel hat einen Ausnahmefehler verursacht.. Ein Verbindungsversuch ist fehlgeschlagen, da die Gegenstelle nach einer bestimmten Zeitspanne nicht richtig reagiert hat, oder die hergestellte Verbindung war fehlerhaft, da der verbundene Host nicht reagiert hat]
[16.07.2017 23:32:31] <18> Info     [RemoteBackupService] Terminating remote session, id: '5dafe576-35f7-4aa6-a7b3-6db104699bc8'
[16.07.2017 23:32:34] <18> Info     [RemoteBackupService] Remote session was successfully terminated.
[16.07.2017 23:32:34] <18> Error    Failed to notify remote VBR server that endpoint job has been updated
[16.07.2017 23:32:34] <18> Error    Connection problems. (Veeam.Backup.Common.CustomStackException)
[16.07.2017 23:32:34] <18> Error    Veeam.Backup.Common.CMethodRetryException:
[16.07.2017 23:32:34] <18> Error       bei Veeam.Common.Remoting.CRetryableMethod.DoInvoke(Delegate dlg, Object[] args)
[16.07.2017 23:32:34] <18> Error       bei Veeam.Common.Remoting.CRetryableMethod.Invoke[Ret](ActionDelegate`1 dlg)
[16.07.2017 23:32:34] <18> Error       bei Veeam.Backup.Core.CForeignInvokerClient.InvokeInternal(CForeignInvokerParams invokerParams)
[16.07.2017 23:32:34] <18> Error    Ein Aufrufziel hat einen Ausnahmefehler verursacht. (Veeam.Backup.Common.CustomStackException)
[16.07.2017 23:32:34] <18> Error    System.Reflection.TargetInvocationException:
[16.07.2017 23:32:34] <18> Error       bei System.RuntimeMethodHandle.InvokeMethod(Object target, Object[] arguments, Signature sig, Boolean constructor)
[16.07.2017 23:32:34] <18> Error       bei System.Reflection.RuntimeMethodInfo.UnsafeInvokeInternal(Object obj, Object[] parameters, Object[] arguments)
[16.07.2017 23:32:34] <18> Error       bei System.Delegate.DynamicInvokeImpl(Object[] args)
[16.07.2017 23:32:34] <18> Error       bei Veeam.Common.Remoting.CRetryableMethod.DoInvoke(Delegate dlg, Object[] args)
[16.07.2017 23:32:34] <18> Error    Von der Übertragungsverbindung können keine Daten gelesen werden: Ein Verbindungsversuch ist fehlgeschlagen, da die Gegenstelle nach einer bestimmten Zeitspanne nicht richtig reagiert hat, oder die hergestellte Verbindung war fehlerhaft, da der verbundene Host nicht reagiert hat. (Veeam.Backup.Common.CustomStackException)
[16.07.2017 23:32:34] <18> Error    System.IO.IOException:
[16.07.2017 23:32:34] <18> Error       bei Veeam.Backup.Core.CForeignInvokerClient.Wait(WaitHandle waitObj, State state, Int32 timeout, String timeoutError)
[16.07.2017 23:32:34] <18> Error       bei Veeam.Backup.Core.CForeignInvokerClient.<>c__DisplayClass2.<InvokeInternal>b__0()
[16.07.2017 23:32:34] <18> Error    Ein Verbindungsversuch ist fehlgeschlagen, da die Gegenstelle nach einer bestimmten Zeitspanne nicht richtig reagiert hat, oder die hergestellte Verbindung war fehlerhaft, da der verbundene Host nicht reagiert hat (Veeam.Backup.Common.CustomStackException)
[16.07.2017 23:32:34] <18> Error    System.Net.Sockets.SocketException:
[16.07.2017 23:32:34] <18> Error       bei System.Net.Sockets.NetworkStream.Read(Byte[] buffer, Int32 offset, Int32 size)

BR
Frank

[Dima P.]

Hi Frank,

Thanks for sharing case ID. Try to start a file level recovery from any backup file residing Veeam repository and share the result.

[omfk]

I've tried to restore a single file from the full backup and it was successful. Right now the incremental backups are running fine and also the "Full backup file merge completed successfully".

BR
Frank

[Dima P.]

Glad to hear you are up and running. Keep us posted if you face this issue again. Cheers!

[omfk]

Glad to hear you are up and running. Keep us posted if you face this issue again. Cheers!

Well, it's not really over.

Yes, I can restore from the backup in question. The failed connection issue, however, is still there. As long as the failures only occur before making the backup itself, it only postpones the backup.

Current suggestion from support is "interference" with AV activities. So far it is not clear if we are talking about the server to be backed up or the VBR server. If this is really an issue then the question remains, why this hasn't happened earlier.

BR
Frank

[Dima P.]

Frank,

Sorry, I thought it was a temporary glitch. If the issue is sporadic - routing, indeed, sounds like good thing to check. Try to add VAW to your AV exclusions, make sure that you have all ports opened between VBR repository and agent. I’d also check the firewall on router: for testing purposes you can disable it and make a couple of test job runs to check the results.

[omfk]

According to the suggestions and the KB article I excluded certain Veeam directories of the VAW and VBR machine from the AV scanning regime. It worked for some days but just some minutes ago the error happened again:

Code: Select all

26.07.2017 21:48:26 :: Error: Ein Verbindungsversuch ist fehlgeschlagen, da die Gegenstelle nach einer bestimmten Zeitspanne nicht richtig reagiert hat, oder die hergestellte Verbindung war fehlerhaft, da der verbundene Host nicht reagiert hat 

What I really find strange is the fact that this messages comes in German, whereas all the rest of the content inside the information window is in English.

When I look up the error message on google I get mostly hits on database issues. Is there an option to get more debug information from VAW?

BR
Frank

[Dima P.]

Any chance this error could be generated by VBR running in German locale?

[omfk]

When I start the VBR console either on my PC or on the VBR server itself, all content is in English. The language of the OS, however, is German.

BR
Frank

[Dima P.]

Frank,

Understood. Then this error might be generated as a response from Veeam B&R server. Do you still have the original case opened (I think the root cause is somewhere in VAW and VBR debug logs)?

[omfk]

The case is still open. Unfortunately I'm on vacation the next two weeks.

BR
Frank

[Dima P.]

Thanks for the clarification. I believe you can ask support folks to keep the case opened for a while and get back to us later. Have a good vacation!

[max_]

Hello,
Do you have a solution/reason for the Error ?
I get the same Error when i try to backup a lokal Windows 10 PC.

Thanks

Max

[Natalia Lupacheva]

Hi Max,

Cannot find if Frank came back to Support with this issue.
You can check if there are some issues with locale and check if all the required ports are open.

If it does not help, please open a support ticket and share the case id here.

Thanks!

[omfk]

Hi Natalia,

my case ID back then was #02223477.

The full reason for that problem wasn't found. We tried several steps (AV exclusions, rescheduling in time to prevent loads, ...). In the end the errors disappeared.

The latest email from your support (which was very helpful and keen to solve the problem)
"Since there is something prevents our agent from start, then my primary guess is an AV's behavior. I can also tell that very often adding Veeam components to AV exception list is not enough, but disabling or removing an AV helps. I understand security requirements, but if it would be possible to disable it just for the time enough for backup to finish, that would be great.
This is related to a VAW server, because we can see, that source agent did not even start, and the error appears at the moment, we we try to reach target repo:

[26.07.2017 21:45:57] <01> Info Initializing backup target, mode: 'Retry', agent sharing mode: 'AgentPerTask', repository: 'InternalHarddisk2_4TB'.
[26.07.2017 21:45:57] <01> Info [ProxyAgentTracker] Stopping agents on Job: '12c7efe7-7031-4baf-b92f-79d6ddc3f50f'
[26.07.2017 21:46:17] <01> Info Networking error: [Ein Aufrufziel hat einen Ausnahmefehler verursacht.. Ein Verbindungsversuch ist fehlgeschlagen, da die Gegenstelle nach einer bestimmten Zeitspanne nicht richtig reagiert hat, oder die hergestellte Verbindung war fehlerhaft, da der verbundene Host nicht reagiert hat]
[26.07.2017 21:48:26] <01> Error Connection problems. (Veeam.Backup.Common.CustomStackException)

It is also worth to check if ports used by VAW are locked by any other application. Also you can check if there is any network connection is allowed, when the backup tries to start. When backup job starts at 21:30, please try to ping your VBR server to check if it is available from the VAW server."


BR
Frank

[Natalia Lupacheva]

Hi Frank,

Thanks a lot for bringing your results, hope it helps if anyone else faces this issue.
Glad you've solved it!

Thanks!