Backup job fails after relocating the repository and changing the IP

[lavicky]

I prepared new hardened linux backup repository in primary DC and I tested the funkcionality with backup job.
The repository was relocated to secondary DC and rescanned after changing of IP.
The same backup job failes now with this error:

12/17/2021 10:11:12 AM :: Error: Failed to connect to the port [hostnameremoved:2503].
--tr:Failed to connect to target endpoint.
--tr:Client failed to process the command. Command: [connectByIPs].
--tr:event:1:
--tr:event:3:

The ports differ in different job runs.
I can see two-way traffic between VBR server and repository on that ports. I see no dropped packet on firewalls. I can see this ports in listening state on the repository.

Support Case # 05188239

[HannesK]

Hello,
hmm, if the rescan was successful and the DNS entry is still the same... and you only changed the IP... then the default guess about DNS cache is probably wrong. If the firewall does not drop anything, that sounds strange.

I recommend to continue with support as guessing network issues on a forum makes little sense.

Best regards,
Hannes

[lavicky]

Hello,
DNS cache on VBR server was cleared before repository rescan. Both VBR server and repository were rebooted.

I checked VBR server logs and I probably find the problem. The old repository IP 10.99.97.54 is still presented somewhere:

Code: Select all

[16.12.2021 16:09:42] <50> Error        Failed to connect to agent's endpoint '[b]10.99.97.54:2501[/b]'. Host: 'hostnameremoved'.
[16.12.2021 16:09:42] <50> Error        A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond 10.99.97.54:2501 (System.Net.Sockets.SocketException)
[16.12.2021 16:09:42] <50> Error           at System.Net.Sockets.Socket.DoConnect(EndPoint endPointSnapshot, SocketAddress socketAddress)
[16.12.2021 16:09:42] <50> Error           at System.Net.Sockets.Socket.Connect(EndPoint remoteEP)
[16.12.2021 16:09:42] <50> Error           at Veeam.Backup.Common.CNetSocket.Connect(IPEndPoint remoteEp)
[16.12.2021 16:09:42] <50> Error           at Veeam.Backup.AgentProvider.CAgentEndpointConnecter.ConnectToAgentEndpoint(ISocket socket, IAgentEndPoint endPoint)
[16.12.2021 16:09:42] <50> Info             [NetSocket] Connecting to '10.99.197.54:2501'.
[16.12.2021 16:09:42] <50> Info         [ReconnectableSessionStarter] Creating reconnectable socket with options: [enabled: timeout 1800000, interval 15000]. Agent on [[b]10.99.97.54:2501[/b];10.99.197.54:2501]. Connection id: f5b66abc-65f2-4998-9ad1-6d4e3e5306aa.
[16.12.2021 16:09:42] <50> Info             [CryptoSocket] Connecting to '10.99.197.54:2501'.
[16.12.2021 16:09:42] <53> Info                   [AP] (3111) output: >
[16.12.2021 16:09:42] <50> Info                   [AP] (3111) state: ready
[16.12.2021 16:09:42] <50> Info             [ProxyAgent] Starting client agent session, id 'ff4522d7-b80c-4db1-8c2e-716517903529', host 'hostnameremoved', agent id 'a0ff1bb9-4aca-4cdd-87d3-f7f3d28e3e35', IPs '10.99.97.54:2501;10.99.197.54:2501', PID '8556'
[16.12.2021 16:09:42] <50> Info             [SocketAgentService] Connecting to agent '-removed-' ('10.99.97.54:2501;10.99.197.54:2501')
[16.12.2021 16:09:42] <50> Info             [NetSocket] Connecting to '[b]10.99.97.54:2501[/b]'.
[16.12.2021 16:10:03] <50> Error        Failed to connect to agent's endpoint '[b]10.99.97.54:2501[/b]'. Host: 'hostnameremoved'.
[16.12.2021 16:10:03] <50> Error        A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond 10.99.97.54:2501 (System.Net.Sockets.SocketException)
[16.12.2021 16:10:03] <50> Error           at System.Net.Sockets.Socket.DoConnect(EndPoint endPointSnapshot, SocketAddress socketAddress)
[16.12.2021 16:10:03] <50> Error           at System.Net.Sockets.Socket.Connect(EndPoint remoteEP)
[16.12.2021 16:10:03] <50> Error           at Veeam.Backup.Common.CNetSocket.Connect(IPEndPoint remoteEp)
[16.12.2021 16:10:03] <50> Error           at Veeam.Backup.AgentProvider.CAgentEndpointConnecter.ConnectToAgentEndpoint(ISocket socket, IAgentEndPoint endPoint)

There is a only one DNS record -hostnameremoved- IN A 10.99.197.54" known for both VBR server and repository.

I created Support Case # 05188239 before. But no answer yet.

[lavicky]

I created new job. No old IP in VBR server logs but job failed the same way

[Dima P.]

Hello Tomáš,

It looks like some components on the repository host are still have the old IP address cached. Have you tried to flush dns on the host in the question? Additionally you can try to remove and the re-add the repository host to your Veeam B&R server. Thanks!

[lavicky]

The problem solved. VBR server had full access to repository but there were missing routing on the proxies.