endpoints (VM and agents) do not fail over to next network if constant peer connection resets

[Mgamerz]

I have an interesting issue I am working with support on.

Original Case 03854935 (now closed)
Additional case 03860162 (thought to be unrelated but is definitely related)

Essentially what is happening is that the people in charge of our router (an external organisation) added some routes when they added some security features to the network that exposed new hosts that were previously unreachable.

I have a network setup such as this. We do not have a dedicated backup network or hardware. Note these are example IPs and not my actual ones:

Backup Server (BU1) with interfaces:

192.168.20.1 <- Backup connection directly to FS1. no switch. Just direct 10gbps wire
10.0.0.200 <- production network on 1Gbps ethernet


File server with interfaces
192.168.20.2 <- Backup connection directly to BU1. no switch. Just direct 10gbps wire
10.0.0.230 <- production network on 1Gbps ethernet (so clients can reach it)

Other systems (including Hyper-V hosts and windows agents)
10.0.0.X <- Production network on 1Gbps ethernet

Our router management group has changed routes and somehow a foreign host with IP 192.168.20.1 is now reachable, where previously all client systems could not reach this IP. It should not be reachable, but for some reason it now is. From talks with them it has to do with some route changes they made.

Preferred network order settings are global, so I must put all networks I prefer:
192.168.20.0/24 (FS1/BU1 10Gbps connection)
10.0.0.0/24 (Production 1Gbps)

In this scenario, my FS1 server will backup over 192.168.20.0 as it sees closest route directly connected. The other systems now see this 192.168.20.1 host that has appeared and is trying to connect to it - however, this system is obviously not backup server. If for some reason this port is listened on (in my case it is, 2501), the connection gets reset.

If this host did not exist:
Veeam backup endpoints (hypervisor/agent) seems to try a few times before giving up and trying next preferred IP. Veeam seems to send client list of interfaces the backup server is listening on and they are tried in-order.

If this host exists:
Veeam backup endpoints (hypervisor/agent) seems to try a few times before giving entirely because of connection resets. Logically, if a connection keeps failing, one would go to the next preferred network. However, it seems Veeam just gives up instead, and as such, backups stop working.


Support has said to use preferred networks feature but this is global only and as such will not work for a network setup like mine, because if I choose the production network as first, my large file server will also backup over the production network (slow). If I choose the private 192.168.20.0 first, my clients are trying to talk to this unknown foreign system and not failing over to the next IP (production network). I get this sounds like a security issue (this is authorized host but I do not know why it is now reachable. It's in a completely different part of my country).

I am trying to get our router management group to remove this route to this host, but I also feel veeam should be able to handle something like this, where if connection fails constantly, it fails over onto the next item in the list.

[HannesK]

Hello,
In general: Veeam is based on Windows. Windows tells the agent which routes to take. We cannot override operating system routing.

Honestly I don't see a reason why agents try to connect to 192.168.20.1. Because you wrote that the "public" backup server IP is 10.0.0.200.

I guess the easiest way to avoid any troubleshooting is changing the IP addresses for the 10GBit backup network. It's just FS1 and the backup server 10GBit interfaces.

Best regards,
Hannes