Stuck IP Address / Can't Backup Endpoints (Agents)

[packetdog]

Support Case #03746099

Good day all, I have a VMware vSphere ESXi server that is running Windows Server Essentials 2019 (192.168.40.40) and a backup repository (Linux based at 192.168.40.10.) Previously, we had this set up on a different subnet. The Veeam Server (WSE2019) was 192.168.1.241. The linux (backup target) box used to be called 192.168.1.242. Ready to get complicated? OK. So the ESXi server let's you do a virtual switch, so we set up the same backup target (Linux box) server as 172.16.0.10. This was set up for the WSE2019 server to backup (since it's running on the same hypervisor) so basically the server was 172.16.0.2. So we added a backup repository as 172.16.0.10 so it would connect via the vSwitch and 'backup locally'. So this was working 100% with the WSE2019 box backing up via this 'second' backup repository @ 172.16.0.10.

The agents were using the backup repository that was addressed via the 192.168.1.242 subnet/localnet. Then we implemented a new network for the 'localnet', so 192.168.1.x became 192.168.40.x. That's great, but all the endpoints stopped working. So initially I thought editing the agent setups would solve it, but that wasn't the case. I was able to get the server (WS2019) working by redoing the backup repositories, and that's good to go. However, I keep getting this error when I try to back up the endpoints: 9/2/2019 5:38:53 PM :: Error: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond 172.16.0.10:2500 So essentially the only WSE2019 ESXi VM used to use this backup repository, and that was fine. The endpoints never used this, but instead used the localnet w/o issue. Since the network numbering change I've been receiving this error. I went as far as to uninstall the endpoint agents, and I even uninstalled Veeam B&R (and deleted the SQL db), and it still persists. I deleted everything and started over. (Clearly not everything, because this address is somewhere.) As of now, there's no mention of 172.16.x.x anywhere in the VM B&R console, but this issue continues.

As I said at the top, I have an open support case, but I thought someone else here might be able to guide me in purging out this incorrect backup repository that is lingering, and apparently only on the Endpoint Backup Agents. To be clear, this duplicate 172.16.x.x backup repository was never used by the endpoints, it was only for the WSE2019 VM to be backed up via the vSwitch 'locally.'

Thanks,
-pd

[Dima P.]

Hello John,

Can you please clarify how the linux repository was added to the Veeam B&R console (linux host or maybe you mounted a folder via CIFS)? You are using Veeam repository as a target for your agents (not CIFS), right? Thanks!

[packetdog]

Hi Dima,

The repository is added as a Linux host (It's also a Linux VM that we're not backing up), connected via SSH. The Linux user has write access to the partition where the backups are stored. I'm not using CIFS as far as I know. This host is the only repository and has an address of 192.168.40.11. The 172.16.x.x subnet is no longer in use. Does that help to clarify?

[Dima P.]

Yes, thank you for the update. Veeam Agent for Windows first connects to Veeam B&R server to get access to the repository and only then starts to transfer the data directly to the repository server. Open Veeam Agent for Windows backup job properties and on the backup server step of the wizard type in the host name of your Veeam B&R server, then click next to check if it resolves correctly.

[packetdog]

Hi Dima, thanks for that. Can you tell me if I understand this correctly:

- Open the Endpoint Backup Agent, edit the job, and for the Veeam B&R Server, use the hostname instead of the IP address, is that correct?

I've tried it both ways, but currently it is set to the hostname of the WSE2019 VM, which is the Veeam B&R server as well. What's next?

Thanks,
-pd

[Dima P.]

Yes, thank you. I believe the issue is here: agent connects to Veeam B&R and gets the information about repositories; VBR resolves your linux box to 172.16.0.10 and passes this address to the agent for direct connection; agent cannot connect to 172.16.0.10 directly thus there is no routing configured. Makes sense?

[packetdog]

It's possible. That sounds like it could be a possible outcome. I will check the DNS tables and flush them. I will get back to you once I have tested this.

Thanks,
-pd

[packetdog]

Hi Dima,

This appears to not be a DNS issue. I've confirmed that the DNS server (the same WSE2019 server) does not have a DNS entry for the BACKUPTARGET machine. In B&R I have this repository set up using the new IP address 192.168.40.11, yet it still is attempting to connect to this old address of 172.16.0.10. What are the next steps to purge this old address out? Not sure how to correct this.

Just a reminder, the Agents /never/ backed up using this address, they had to use the other clone of the backup repository because they couldn't connect to that subnet. So, somehow these agents have picked up this 172.16.0.10 address that only the server was using, and I can't purge it. Please let me know what you think the next steps are.

Thank you,
-pd

[Dima P.]

John,

Can you ping linux repository from endpoint machine by it's host name to check how host name is resolved?

[packetdog]

Nope, the hostname is not registered in DNS. It's a linux machine on the network, and it's statically assigned. Since we're referencing it via IP address in B&R, we did not give it a name in DNS. We've never needed DNS to make this work before. When you attempt a DNS lookup on the endpoint it does not resolve with backuptarget, or with backuptarget.domain.local.

Dima, is there a way to purchase Basic support for this incident so I can engage more hands on with someone from the support team? This has been an issue for over a week and we need to explore options to escalate this.

[Dima P.]

Hello John,

I've checked your case details and the case is now closed. Can you please confirm that the issue is resolved? Thank you!