Offsite backups-port 2501

[jakub.krocek]

Hello,
Case number #05739588
I have four sites, lets name them Z[1.x], K[30.x], M[2.x],D[192.x]
Each of those sites have Veeam R&B server as 105 (Z has server on address 1.105, and so on)
There are NASes as repositories, which are 1.200, 30.49, 2.6 and 192.250
Onsite backups are working fine, no problem there.
Offsite backups are different story, so bear with me.
I'd like to have backups af follows:
D->K, Z->K, M->K and K->M
It's not bulletproff, but thats not the point.
I've set up backup copy jobs that utilize remote Gateway servers for better stability.
One example:
On server D[192.105] backup copy job utilizes SMB repository K[30.49] with Gateway server K[30.105] and everything works fine
And it works for every case, even every permutation, but K->M.
In K->M case job fails with

Code: Select all

24.11.2022 16:02:43 :: Processing XX.XX.XX.XX Error: Failed to connect to the port [M[2.105]:2501].
--tr:Failed to connect to target endpoint.
--tr:Client failed to process the command. Command: [connectByIPs].
--tr:event:1:
--tr:event:3:
  

Wheter Gateway server on repository is FQDN on IP adress is irrelevant, outcome is always same.
D[192.105],K[30.105],M[2.105],Z[1.105] are all fresh Windows 10 installs with firewall turned off.
All locations ale availible to each other.
I can even telnet on port 2501 from K[30.105] to M[2.105]
On M[2.105]

Code: Select all

Get-NetTCPConnection -LocalPort 2501

Even shows established connections from K[30.105] and open listening on 2501

It's worth mentioning, that
M->K, M->Z, M->D,
Z->K, Z->M, Z->D
D->Z, D->K, D->M
K->Z and K->D works

I have no idea how is that possible or what to do to make it working.

[Andreas Neufert]

I would try the following.
Shutdown all Veeam services on the 2 endpoints. Then check with
netstat -an if the port is listening (maybe used by another service) on both sides.

Check for Firewall settings again. Maybe on the router switches ? Maybe there is a packed filter inspection in place that block specific things?
Maybe shutdown all Veeam services on both sides and use a tool that can communicate over a selected port. Like this:
https://stackoverflow.com/questions/100 ... lient-port
So that you can test the connection.

Make sure that the antivirus is configured according to Veeam standards: https://www.veeam.com/kb1999

Make sure that all services are running again.

Overall TCP2500 and higher are only used on demand, so by default it should not be a listening port. It only listens when transport agent is started (Veeam.Agent name in the process view) and then first connection use TCP2500 and next one 2501, next one 2502 and so on.

[jakub.krocek]

Hello Andreas,
First off: Antivirus is correctly set up Bitdefender - same config on all four B&R servers.
I've checked connectivity on port 2500 with services stopped:

Code: Select all

C:\iperf-3.1.3-win64>iperf3.exe -p 2500 -s
-----------------------------------------------------------
Server listening on 2500
-----------------------------------------------------------
Accepted connection from XXX.XXX.30.105, port 60173
[  5] local XXX.XXX.2.105 port 2500 connected to XXX.XXX.30.105 port 60174
[ ID] Interval           Transfer     Bandwidth
[  5]   0.00-1.00   sec  14.7 MBytes   123 Mbits/sec
[  5]   1.00-2.00   sec  13.8 MBytes   116 Mbits/sec
[  5]   2.00-3.00   sec  15.5 MBytes   130 Mbits/sec
[  5]   3.00-4.00   sec  15.6 MBytes   131 Mbits/sec
[  5]   4.00-5.00   sec  14.7 MBytes   123 Mbits/sec
[  5]   5.00-6.00   sec  15.2 MBytes   127 Mbits/sec
[  5]   6.00-7.00   sec  15.4 MBytes   129 Mbits/sec
[  5]   7.00-8.00   sec  15.8 MBytes   133 Mbits/sec
[  5]   8.00-9.00   sec  14.8 MBytes   124 Mbits/sec
[  5]   9.00-10.00  sec  14.9 MBytes   125 Mbits/sec
[  5]  10.00-10.03  sec   554 KBytes   133 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bandwidth
[  5]   0.00-10.03  sec  0.00 Bytes  0.00 bits/sec                  sender
[  5]   0.00-10.03  sec   151 MBytes   126 Mbits/sec                  receiver
-----------------------------------------------------------

I re-started the services and saw on K[30.105] that repository M[2.6] via FQDN of M[2.105] was "Unavailable", which is weird, because adress translation worked (i could ping the FQDN and it returned correct IP). So I changed gateway server to IP address, repository was suddenly available, so I've tried backup copy job, which failed as before.

Port statuses of M[2.105] after failed copy backup:

Code: Select all

PS C:\WINDOWS\system32>  Get-NetTCPConnection -LocalPort 2500

LocalAddress                        LocalPort RemoteAddress                       RemotePort State       AppliedSetting
------------                        --------- -------------                       ---------- -----       --------------
XXX.XXX.2.105                       2500      XXX.XXX.30.105                        49592      Established Internet
XXX.XXX.2.105                       2500      XXX.XXX.30.105                        49591      Established Internet
XXX.XXX.2.105                       2500      XXX.XXX.30.105                        49590      Established Internet
127.0.0.1                           2500      127.0.0.1                           49254      Established Internet
0.0.0.0                             2500      0.0.0.0                             0          Listen


PS C:\WINDOWS\system32>  Get-NetTCPConnection -LocalPort 2501

LocalAddress                        LocalPort RemoteAddress                       RemotePort State       AppliedSetting
------------                        --------- -------------                       ---------- -----       --------------
XXX.XXX.2.105                       2501      XXX.XXX.30.105                        50404      Established Internet
XXX.XXX.2.105                       2501      XXX.XXX.30.105                        50403      Established Internet
XXX.XXX.2.105                       2501      XXX.XXX.30.105                        50402      Established Internet
127.0.0.1                           2501      127.0.0.1                           50220      Established Internet
0.0.0.0                             2501      0.0.0.0                             0          Listen

[Andreas Neufert]

Do I understand it correctly that you have NAT in between the 2 environments?

NAT is not supported for the traffic between Veeam components with only a few exceptions around Veeam Cloud Connect.

[jakub.krocek]

Sorry for the delay,
Yes, there is NAT on the edge routers. Those are separate site with different subnets so there is no other way.
>NAT is not supported for the traffic between Veeam components
If not, then what si the best practice for offsite backups? You got me realy confused.

[Andreas Neufert]

Between the Veeam Servers there can not be any NAT (we connect from both sides and so we can not connect in one direction).
You need to establish somehow a direct connection of VPN.
Even a VPN on the Veeam Repo server would likely do it so that it gets an IP address that is directly reachable from the Veeam environment on the other site.

Other option would be to operate an object storage behind NAT and use this storage (if the object storage support this but usually I think this is the case).

Another option is to create a Cloud Connect (Enterprise)-Backup service. This is basically a service that service provider can implement (behind NAT possible) to offer Backup as a Service. I do not know if you eligible to get licenses and installer as it is usually a cloud partner offering.

I think looking at the VPN options of your firewalls or specific software, is the best option.
Potentially even Veeam PN (simple VPN tool) can help you with this. I would ask my service partner for this if you do not have experience with general VPN and networking.

[jakub.krocek]

Hello,
I am fresh from sick leave and my colleague assured me, that there is no NAT/Masquerade between the sites - my bad.
Even if there was it doesn't explain why 3/4 of solution works when all servers and sites are equally set up.
So far I'll stick to remote repository without proxy and monitor it closely.
Maybe in the next version this will be magically fixed, or I'll have more luck in finding the error in the future.

Thank you for your time Andreas

[Andreas Neufert]

Reach out to your Veeam sales representative and ask if a Solutions Architect could check this in a remote session. Or go to one of our service partner (they know not only Veeam but networks and other things) and ask them to help.