Failed to connect to port (Server:2501)

[SunkistDavid]

We setup a test environment for V6 while maintaining our V5 product in production. Glad we did. We are currently getting error: Error: Client error: Failed to connect to the port (CorpVeeamProxy:2501) when running a replication job.

We have an active case in support: 5162666

At this point we are a level 3 case and have been told may not get a call back today, been working on since last Wednesday, because as we “are probably aware (As you are experiencing the same), many people are under pressure to complete projects before the New Year.” Per tech support.

On Wed we worked with support and they could not get the replication to work. We uploaded technical support logs. I had only two jobs and they were both failing. On Thursday I came into office and one of the jobs showed success and when I checked remote site the replica was in place. I started the “success” job manually and it failed. HUH? I forwarded the success log and the failure log to Veeam.

I created two more jobs one on our .xx network and one on our .yy network, network design discussed below, and they all failed. On Thursday my other tech indicated to me that they had a similar problem due to port usage in the 2500+ range with V5 and the resolution was to set on the Data Transfer tab of the computer under the Server section to be “FORCED AGENTLESS MODE” and this resolved the issue. I asked Tech Support if this was possible as I didn’t see the settings anymore and I was told no since using a proxy. I then set all port settings where applicable to 3000 – 5000 but it seems the agent is still using 2501.

I have attempted the jobs to proxy to the Backup and Replication server and to the separate proxy server I have on the local side. Each method failed. When I use the BR server I get error: Client error: No such host is known. I even tried job to BR server as proxy with 2nd NIC to .YY network disabled so that I wouldn’t possibly get occurrence of port flapping.

On Friday I uploaded wire shark packet capture information run on the BR server during a job run.

Closed on Monday and I come into office on Tuesday and 3 of the 5 jobs show success. The 5 jobs are all scheduled to run Tues and Wed at 4:30pm. I manually rerun 2 of the 3 success jobs and they FAIL with the 2501 error. What the ?.

Here is my network design:

Two sites – Primary and DR

Primary Site
BR Server with Backup and Replication installed. Two Nics (.XX network and .yy network)
Proxy server setup as proxy server via the BR Server. Two Nics (XX network and .YY Network)
ESXi 4.X servers and association SAN

REMOTE SITE
The remote site is a .ZZ network presented through a firewall as a .YY Network.
Proxy server setup via the BR server at primary site. (Single .ZZ network)
ESX Server and SAN are .ZZ network and as indicated presented as .YY network so I can see them from the .XX and .YY network.

OK, as I told support I’m getting pressured to complete this and have fully functioning and working replication by the end-of-the year so I need some additional ideas.

[tsightler]

So to me your errors messages are pretty clear, you are unable to make a connection from the target proxy back to the source proxy. Sounds like DNS and/or network connectivity issues. During replication a connection is made from the target proxy back to the source on port 2501. Since your source proxy is dual-homed, it sounds like at least one of the IP addresses is not reachable from the remote location. That might explain why on one occasion it worked, perhaps when it makes connection from .ZZ -> .XX it works, while from .ZZ -> .YY it does not. Just a thought. Things I would check:

Do you have full connectivity/routing from the .ZZ network to both the .XX and .YY networks?
Are there firewalls between these networks?

The Veeam logs are actually pretty detailed and while quite wordy, it's pretty easy to figure out exactly what IP addresses the proxies are using.

[SunkistDavid]

Yes, this is why I indicated I disabled the second nic to see if have port flapping or similar. So on the local side I disabled the 2nd NIC on the BR server as well as the proxy server and it still gave the same error.

Your reply doesn't help me when I perform the replication job using the same server as BR and Proxy even when I disable the 2nd NIC.

I forwarded my logs to support last Wed and they haven't provided any feedback.

In regards to DNS: I was reading about this in V5 and thought it was resolved in V5.01 so didn't post that I tested this out also. Never-the-less, I unregistered the 2nd NIC in DNS and made sure no remenants in DNS for both the local BR and Proxy server and still had job failures.

[Gostev]

Re-running the job may produce different results due to the fact that backup proxies are assigned dynamically (unless you have forced them to static in the replication job settings). Some of your backup proxies might not have this issue, you can identify from the real-time statistic window's log which proxies source and target proxies were used when the job completed succesfully, and which when the job failed - and go from there.

BTW, not sure what you are saying about v5 and resolved DNS issue. If there is DNS issue, it cannot be resolved in our product. DNS has to work properly if you are using DNS names. May be that DNS name still resolves into the disabled NIC's IP address (the fact that NIC is disabled obviously does not affect existing DNS records).

[Gostev]

Another idea - you may only have port 2500 open, but not 2501 and above. On backup proxy, there is one data mover agent started for each job. The first agent that starts will take up port 2500, the next agent will take 2501, and so on. Which would explain why some job complete normally, and other jobs fail.

[tsightler]

Your reply doesn't help me when I perform the replication job using the same server as BR and Proxy even when I disable the 2nd NIC.

I'm sorry that my reply wasn't helpful but I don't fully understand what "performing the replication job using the same server as BR and Proxy" means. Are you saying that you attempted to use the same proxy for both source and target?

If you're on the forum looking for some "magic bullet" solution, I think it is unlikely. In my experience "unable to connect to port 2501" has always come down to some type of network connectivity, firewall or security software issue. If I were troubleshooting this issue I would look at the logs to determine what IP addresses are being used by the proxies, I would also monitor the proxies while a job was running and see if there is a VeeamAgent process listening on port 2501. Assuming there is (there certainly should be), then it pretty much has to be a network/security software issue. If there's not, then something with Veeam is not working correctly.

[SunkistDavid]

As noted I have on the local side two servers. One is the Backup and Replication server and one is the Proxy server. What I meant is I manually selected the Backup and Replication server to act also as the proxy instead of letting it manually select the regular proxy server.

The jobs show connection to the local proxy and the remote proxy. I'm performing some other tests to try to determine if its a network issue.

Thanks

David

[SunkistDavid]

Regarding the use of port 2500, 2501 is there a way to change this so it starts with port 3000 then 3001?

[tsightler]

As noted I have on the local side two servers. One is the Backup and Replication server and one is the Proxy server. What I meant is I manually selected the Backup and Replication server to act also as the proxy instead of letting it manually select the regular proxy server.

Got it. So in both of these cases there is a target proxy and a source proxy involved. The error is not indicating an issue with the B&R server communicating with the proxy, it is indicating the failure of the target proxy to make it's connection back to the source proxy to retrieve the data so you should focus on the proxy-to-proxy portion of the connection.

[SunkistDavid]

Thanks working on that but still would like to kow if can change the agent port to the 3000 range. I have my reasons for asking.

[SunkistDavid]

Also regarding DNS

My DNS is setup correctly as it only shows the XX adapter registration and the YY adapter registration. I aslo do not have a default gateway on the second NIC card and its not registered in DNS. When I ping VeeamBR and VeeamProxy I get their XX address and when I ping DRVeeaamProxy I get the ZZ address.

So

Local

VeeamBR xx.xx.xx.139
yy.yy.yy.169

VeeamProxy
xx.xx.xx.140
yy.yy.yy.170

DR

DRVeeamProxy
zz.zz.zz.64

[Gostev]

Thanks working on that but still would like to kow if can change the agent port to the 3000 range. I have my reasons for asking.

Sure thing. In the Servers tree, find the Windows server where you have the backup proxy installed, right-click and select Properties, go to the Credentials step of the wizard, and click Ports.

[SunkistDavid]

Interesting as I did this for each of the proxies and set the range from 3000 to 5000 for data transfer and I still get error on port 2501 and it indicates in the logs trying to connect on port 2501. So where is my disconnect?

[Gostev]

Assuming you did everything right - either server and proxy reboot might be required, or there is UI bug causing the port settings to be ignored. If reboot does not help, attach the new logs (with port settings changed) and ask support to check this part specifically. Thanks!

[SunkistDavid]

Yea I rebooted both proxy servers last week so no help. I will send the new logs to support.

[Gostev]

Actually, I was talking about rebooting them after changing the port setting - last week's reboot might just not cut it, if you have changed those just now or was it something you did last week?

[SunkistDavid]

Ok, I built out a new server on the DR side that is located on a different ESX server and SAN. This ESX server is not being monitored by our DR provider so should not have any port access issues. I repointed a job to the 'new' proxy and it still failed with the same error of Error: Client error: Failed to connect to the port [VeeamProxy:2501]. The job went all the way through and showed connections to both proxies and then when processing VM got error. This looks like a local side proxy issue not a remote proxy issue. I have uploaded the subsequent support files via e-mail to the support technician. Note: I again rebooted all V6 machines on both sides of network prior to rerunning job.

[Gostev]

I am really curios as to why the port setting are being ignored. As to the actual issue, I am now about 99% sure this is caused by firewall somewhere.

[SunkistDavid]

Could be but this does not explain why sometimes it works and sometimes it doesn't. Also I have never had a manual run success only a scheduled run success.
A slight update it fails right after the preparing the replica VM completes.

I just disabled the 2nd nic on both servers on the local side so the entire server environment is 1 NIC and still fails.

[Gostev]

David, please no log snippets inline in the forum posts, as requested when you click New Topic. Thank you for understanding.

[SunkistDavid]

an update: I removed the proxy services from both the local and remote proxy and it uninstalled the service. I rebooted each server. I re-added the servers to the server list and the backup proxy list. I made sure to change the port setting to 3000-5000 when setting-up the server in the Server section. Re-ran job - same failure port 2501. HUM

[ssimakov]

David, could you please provide some insight on the connection between main site and DR in regards to network configuration? Like do you control this connection, how many hops are there, is there any kind of tunnelling (VPN for example) between two networks, is there any kind of address translation and so on? Thanks!

[SunkistDavid]

Ok, on the proxy server ran Netstat -an and I don't see port 2500 or 2501 as a listening port. I see TCP 6160/61/62 listening.
I tried to telnet from my workstation to the server port 2500 and 2501 and it failed with could not open connection on port.
Since I didn't post earlier I'm running Windows Server 2008 R2

[Gostev]

250* will only appear when you start the job and the data mover agent is launched on proxy.
While the 616* ports are from other Veeam components (these do listen persistently).

[SunkistDavid]

Regarding the connection between the main site and DR

1) We control connection
2) 6 Hops - Firewall, 3 hop WAN provider, Firewall
3) no tunneling or VPN
4) At the remote site the Y.Y.Y.Y network we manage and its visible through the firewall to the X and Y networks at the local site. As a note we don't re-IP to the Y network as we also have X and Y networks hidden behind the firewall.

Hope that helps.

[SunkistDavid]

As a reminder our V5 product completely works but we did have a similar problem, per my other tech as I have only been with this company a couple months, and it was resolved by selecting FORCED AGENTLESS MODE.

[SunkistDavid]

question: Is the remote proxy trying to talk back to the local proxy or is the Backup Replication Server at the local site trying to talk back to the local proxy in this design?
If the first I will ask my network team to check logs on the Firewalls to see what if any port 2500/01 communications is occurring if its the latter its internetwork communication and something else is going on.

[foggy]

David, in this design the remote proxy talks back directly to the local proxy. Have you considered this topic? Might be the case with your environment.

[SunkistDavid]

Thanks for reply - here is the status update this morning
I have 5 jobs and 3 of them show success for scheduled runs last night with 2 failed

CTX01 X..X.X.X failed 4:30pm
NU01 X.X.X.X Success 4:30pm
RA01 Y.Y.Y.Y Success 10:00pm
PS03 Y.Y.Y.Y Success 4:30pm
SL01 Y.Y.Y.Y Failed 10:00pm

PS03 and SL01 are on the same local ESX server
I have both success and failures independent of them being on the Y network or the X network or what time they run at.

When I manually run any of the scheduled success jobs they fail

How can this be a firewall or network issue Foggy and Gostev if I showing success on some of these jobs.

[SunkistDavid]

The only difference at this time between scheduled and manual jobs is I'm attempting to run manual jobs during the day between 7:30 and 4:30pm. During this time I have the throttling enabled whereas when the after 4:30 scheduled jobs do not have the throttling enabled. Unnfortunately, this does not explain why I have 2 jobs failing.