Failed to connect to port (Server:2501)

[Gostev]

How can this be a firewall or network issue Foggy and Gostev if I showing success on some of these jobs.

Well, frankly the fact that some job work, and other fail with the network connection error, is exactly what points at intermittent network connection issue to me. Especially when you say that even successful job would fail when run manually at a different time.

[SunkistDavid]

I'm sorry Gostev that doesn't make sense to me. I have three jobs at 4:30pm run and two succeed and 1 fails, I have two jobs at 10:00pm and one succeeds and one fails. This has occurred exactly the same two nights in a row. I can't be having intermittent failure at exactly the same time a specific job runs. Any attempt at manual run fails. If I had a network connection issue wouldn't I be having an issue with my V5 product which is connected along the same network path replicating the same servers I'm trying to replicate? OK we ran two more tests and my FW Admin came back with the following:

2011-12-28 09:19:05 Permit x.x.x.x:50685 RemoteProxy:2500 x.x.x.x:50685 RemoteProxy:2500 58 sec 1340 1272 TCP PORT 2500 Close - TCP FIN
2011-12-28 09:36:29 Permit x.x.x.x:50721 RemoteProxy:2500 x.x.x.x:50721 RemoteProxy:2500 59 sec 1340 1272 TCP PORT 2500 Close - TCP FIN

x.x.x.x is the Backup and Replication Server address not the localProxy address. Our logs do not show any communication attempts between local and remote proxy at least on the DR Firewall.

[SunkistDavid]

Good Update: Got it to work but now need to figure out why this is occurring

I noticed when I run the job the local proxy was connecting in NBD mode while the remote proxy was showing as HotAdd;NBD. So on my two proxies I changed their transport method to Network under the Transport Mode setting.

Now my jobs are succeeding.

Another concern I just noted was that the first job that ran ran at 28MB/s but I have a network throttle to from the X-->Z and the Y-->Z network of 10MB/s so interesting that it exceeded the the throttle limitations.

The second job is running at 2 MB/s so that is OK.

Thoughts

[Gostev]

Another concern I just noted was that the first job that ran ran at 28MB/s but I have a network throttle to from the X-->Z and the Y-->Z network of 10MB/s so interesting that it exceeded the the throttle limitations.

How did you determine the speed at which the data was pushed across the wire? There is no such counter in the job statistics...

[SunkistDavid]

The processing rate? Is the processing rate represent both Local/Network/remote network transfer speed? If that is the case whenever I have a throttle in place the bottleneck should always be network especially if I have it set at 10mb/s.

Actually, I'm more concerned with the first issue then I am performance right now so I can understand why when I changed the Transport Mode on the Proxies from Automatic to Network my jobs started working and what is the ramifications of this.

[Gostev]

Sorry, I cannot help you with the first issue via the forum, as this requires closer investigation by our technical specialists (logs, webex etc.) and cannot be done via the forum.

The other question though is exactly where I can assist and explain

- Processing rate is total size of all source VM disks divided by the job run time. Because of that, this will provide nice big numbers during incremental runs, when there is a little change data that need to be processed.

- Also, there is souce data read speed counter next to each processed virtual disk, this is disk read speed. This is generally more useful counter.

There are no counters that show network transfer speed, but you can expect it to be a few times less than disk read speed counter due to source-side dedupe and compression (looks at read vs. transferred values).

[SunkistDavid]

Thanks Gustev for all the replies. Tech support just e-mailed they are reviewing the logs for the hotadd issue. Hopefully we are getting closer to a full system deploy now.

[SunkistDavid]

OK I'm back to square one with the same error. What I did; to resolve issue that I installed to a remote SQL server I uninstalled Veeam and then reinstalled using the local install version of the DB. I recreated all my server connections and proxy servers with the same configuration I had previously including setting the ports to communicate from 3000-5000 and setting transport settings to Network. I recreated my jobs with manual selections for the source and destination proxy servers ran job and they fail with failing to connect to the source proxy:2501 error. UGG - I even tried removing and reinstalling the proxy server transports, trying automatic selection of proxies and so on and NOTHING same error. Why it worked before and not now I can't figure out.

The only difference between this and the previous install is I installed backup_6.0.0.158.hf1_setup so now I'm on version 6.0.0.158

My case number is 5162666 and my understanding its been pushed to 2nd level support and now I'm awaiting reply.

While I'm waiting for tech support any further ideas?

[SunkistDavid]

Update: We got it working - a little confused why worked before we did modification noted below but now replication with NBD and HotAdd are working.

For some reason while the initial install with remote DB worked using NBD transport method the new install did not work. At first thinking was Veeam, then the server, and then the network routes. After a lot of thinking and testing we added a static route on our DR firewall back to the Veeam servers and boom everything started working. Both replication with NBD and HotAdd worked properly. Confusing but now working.

[hansis100]

[merged]

Hello!

I have a problem with Proxy-Servers.

I have 2 Proxy-Servers (My VC-Center-Servers). I have added the 2 Servers as vSphere and as Windows-Server.
One at the source-side (ATVCENTER2) and the other on the target-side (ATVCENTER3).
On the ESX-Hosts I have opened the ports 2500-2510: esxcfg-firewall -openport 2500:2510,tcp,in,VeeamSCP

With the Default-Proxy I don't have any problems.

But when I make a Backup I get the error: Error: Client error: Failed to connect to the port [ATVCENTER2:2501].

What else do I have to do?

thx.
hansi

[hansis100]

Hello again:

I have forgotten to tell you my version:

Veeam: 6
Vmware: 4.1

[snake]

did you ever solve this ?
I am having same issue, cannot perform any backup or replication as cannot connect on port 2501
there is no firewall blocking the port.
if I telnet to the server on 2501 then it cannot connect
If I run netstat on the server, there is nothing listening on that port either.

This is very simple server to server, no proxies etc.

[Vitaliy S.]

Hi Russ,

This is very simple server to server, no proxies etc.

Do you run a just single backup server which has proxy and repository server roles enabled by default? Have you disabled a Windows firewall on this server?

Thanks!

[snake]

I have 3 servers in this network
VEEAM is installed on server1
and I am using it to backup and replicate VM's on server 2 and server 3

The backup repository where I am backing up to is on a totally different network in another DC.
I have successfully done backup and replication from server1 to backup server
I had the 2501 error on server 3 and that someone mysteriously fixed itself and started working. Although I still cannot telnet on 2501, so there is still nothing listening on this port.
Now I am having the same issue with server 2, everything works up until it gets to trying to connect on 2501 and then it fails.

Clealry the host does exist and it does resolve otherwise all the other steps in the backup would fail also, and I would even have been able to add the server to veeam to begin with,
I can successfully browse to all the admin$ shares from each server to the other servers.

There is definitely no firewall causing this problem.

[Vitaliy S.]

Now I am having the same issue with server 2, everything works up until it gets to trying to connect on 2501 and then it fails....
...There is definitely no firewall causing this problem.

Hmm...in this case I guess it's the best time to involve our support engineers into this case. Can you please open a support ticket to verify your setup via remote WebEx session?

[snake]

HERE is the cause in case it helps anyone else.

The VEEAM server and the hypervisor being backed up are on a network with PRIVATE NAT addressing.
The backup/destination server is on a completely different network in a different DC with PUBLIC address only.

During the replication process VEEAM was sending the PRIVATE IP address to the backup server and the backup server was then trying to use that to communicate with the hypervisor server, which of course failed.

The problem is that the logs do not tell you what server is trying to connect to what server only that there was an error, so I naturally assumed the error was VEEAM trying to connect to the hypervisor, which should have worked on the PRIVATE IP.
I did not even realise the backup server tries to connect to the source server.

So a note to VEEAM support, please improve the logging to it shows where the communication breakdown is (which server to which server), had I know this I could have solved the problem weeks ago.

[cheib]

Thanks SunkistDavid this solved my problem ) and as you said, no clear statement in the logfiles !

[J0rn]

HERE is the cause in case it helps anyone else.

The VEEAM server and the hypervisor being backed up are on a network with PRIVATE NAT addressing.
The backup/destination server is on a completely different network in a different DC with PUBLIC address only.

During the replication process VEEAM was sending the PRIVATE IP address to the backup server and the backup server was then trying to use that to communicate with the hypervisor server, which of course failed.

The problem is that the logs do not tell you what server is trying to connect to what server only that there was an error, so I naturally assumed the error was VEEAM trying to connect to the hypervisor, which should have worked on the PRIVATE IP.
I did not even realise the backup server tries to connect to the source server.

So a note to VEEAM support, please improve the logging to it shows where the communication breakdown is (which server to which server), had I know this I could have solved the problem weeks ago.

This forum thread is still the top result for the error, so I will now say that upon reading this I looked at the Proxy settings on VEEAM and found "Run server on this side" - which I would safely assume means the agent does not need to establish a connection to the VEEAM server directly.

This overcomes my NAT issue and everything is now working

[PetrM]

Hello and Welcome to Veeam R&D Forums!

Glad to hear that you've managed to find a workaround! Anyway, I suggest everyone reading this page of our help center to get a better idea of "Run server on this side" option:

In the NAT scenario, the outside client cannot initiate a connection to the server on the NAT network. As a result, services that require initiation of the connection from outside can be disrupted. With this option selected, you will be able to overcome this limitation and initiate a "server-client" connection — that is, a connection in the direction of the Microsoft Windows server.

Thanks!