Failed to connect to port (Server:2501)

#1 VM Backup : Modern Data Protection for VMware vSphere and Microsoft Hyper-V

Failed to connect to port (Server:2501)

Veeam Logoby SunkistDavid » Tue Dec 27, 2011 5:21 pm

We setup a test environment for V6 while maintaining our V5 product in production. Glad we did. We are currently getting error: Error: Client error: Failed to connect to the port (CorpVeeamProxy:2501) when running a replication job.

We have an active case in support: 5162666

At this point we are a level 3 case and have been told may not get a call back today, been working on since last Wednesday, because as we “are probably aware (As you are experiencing the same), many people are under pressure to complete projects before the New Year.” Per tech support.

On Wed we worked with support and they could not get the replication to work. We uploaded technical support logs. I had only two jobs and they were both failing. On Thursday I came into office and one of the jobs showed success and when I checked remote site the replica was in place. I started the “success” job manually and it failed. HUH? I forwarded the success log and the failure log to Veeam.

I created two more jobs one on our .xx network and one on our .yy network, network design discussed below, and they all failed. On Thursday my other tech indicated to me that they had a similar problem due to port usage in the 2500+ range with V5 and the resolution was to set on the Data Transfer tab of the computer under the Server section to be “FORCED AGENTLESS MODE” and this resolved the issue. I asked Tech Support if this was possible as I didn’t see the settings anymore and I was told no since using a proxy. I then set all port settings where applicable to 3000 – 5000 but it seems the agent is still using 2501.

I have attempted the jobs to proxy to the Backup and Replication server and to the separate proxy server I have on the local side. Each method failed. When I use the BR server I get error: Client error: No such host is known. I even tried job to BR server as proxy with 2nd NIC to .YY network disabled so that I wouldn’t possibly get occurrence of port flapping.

On Friday I uploaded wire shark packet capture information run on the BR server during a job run.

Closed on Monday and I come into office on Tuesday and 3 of the 5 jobs show success. The 5 jobs are all scheduled to run Tues and Wed at 4:30pm. I manually rerun 2 of the 3 success jobs and they FAIL with the 2501 error. What the ?.

Here is my network design:

Two sites – Primary and DR

Primary Site
BR Server with Backup and Replication installed. Two Nics (.XX network and .yy network)
Proxy server setup as proxy server via the BR Server. Two Nics (XX network and .YY Network)
ESXi 4.X servers and association SAN

REMOTE SITE
The remote site is a .ZZ network presented through a firewall as a .YY Network.
Proxy server setup via the BR server at primary site. (Single .ZZ network)
ESX Server and SAN are .ZZ network and as indicated presented as .YY network so I can see them from the .XX and .YY network.

OK, as I told support I’m getting pressured to complete this and have fully functioning and working replication by the end-of-the year so I need some additional ideas.
SunkistDavid
Enthusiast
 
Posts: 66
Liked: never
Joined: Tue Nov 29, 2011 10:32 pm
Full Name: David

Re: Failed to connect to port (Server:2501)

Veeam Logoby tsightler » Tue Dec 27, 2011 5:46 pm 1 person likes this post

So to me your errors messages are pretty clear, you are unable to make a connection from the target proxy back to the source proxy. Sounds like DNS and/or network connectivity issues. During replication a connection is made from the target proxy back to the source on port 2501. Since your source proxy is dual-homed, it sounds like at least one of the IP addresses is not reachable from the remote location. That might explain why on one occasion it worked, perhaps when it makes connection from .ZZ -> .XX it works, while from .ZZ -> .YY it does not. Just a thought. Things I would check:

Do you have full connectivity/routing from the .ZZ network to both the .XX and .YY networks?
Are there firewalls between these networks?

The Veeam logs are actually pretty detailed and while quite wordy, it's pretty easy to figure out exactly what IP addresses the proxies are using.
tsightler
Veeam MVP
 
Posts: 3385
Liked: 765 times
Joined: Fri Jun 05, 2009 12:57 pm
Full Name: Tom Sightler

Re: Failed to connect to port (Server:2501)

Veeam Logoby SunkistDavid » Tue Dec 27, 2011 6:04 pm

Yes, this is why I indicated I disabled the second nic to see if have port flapping or similar. So on the local side I disabled the 2nd NIC on the BR server as well as the proxy server and it still gave the same error.

Your reply doesn't help me when I perform the replication job using the same server as BR and Proxy even when I disable the 2nd NIC.

I forwarded my logs to support last Wed and they haven't provided any feedback.

In regards to DNS: I was reading about this in V5 and thought it was resolved in V5.01 so didn't post that I tested this out also. Never-the-less, I unregistered the 2nd NIC in DNS and made sure no remenants in DNS for both the local BR and Proxy server and still had job failures.
SunkistDavid
Enthusiast
 
Posts: 66
Liked: never
Joined: Tue Nov 29, 2011 10:32 pm
Full Name: David

Re: Failed to connect to port (Server:2501)

Veeam Logoby Gostev » Tue Dec 27, 2011 6:10 pm

Re-running the job may produce different results due to the fact that backup proxies are assigned dynamically (unless you have forced them to static in the replication job settings). Some of your backup proxies might not have this issue, you can identify from the real-time statistic window's log which proxies source and target proxies were used when the job completed succesfully, and which when the job failed - and go from there.

BTW, not sure what you are saying about v5 and resolved DNS issue. If there is DNS issue, it cannot be resolved in our product. DNS has to work properly if you are using DNS names. May be that DNS name still resolves into the disabled NIC's IP address (the fact that NIC is disabled obviously does not affect existing DNS records).
Gostev
Veeam Software
 
Posts: 15606
Liked: 779 times
Joined: Sun Jan 01, 2006 1:01 am
Full Name: Anton Gostev

Re: Failed to connect to port (Server:2501)

Veeam Logoby Gostev » Tue Dec 27, 2011 6:16 pm

Another idea - you may only have port 2500 open, but not 2501 and above. On backup proxy, there is one data mover agent started for each job. The first agent that starts will take up port 2500, the next agent will take 2501, and so on. Which would explain why some job complete normally, and other jobs fail.
Gostev
Veeam Software
 
Posts: 15606
Liked: 779 times
Joined: Sun Jan 01, 2006 1:01 am
Full Name: Anton Gostev

Re: Failed to connect to port (Server:2501)

Veeam Logoby tsightler » Tue Dec 27, 2011 6:27 pm

SunkistDavid wrote:Your reply doesn't help me when I perform the replication job using the same server as BR and Proxy even when I disable the 2nd NIC.


I'm sorry that my reply wasn't helpful but I don't fully understand what "performing the replication job using the same server as BR and Proxy" means. Are you saying that you attempted to use the same proxy for both source and target?

If you're on the forum looking for some "magic bullet" solution, I think it is unlikely. In my experience "unable to connect to port 2501" has always come down to some type of network connectivity, firewall or security software issue. If I were troubleshooting this issue I would look at the logs to determine what IP addresses are being used by the proxies, I would also monitor the proxies while a job was running and see if there is a VeeamAgent process listening on port 2501. Assuming there is (there certainly should be), then it pretty much has to be a network/security software issue. If there's not, then something with Veeam is not working correctly.
tsightler
Veeam MVP
 
Posts: 3385
Liked: 765 times
Joined: Fri Jun 05, 2009 12:57 pm
Full Name: Tom Sightler

Re: Failed to connect to port (Server:2501)

Veeam Logoby SunkistDavid » Tue Dec 27, 2011 7:05 pm

As noted I have on the local side two servers. One is the Backup and Replication server and one is the Proxy server. What I meant is I manually selected the Backup and Replication server to act also as the proxy instead of letting it manually select the regular proxy server.

The jobs show connection to the local proxy and the remote proxy. I'm performing some other tests to try to determine if its a network issue.

Thanks

David
SunkistDavid
Enthusiast
 
Posts: 66
Liked: never
Joined: Tue Nov 29, 2011 10:32 pm
Full Name: David

Re: Failed to connect to port (Server:2501)

Veeam Logoby SunkistDavid » Tue Dec 27, 2011 7:09 pm

Regarding the use of port 2500, 2501 is there a way to change this so it starts with port 3000 then 3001?
SunkistDavid
Enthusiast
 
Posts: 66
Liked: never
Joined: Tue Nov 29, 2011 10:32 pm
Full Name: David

Re: Failed to connect to port (Server:2501)

Veeam Logoby tsightler » Tue Dec 27, 2011 7:27 pm

SunkistDavid wrote:As noted I have on the local side two servers. One is the Backup and Replication server and one is the Proxy server. What I meant is I manually selected the Backup and Replication server to act also as the proxy instead of letting it manually select the regular proxy server.


Got it. So in both of these cases there is a target proxy and a source proxy involved. The error is not indicating an issue with the B&R server communicating with the proxy, it is indicating the failure of the target proxy to make it's connection back to the source proxy to retrieve the data so you should focus on the proxy-to-proxy portion of the connection.
tsightler
Veeam MVP
 
Posts: 3385
Liked: 765 times
Joined: Fri Jun 05, 2009 12:57 pm
Full Name: Tom Sightler

Re: Failed to connect to port (Server:2501)

Veeam Logoby SunkistDavid » Tue Dec 27, 2011 7:35 pm

Thanks working on that but still would like to kow if can change the agent port to the 3000 range. I have my reasons for asking.
SunkistDavid
Enthusiast
 
Posts: 66
Liked: never
Joined: Tue Nov 29, 2011 10:32 pm
Full Name: David

Re: Failed to connect to port (Server:2501)

Veeam Logoby SunkistDavid » Tue Dec 27, 2011 7:51 pm

Also regarding DNS

My DNS is setup correctly as it only shows the XX adapter registration and the YY adapter registration. I aslo do not have a default gateway on the second NIC card and its not registered in DNS. When I ping VeeamBR and VeeamProxy I get their XX address and when I ping DRVeeaamProxy I get the ZZ address.

So

Local

VeeamBR xx.xx.xx.139
yy.yy.yy.169

VeeamProxy
xx.xx.xx.140
yy.yy.yy.170

DR

DRVeeamProxy
zz.zz.zz.64
SunkistDavid
Enthusiast
 
Posts: 66
Liked: never
Joined: Tue Nov 29, 2011 10:32 pm
Full Name: David

Re: Failed to connect to port (Server:2501)

Veeam Logoby Gostev » Tue Dec 27, 2011 8:17 pm

SunkistDavid wrote:Thanks working on that but still would like to kow if can change the agent port to the 3000 range. I have my reasons for asking.

Sure thing. In the Servers tree, find the Windows server where you have the backup proxy installed, right-click and select Properties, go to the Credentials step of the wizard, and click Ports.
Gostev
Veeam Software
 
Posts: 15606
Liked: 779 times
Joined: Sun Jan 01, 2006 1:01 am
Full Name: Anton Gostev

Re: Failed to connect to port (Server:2501)

Veeam Logoby SunkistDavid » Tue Dec 27, 2011 8:22 pm

Interesting as I did this for each of the proxies and set the range from 3000 to 5000 for data transfer and I still get error on port 2501 and it indicates in the logs trying to connect on port 2501. So where is my disconnect?
SunkistDavid
Enthusiast
 
Posts: 66
Liked: never
Joined: Tue Nov 29, 2011 10:32 pm
Full Name: David

Re: Failed to connect to port (Server:2501)

Veeam Logoby Gostev » Tue Dec 27, 2011 8:51 pm

Assuming you did everything right - either server and proxy reboot might be required, or there is UI bug causing the port settings to be ignored. If reboot does not help, attach the new logs (with port settings changed) and ask support to check this part specifically. Thanks!
Gostev
Veeam Software
 
Posts: 15606
Liked: 779 times
Joined: Sun Jan 01, 2006 1:01 am
Full Name: Anton Gostev

Re: Failed to connect to port (Server:2501)

Veeam Logoby SunkistDavid » Tue Dec 27, 2011 9:24 pm

Yea I rebooted both proxy servers last week so no help. I will send the new logs to support.
SunkistDavid
Enthusiast
 
Posts: 66
Liked: never
Joined: Tue Nov 29, 2011 10:32 pm
Full Name: David

Next

Return to Veeam Backup & Replication



Who is online

Users browsing this forum: AdrianHinton, Bing [Bot], DaFresh, Google [Bot] and 39 guests