after 7.0 upgrade replication fails due to firewall rules

[bkc]

see case 00492657 and case 00492670

case 00492670 Summary:

have two veeam servers, in US and CA, both 64-bit Windows 7 machines. Each is a proxy for the other and we replicate between the two locations

replication jobs from CA to US fail because connection to veeamagent.exe fails.. e.g. veeam (CA) tries to connect to US server on port 2502, times out.

Turning off windows firewall on US veeam server fixes the problem.

Veeam Backup Proxy Service on the US server launches C:\Program Files (x86)\Veeam\Backup Transport\x86\VeeamAgent.exe as the agent

However windows firewall rules list only these VeeamAgents in the Veeam Networking Group

C:\Program Files (x86)\Veeam\Backup Transport\VeeamAgent.exe

C:\Program Files (x86)\Veeam\Backup Transport\x64\VeeamAgent.exe


Two possiblities:

1. Firewall rules were not correctly updated when upgrading from 6.5 to 7.0 (then we added R2 Patch)

2. Proxy service is launching the wrong VeeamAgent

On US server, all 4 of these VeeamAgent.exe files exist:

* C:\Program Files (x86)\Veeam\Backup Transport\VeeamAgent.exe
C:\Program Files (x86)\Veeam\Backup Transport\VeeamAgent64.exe
C:\Program Files (x86)\Veeam\Backup Transport\x86\VeeamAgent.exe
* C:\Program Files (x86)\Veeam\Backup Transport\x64\VeeamAgent.exe

They are all different sizes, different files. Which is the correct VeeamAgent to be run on this Windows 7 64 bit host?

* indicates executable listed in firewall rules

When upgrading Veeam from 6.5 to 7 in the CA side (I think US side still had 6.5), we may have allowed it to update all remote proxy machines, including the US server. Could this have led to the corruption?

---

case 00492657 Summary

Replication job from US to CA fails with 12/19/2013 12:56:12 AM :: Processing 'Bender - Windows 2k' Error: Client error: No such host is known

Log file shows

9.12.2013 13:32:46] <11> Error Failed to connect to server 'VMTEAMER', IPs '192.168.1.23;VMTEAMER', port '2504' at Veeam.Backup.AgentProvider.CBackupClient.Connect(CServerConnParams prms)

I suspect this is the same issue, that the wrong Veeam Agent is being run.. one that is not in the firewall rules.

Will know tonight if US->CA replication job works w/ firewall off (already have confirmed that CA->US works with firewall off)

[bkc]

more info on case 00492657

With windows firewall disabled, replication works.

On the US veeam server, two different copies of VeeamAgent.exe are running

One of them is 32 bit, the other is 64 bit.

wait.. now there's (2) 32 bit agents and one 64 bit agent.

The 32 bit agent is in C:\Program Files (x86)\Veeam\Backup Transport\x86 and this is *NOT* listed in the firewall rules.

Why is Proxy launching this agent?

[Vitaliy S.]

Hi Brad,

They are all different sizes, different files. Which is the correct VeeamAgent to be run on this Windows 7 64 bit host?

Veeam backup manager launches up to 3-4 VeeamAgents per each job, so I believe all of them might be required.

When upgrading Veeam from 6.5 to 7 in the CA side (I think US side still had 6.5), we may have allowed it to update all remote proxy machines, including the US server. Could this have led to the corruption?

Don't think so.

Why is Proxy launching this agent?

Proxy as well as repository is using these data movers (VeeamAgents) to compress, undcompress, dedupe etc. data.

The 32 bit agent is in C:\Program Files (x86)\Veeam\Backup Transport\x86 and this is *NOT* listed in the firewall rules.

I would suggest adding all *VeeamAgents* to the exclusion list in the firewall and verifying that all necessary ports are open to make it work.

The list of ports can be found in our online help center > http://helpcenter.veeam.com/backup/70/v ... tsub=ports

Hope this helps!

[bkc]

Hi, thanks for your reply.

Before addressing your suggestions I would like to clarify:

1. Veeam installation adds rules to windows firewall to allow incoming connections to proxy and agents.

2. Version 7 release notes do not instruct the end-user to manually adjust windows firewall rules

3. the actual veeamagent.exe that is executing is not one of the ones that had been added to the firewall rules during the installation/upgrade process

4. therefore there is a bug either in the installation, the upgrade process, the remote proxy upgrade process or a glitch in the proxy in that it is launching the wrong veeamagent.exe

The end result is that the veeamagent.exe that is actually running cannot receive incoming tcp connections from outside sources

I certainly can add the actual running veeamagent.exe to the windows firewall rules. But I do not want to do that yet.

I really want to know where the problem lies, wrong agent, or broken install. Also given that the error message shown to the user is rather cryptic (at least in one case) "Client error: No such host is known", I wonder if you'll have other customers encounter this situation and not be able figure it out without opening support cases.

So I think it's worth looking at this a little more.

Now to your response:

Veeam backup manager launches up to 3-4 VeeamAgents per each job, so I believe all of them might be required.

Sorry my question wasn't clear enough. yes the system will launch multiple instances of veeamagent.exe at the same time. I understand that.

My question is that there are 4 different VeeamAgent.exe programs installed on the host. I expect that only one of those agents would be the correct one to run, which one of these is the correct agent:

C:\Program Files (x86)\Veeam\Backup Transport\VeeamAgent.exe
C:\Program Files (x86)\Veeam\Backup Transport\VeeamAgent64.exe
C:\Program Files (x86)\Veeam\Backup Transport\x86\VeeamAgent.exe
C:\Program Files (x86)\Veeam\Backup Transport\x64\VeeamAgent.exe

I bet some of these are left-overs from 6.5, or 7.0 pre-R2 patch

Why is Proxy launching this agent?

Proxy as well as repository is using these data movers (VeeamAgents) to compress, undcompress, dedupe etc. data.

Let me rephrase my question. How does the proxy determine which of the 4 VeeamAgent.exe programs it should run?

Getting back to the nature of the 'bug', either the incorrect/incomplete firewall rules were added during install or upgrade, OR the proxy is launching the wrong agent. Where is the configuration setting
for the proxy that determines which VeeamAgent.exe is the correct one to launch?

The list of ports can be found in our online help center > http://helpcenter.veeam.com/backup/70/v ... tsub=ports

That document is useful to configure intervening firewalls between sites, but should not apply to the actual veeam host itself because veeam software installation adds the necessary rules to windows firewall (or is supposed to)

Summary, pick one:

* missing firewall rules due to broken install/upgrade

* proxy config is messed up and it's launching the wrong agent.

which one is the actual cause of the problem?

ps. the CA veeam host is a mirror image of the US host, and surprise, it has this rule that's not in the US proxy:

Name Group Profile Enabled Action Override Program Local Address Remote Address Protocol Local Port Remote Port Allowed Users Allowed Computers
Veeam Backup Agent (Veeam Backup Transport) (In) Veeam Networking All Yes Allow No C:\Program Files (x86)\Veeam\Backup Transport\x86\VeeamAgent.exe Any Any TCP Any Any Any Any

I guess that answers the question.. install/update didn't add the rule to the US veeam host.

[Vitaliy S.]

1. Veeam installation adds rules to windows firewall to allow incoming connections to proxy and agents.

2. Version 7 release notes do not instruct the end-user to manually adjust windows firewall rules

3. the actual veeamagent.exe that is executing is not one of the ones that had been added to the firewall rules during the installation/upgrade process

4. therefore there is a bug either in the installation, the upgrade process, the remote proxy upgrade process or a glitch in the proxy in that it is launching the wrong veeamagent.exe

The end result is that the veeamagent.exe that is actually running cannot receive incoming tcp connections from outside sources

Yes, you're correct. Everything should be done automatically via setup process.

I certainly can add the actual running veeamagent.exe to the windows firewall rules. But I do not want to do that yet.

I really want to know where the problem lies, wrong agent, or broken install. Also given that the error message shown to the user is rather cryptic (at least in one case) "Client error: No such host is known", I wonder if you'll have other customers encounter this situation and not be able figure it out without opening support cases.

Different agents are triggered based on the activity and system platform (x86/x64) you're running. I haven't seen similar error messages that related to firewall issues before, so it's better to open a support case and investigate why firewall rules haven't been applied properly.

Thanks!

[foggy]

I would add that running different versions of Veeam B&R components on the same server is generally not supported. This fact could cause problems with firewall rules as well. Weren't you considering an upgrade of the second Veeam B&R server?

[fast_cat]

Hi,

I have the same problem on v7 when attempting ~Guest File restores. The connection between the backup server and the repository server fails becasue the Veeam Rule in the Windows firewall on hte Repository server is using the x64 VeeamAgent.exe, whereas the connection is using the x86 path and hence fails unless the firewall is turned off or the rule modified.

I have an open ticket for this - Case # 00496760

So firstly ..which VeeamAgent.exe should be used ? because the one being used currently does not match the one in the Firewall rule.

regards

[Gostev]

x64 agent in case of x64 backup repository... possibly there is a bug setting firewall rule, I will have QC check on this. Thanks!

[Gostev]

We could not reproduce the issue internally, the firewall rules are created correctly (for both 32 and 64 bit agents). Perhaps in your case something or someone has deleted the rule after it has been created.

[pizzim13]

I am experiencing the same firewall issue with my proxies and repos after upgrading from 6.5 to 7r2

[Gostev]

QC engineer also thought this might be upgrade related, and specifically tested upgrade from 6.5 to 7.0 (I did not even ask him to do this). But no issues either, both rules are in place.