Failed to upgrade proxy agent

[techK]

Updated my V365 to the latest v8 and it seems to not able to update my proxy agent, not able to allow me to do backups. Error its throwing:

Authentication failed, see inner exception. Rollback error: Handshake failed. Reason:Unable to read data from the transport connection. An existing connection was forcibly closed by the remote host.

Case for this: Case #07407217 but have not gotten an update since Friday.

Firewall on the proxy server is off and still throws that error, not sure what's going on?

[G_Mellor]

Have you tried disabling the firewall on the Veeam for M365 server itself? Obviously not as a permanent solution, but just to prove where the issue lies. I had the same issue on Friday when upgrading.

There's a couple of new ports on the main App server that the Proxy needs to be able to connect to: NATS on port 4222 and PostgreSQL on port 5432.

Veeam B&R creates Windows firewall rules for it's components when they're installed - it would be very nice if Veeam for M365 would do the same! Yes, the ports are documented (https://helpcenter.veeam.com/docs/vbo36 ... tml?ver=80), but it's not clear at first read through that the last two entries are new and that each proxy needs to be able to talk to PostgreSQL and NATS.

[Mike Resseler]

@G_Mellor Good thinking!

@techK This might indeed be the issue. In v8 has significant architecture changes to support huge environments and moving away from the v7 and previous config database to PostgreSQL, and using NATS for queuing the backup of the separate objects is another one (also proxy pools, make sure you read up on that if you have a bit of a larger environment). For the creation of those rules, I will investigate why it hasn't done this. I might be wrong but it does it in a new installation, maybe it failed because it was an upgrade, need to find out.

[techK]

Have you tried disabling the firewall on the Veeam for M365 server itself? Obviously not as a permanent solution, but just to prove where the issue lies. I had the same issue on Friday when upgrading.

There's a couple of new ports on the main App server that the Proxy needs to be able to connect to: NATS on port 4222 and PostgreSQL on port 5432.

Veeam B&R creates Windows firewall rules for it's components when they're installed - it would be very nice if Veeam for M365 would do the same! Yes, the ports are documented (https://helpcenter.veeam.com/docs/vbo36 ... tml?ver=80), but it's not clear at first read through that the last two entries are new and that each proxy needs to be able to talk to PostgreSQL and NATS.

Thanks, I opened the ports on the proxy server, but still not connecting even with firewall disabled on both proxy and server and even opened the ports on the server.

[Mike Resseler]

In that case, please create a support ticket and post the ticket ID here. That means something else is wrong I'm afraid

[sebkoe]

i have the same issue when i try to upgrade my proxy and also no answer yet. Case is #03365075

[dotdk]

Was this problem resolved? We are seeing the same problem but still no resolution with support.

[sebkoe]

nope, i had to do a rollback.

[jasonede]

I had this issue and then found there are some pre-requisites that need to be manually updated on each proxy and then the update ran through ok. https://helpcenter.veeam.com/docs/vbo36 ... tml?ver=80

[dotdk]

Ouch, what was the conclusion by support?

[dotdk]

I had this issue and then found there are some pre-requisites that need to be manually updated on each proxy and then the update ran through ok. https://helpcenter.veeam.com/docs/vbo36 ... tml?ver=80

Can you share if there were any other prereqs than "Microsoft .NET and Windows PowerShell must be installed on a target machine in advance."

[jasonede]

I just updated those 2. I'm sure I found the exact versions needed somewhere but can't find that info now.

[adrian_mth]

Hi, I have exactly the same problem.
.NET and Powershell 7 are installed. Firewall is deactivated on the VBO and the proxy. Nevertheless, I run into the problem again and again. I have already tried to upgrade the HW (VM) so that there is simply more ‘power’ available for the update, but that doesn't change anything either. I've been trying to install v8 in the now different versions on our dev environment (1:1 like Prod) since release. So far it hasn't worked once. I have already opened a ticket about this. The solution should be: Delete the entire proxy, change all jobs to the default repo and completely reinstall the proxy, reconnect all repos and then change the jobs back to the correct proxies/repos. Sorry but this is not a possible solution for us especially not if we have to do the same in the prod environment!!

[adrian_mth]

Any Update here?

[Polina]

Hi Adrian,

Can you please send me your support case ID?

[adrian_mth]

The problem sounded like a general problem so I did not open a case until now and hoped for a solution from the others. I have now opened a case because the problem still exists: #07506859

[adrian_mth]

We were able to solve the problem ourselves. We looked at the Windows logs and found that there were some NTLM Auth errors. As our AD is very hardened, NTLM is deactivated. We just re-enabled it for the veeam servers and the update worked. Maybe this will help one or the other.

[JRRW]

FWIW
Upgraded from 7 > 8
Console/Proxy on Server 2019 Datacenter UI / Server 2019 Datacenter Core

• Had to manually install Powershell 7.5 on the Proxy (it claimed 7.4, but installing 7.4 did not resolve)
• Had to install .net 8 RUNTIME
• Had to open ports 4222 and 5432
• Had to reboot my Proxy - it just didn't like something despite all of this

Here's a dirty PS script for the firewall stuff. I was going to script out the download/install of the apps but I'm too lazy at the moment. Note: This is all ports that might be needed by backup for M365 - customize if you don't need like, 25 open.

Code: Select all

$ruleName = "Veeam M365 Portz"

# Remove existing rule (if exists)
if (Get-NetFirewallRule -DisplayName $ruleName -ErrorAction SilentlyContinue) {
    Remove-NetFirewallRule -DisplayName $ruleName -Confirm:$false
}

# Define all required ports - this includes some that not all proxies/consoles may need
$allPorts = "9192,9191,5432,4222,443,445,135,5985,5986,25,465,587,9193"

# Create the firewall rule based on what you listed before
New-NetFirewallRule -DisplayName $ruleName `
    -Direction Inbound `
    -Action Allow `
    -Profile Any `
    -EdgeTraversalPolicy Block `
    -Protocol TCP `
    -LocalPort $allPorts `
    -RemotePort Any `
    -LocalAddress Any `
    -RemoteAddress Any `
    -Program Any `
    -Service Any

Once I did this? Worked fine... /shrug not sure why this wasn't QA'd before release though - upgrades should be your primary focus on QA IMHO.