Port Exhaustion

[micoolpaul]

Case Number: 06154615 (This case was accidentally closed by a colleague, I'm getting a new one raised via case administrator, will update post when I've got the new one.)

Are there any key logs I can look at to determine why so many RPC ports are being consumed by a backup system.

The architecture is quite flat. Two sites, governed by one VBR server Virtual Machine (6 vCPU, 24GB RAM). 2x Physical servers per site with the VMware & Agent Proxy Server Roles + Backup Repository roles. These servers are pretty beefy, 48 cores each, 192GB RAM, SSD boot volume, 40GbE networking etc.

At the time of writing, the server has 23 jobs 'running', in reality most are queued. The VBR server doesn't have any proxy/repository/any type of role deployed to it, all the jobs that can use guest interaction proxy have physical servers explicitly defined to exclude the VBR server also. In short, I've slimmed down the VBR's footprint as much as possible to ensure it is only orchestrating workloads, not processing them.

The problem I'm seeing is that I've got my jobs queued up, the breakdown of which is:
7x File Backup Jobs
14x Backup Copy Jobs
2x SOBR Tiering Jobs

But if I do a netstat -abno output, I can see Veeam.BackupManager.exe is consuming over 1.5k RPC ports connecting to a lot of the same ports on these servers again and again.

Now I know that I'm provisioning multiple roles to them, and have things such as multiple repositories on them (approximately 20 per server, long story but was justified by one of your Systems Architects related to the way RMAN needs to work) but this feels extremely excessive for a server that should only be orchestrating.

Does this sound expected? The reason I'm raising this is that intermittently the server is becoming completely exhausted of RPC ports, and then jobs will randomly fail as Veeam becomes unable to use an RPC port to check the health of a repository for example.

Additional case reference: 06208303

Thanks.
Michael

[PetrM]

Hi Michael,

It does not sound expected, please continue working with our support team on this issue.

Thanks!

[gmcoburn]

@micoolpaul

We are experiencing this issue as well. Have you had a resolution?

[micoolpaul]

Hi @gmcoburn

Case still ongoing, so far they’ve said the common reasons are:
AV exclusions
Firewalls closing ports and Veeam being unaware
Repositories without concurrency limits (or excessively high limits)

None of these are our issue however, but I hope by sharing these that one might help you.

[gmcoburn]

Thanks Michael,

I'll review these settings, but I'm pretty sure were covered off on them. While our repo concurrencies aren't the default 4, there are in the range of 6 to 8 depending on the cpu in the repo, so nothing excessive.

I'll confirm our AV exclusions, and confirm the FW traffic is OK.

I did just see your earlier community post where you said the same things, but it looks like it's still an issue based on your comments here. Ours just came out of the blue, with Veeam running reliably for a long time beforehand.

Thanks for the guidance.

George