Veeam Server 100 % CPU / RPC errors and backup failures

[mkretzer]

Hello,

case 02037727 and 02061258

since the upgrade to WIndows 2016 our application backups get RPC errors every 2-3 runs (Processing XXX Error: Failed to call RPC function 'StartAgent': Timed out requesting agent port for client sessions.)
Whats even a bigger problem is that when that happens CPU usage of our Veeam Server goes up to 100 % and stays like that for some time (sometimes 20 minutes, sometimes four hours).

This is really a problem for us. Increasing the port range might help. But doing that is not working for a long time because the port range for our Veeam Server gets reset to 2500 - 5000 after a short while...

Markus

[mkretzer]

Port range increase did definately not help. Issue still happened.

[rendest]

We are receiving an increase of the following errors, since using 2016 as proxies:

9/02/2017 15:17:04 :: Processing [redacted] Error: Access is denied.

RPC function call failed. Function name: [InvokerTestConnection]. Target machine: [10.3.228.93].

[mkretzer]

The issue still happens if we use a guest interaction proxy with windows server 2012R2 as it seems the issue is really on the Veeam server itself.

When the high cpu is in effect the problematic backups take forever to retry which leads me to believe this in really a Veeam issue.
Increasing the portrange to 10000 - 20000 with a script from support every 30 seconds does not help.

Still, support does not react (primary case is 02065285). How can we escalate such a case?

Markus

[mkretzer]

Still, support does not react (primary case is 02065285). How can we escalate such a case?

Support reacted. Sorry for panicing

[mkretzer]

Just a quick update FYI:
We analyzed the issue with Veeam support and found Microsoft Network Teaming taking up alot of ressources when the issue happens.

We disabled the Network Teaming and updated Network Card Drivers. Right now it looks alot better.

Is someone else using Microsoft Network Teaming with Windows 2016 (the thing introduced in 2012)?

Markus

[jgrote]

We are seeing the same issue since upgrading to WIndows 2016 for our primary server to take advantage of REFS. Not seeing the CPU or memory problems, just the RPC StartAgent problem. A reboot seems to help but it comes back. Got our team opening a ticket on it, will let you know what we find.

[mkretzer]

Finally! We thought it was fixed but then it happened again yesterday! CPU does not always go up to 100%, system is just quite of unresponsive for a few minutes but CPU is sometimes only at 70% - 80%.

We were quite desperate because nobody else seemed to have this.

Veeam support let us do performance recordings with xperf, i uploaded the files this night. I cannot find any driver causing this from my first look in the files...

@jgrote: Can you post the ticket number here? And are you sure that before the error message appears the CPU usage of the Veeam or Repository server does not go up?

[Kas_Tigar]

We have similar issue. Windows Server 2016 as repository with ReFS 3.1 volume. Hotfix KB3216755 already applied.
22.02.2017 02:20:06 :: Failed to compact full backup file Details: Agent: Failed to process method {Transform.CompileFIB}: Not enough storage is available to process this command.
Failed to write data to the file [.temp].
Volume does have 27TB free, Main backup file is 24.6TB
Temp size shown 1,86TB
[ID# 02076088]

Another Windows Server 2016 as repository with ReFS 3.1 volume. Hotfix KB3216755 already applied.
backup failed because repository server utilize 100% CPU (4 * CPU) and freeze. Error:
21.02.2017 19:22:13 :: Processing Error: bad allocation
Failed to upload disk.
Agent failed to process method {DataTransfer.SyncDisk}.
Because repository server does not answering backup server get error
22.02.2017 04:00:31 :: Processing Error: Failed to call RPC function 'StartAgent': Timed out requesting agent port for client sessions.
[ID# 02069864]

After a week struggling with Veeam 9.5 U1, Server 2016 and ReFS 3.1 have to say - at the moment it is useless!

[Gostev]

I am not sure I follow you here, issue similar issue to what? This thread is about memory consumption. Besides, KB3216755 should not be installed on Windows Server 2016 to start with - it is for Windows 10, and it has a known critical bug regardless...

[mkretzer]

Because repository server does not answering backup server get error
22.02.2017 04:00:31 :: Processing Error: Failed to call RPC function 'StartAgent': Timed out requesting agent port for client sessions.
[ID# 02069864]

After a week struggling with Veeam 9.5 U1, Server 2016 and ReFS 3.1 have to say - at the moment it is useless!

Hey our RPC issue! Finally another user!
veeam-backup-replication-f2/veeam-serve ... 40810.html

@Gostev: You still think this is an isolated issue? This is the third customer with this issue. And if you don't check logs or do guest integration backups you might never see that you have a RPC issue...

[Kas_Tigar]

I am not sure I follow you here, issue similar issue to what? This thread is about memory consumption. Besides, KB3216755 should not be installed on Windows Server 2016 to start with - it is for Windows 10, and it has a known critical bug regardless...

I saw the same problem desciption in this thread that repository server becaume unresponsive and have to be reseted. This is my issue too.
Suggestion to install KB I got from Veeam support. Besides on MS article about veeam and refs problems says that this hotfix is for Server 2016 as well.
Critical bug is about SQL, I do not have SQL on Repository servers.

[Gostev]

Critical bug is in System.Data.dll and it will cause its process on backup server to eventually consume all available memory regardless of SQL Server location, so I recommend you uninstall it ASAP from the backup server at least. It *should* be OK to keep on repository servers though.

[Mike Resseler]

There seems to be a replacement KB now for KB3216755 but I don't know if the bug is fixed in that one: https://support.microsoft.com/en-us/help/4010672

[ds2]

DO NOT INSTALL THIS KB!!! It is only the Server-Version of KB3216755. It comes to the same issue as I reported here...

veeam-backup-replication-f2/after-the-r ... ml#p228297

[Mike Resseler]

Hmmm,

I didn't know that. Since KB3216755 can also be installed on Server OS (which many did). But you are probably right and I checked the information on the system.data.dll and the versions are the same in both KB's which means the issue is still there.

[Kas_Tigar]

Critical bug is in System.Data.dll and it will cause its process on backup server to eventually consume all available memory regardless of SQL Server location, so I recommend you uninstall it ASAP from the backup server at least. It *should* be OK to keep on repository servers though.

We do not have any repository on Backup Server. This Hotfix is installed on repository servers only. So should not be a problem.

There seems to be a replacement KB now for KB3216755 but I don't know if the bug is fixed in that one: https://support.microsoft.com/en-us/help/4010672

Yes, I installed this hotfix last week as well on all repository servers. I am not sure this hotfix address to solve this issue but after installing, no server hang was experienced any more.

I got another issue (compact job still failed but no server hang any more and the error is different) after installing this hotfix and working with veeam support to address it.

[Mike Resseler]

Thanks Kas_Tigar for the updated information. Please keep us informed about the other issue also! Very much appreciated

[mkretzer]

Sorry but this is kind of off-topic. The RPC issue has nothing to do with the REFS issues or the MS hotfix.

About the original RPC issue: Veeam support instructed us to create a FileCacheLimitPercent setting in Registry and set it to 0. Since then we had no issues. We are still not 100 % certain that it is finally solved but we are optimistic...

[DaStivi]

anyone still having issues that veeam repository servers with refs used 100% CPU and sometimes freezes then??

as far as i understand every hotfix should be included in the latest cumulative hotfixes, as my server has build 14393.1358 it should have the fixes allready in?

it's a virtual proxy, and locked up all 8 vCPUs, even VMware console does only display blue screen, no logonscreen there...

[Gostev]

I don't believe any ReFS-related hot fixes have made it to Windows Update yet.

[sunta]

no solution yet? saw this week in windows2012R2 environment
"Failed to perform backup file verification Error: Access is denied. RPC function call failed. Function name: [InvokerTestConnection]. Target machine: [10.0.n.n].""

[Gostev]

You should investigate the root cause with support, as Access Denied is a very generic error that can be caused by anything.