Comprehensive data protection for all workloads
Post Reply
mkretzer
Veeam Legend
Posts: 1140
Liked: 387 times
Joined: Dec 17, 2015 7:17 am
Contact:

Veeam Server 100 % CPU / RPC errors and backup failures

Post by mkretzer »

Hello,

case 02037727 and 02061258

since the upgrade to WIndows 2016 our application backups get RPC errors every 2-3 runs (Processing XXX Error: Failed to call RPC function 'StartAgent': Timed out requesting agent port for client sessions.)
Whats even a bigger problem is that when that happens CPU usage of our Veeam Server goes up to 100 % and stays like that for some time (sometimes 20 minutes, sometimes four hours).

This is really a problem for us. Increasing the port range might help. But doing that is not working for a long time because the port range for our Veeam Server gets reset to 2500 - 5000 after a short while...

Markus
mkretzer
Veeam Legend
Posts: 1140
Liked: 387 times
Joined: Dec 17, 2015 7:17 am
Contact:

Re: Veeam Server 100 % CPU / RPC errors and backup failures

Post by mkretzer »

Port range increase did definately not help. Issue still happened.
rendest
Influencer
Posts: 20
Liked: 6 times
Joined: Feb 01, 2017 8:36 pm
Full Name: Stef
Contact:

Re: Veeam Server 100 % CPU / RPC errors and backup failures

Post by rendest »

We are receiving an increase of the following errors, since using 2016 as proxies:

9/02/2017 15:17:04 :: Processing [redacted] Error: Access is denied.

RPC function call failed. Function name: [InvokerTestConnection]. Target machine: [10.3.228.93].
mkretzer
Veeam Legend
Posts: 1140
Liked: 387 times
Joined: Dec 17, 2015 7:17 am
Contact:

Re: Veeam Server 100 % CPU / RPC errors and backup failures

Post by mkretzer »

The issue still happens if we use a guest interaction proxy with windows server 2012R2 as it seems the issue is really on the Veeam server itself.

When the high cpu is in effect the problematic backups take forever to retry which leads me to believe this in really a Veeam issue.
Increasing the portrange to 10000 - 20000 with a script from support every 30 seconds does not help.

Still, support does not react (primary case is 02065285). How can we escalate such a case?

Markus
mkretzer
Veeam Legend
Posts: 1140
Liked: 387 times
Joined: Dec 17, 2015 7:17 am
Contact:

Re: Veeam Server 100 % CPU / RPC errors and backup failures

Post by mkretzer »

mkretzer wrote: Still, support does not react (primary case is 02065285). How can we escalate such a case?
Support reacted. Sorry for panicing :-)
mkretzer
Veeam Legend
Posts: 1140
Liked: 387 times
Joined: Dec 17, 2015 7:17 am
Contact:

Re: Veeam Server 100 % CPU / RPC errors and backup failures

Post by mkretzer » 1 person likes this post

Just a quick update FYI:
We analyzed the issue with Veeam support and found Microsoft Network Teaming taking up alot of ressources when the issue happens.

We disabled the Network Teaming and updated Network Card Drivers. Right now it looks alot better.

Is someone else using Microsoft Network Teaming with Windows 2016 (the thing introduced in 2012)?

Markus
jgrote
Influencer
Posts: 13
Liked: 4 times
Joined: Jul 13, 2010 12:14 am
Full Name: Justin Grote
Contact:

Re: Veeam Server 100 % CPU / RPC errors and backup failures

Post by jgrote » 1 person likes this post

We are seeing the same issue since upgrading to WIndows 2016 for our primary server to take advantage of REFS. Not seeing the CPU or memory problems, just the RPC StartAgent problem. A reboot seems to help but it comes back. Got our team opening a ticket on it, will let you know what we find.
mkretzer
Veeam Legend
Posts: 1140
Liked: 387 times
Joined: Dec 17, 2015 7:17 am
Contact:

Re: Veeam Server 100 % CPU / RPC errors and backup failures

Post by mkretzer »

Finally! We thought it was fixed but then it happened again yesterday! CPU does not always go up to 100%, system is just quite of unresponsive for a few minutes but CPU is sometimes only at 70% - 80%.

We were quite desperate because nobody else seemed to have this.

Veeam support let us do performance recordings with xperf, i uploaded the files this night. I cannot find any driver causing this from my first look in the files...

@jgrote: Can you post the ticket number here? And are you sure that before the error message appears the CPU usage of the Veeam or Repository server does not go up?
Kas_Tigar
Influencer
Posts: 13
Liked: 2 times
Joined: Jan 29, 2015 9:18 am
Contact:

[MERGED] 9.5/ReFS/Server 2016 Memory Consumption

Post by Kas_Tigar »

We have similar issue. Windows Server 2016 as repository with ReFS 3.1 volume. Hotfix KB3216755 already applied.
22.02.2017 02:20:06 :: Failed to compact full backup file Details: Agent: Failed to process method {Transform.CompileFIB}: Not enough storage is available to process this command.
Failed to write data to the file [.temp].
Volume does have 27TB free, Main backup file is 24.6TB
Temp size shown 1,86TB
[ID# 02076088]

Another Windows Server 2016 as repository with ReFS 3.1 volume. Hotfix KB3216755 already applied.
backup failed because repository server utilize 100% CPU (4 * CPU) and freeze. Error:
21.02.2017 19:22:13 :: Processing Error: bad allocation
Failed to upload disk.
Agent failed to process method {DataTransfer.SyncDisk}.
Because repository server does not answering backup server get error
22.02.2017 04:00:31 :: Processing Error: Failed to call RPC function 'StartAgent': Timed out requesting agent port for client sessions.
[ID# 02069864]

After a week struggling with Veeam 9.5 U1, Server 2016 and ReFS 3.1 have to say - at the moment it is useless!
Gostev
Chief Product Officer
Posts: 31460
Liked: 6648 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: 9.5/ReFS/Server 2016 Memory Consumption

Post by Gostev » 1 person likes this post

I am not sure I follow you here, issue similar issue to what? This thread is about memory consumption. Besides, KB3216755 should not be installed on Windows Server 2016 to start with - it is for Windows 10, and it has a known critical bug regardless...
mkretzer
Veeam Legend
Posts: 1140
Liked: 387 times
Joined: Dec 17, 2015 7:17 am
Contact:

Re: 9.5/ReFS/Server 2016 Memory Consumption

Post by mkretzer »

Kas_Tigar wrote: Because repository server does not answering backup server get error
22.02.2017 04:00:31 :: Processing Error: Failed to call RPC function 'StartAgent': Timed out requesting agent port for client sessions.
[ID# 02069864]

After a week struggling with Veeam 9.5 U1, Server 2016 and ReFS 3.1 have to say - at the moment it is useless!
Hey our RPC issue! Finally another user!
veeam-backup-replication-f2/veeam-serve ... 40810.html

@Gostev: You still think this is an isolated issue? This is the third customer with this issue. And if you don't check logs or do guest integration backups you might never see that you have a RPC issue...
Kas_Tigar
Influencer
Posts: 13
Liked: 2 times
Joined: Jan 29, 2015 9:18 am
Contact:

Re: 9.5/ReFS/Server 2016 Memory Consumption

Post by Kas_Tigar »

Gostev wrote:I am not sure I follow you here, issue similar issue to what? This thread is about memory consumption. Besides, KB3216755 should not be installed on Windows Server 2016 to start with - it is for Windows 10, and it has a known critical bug regardless...
I saw the same problem desciption in this thread that repository server becaume unresponsive and have to be reseted. This is my issue too.
Suggestion to install KB I got from Veeam support. Besides on MS article about veeam and refs problems says that this hotfix is for Server 2016 as well.
Critical bug is about SQL, I do not have SQL on Repository servers.
Gostev
Chief Product Officer
Posts: 31460
Liked: 6648 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: 9.5/ReFS/Server 2016 Memory Consumption

Post by Gostev »

Critical bug is in System.Data.dll and it will cause its process on backup server to eventually consume all available memory regardless of SQL Server location, so I recommend you uninstall it ASAP from the backup server at least. It *should* be OK to keep on repository servers though.
Mike Resseler
Product Manager
Posts: 8044
Liked: 1263 times
Joined: Feb 08, 2013 3:08 pm
Full Name: Mike Resseler
Location: Belgium
Contact:

Re: Veeam Server 100 % CPU / RPC errors and backup failures

Post by Mike Resseler »

There seems to be a replacement KB now for KB3216755 but I don't know if the bug is fixed in that one: https://support.microsoft.com/en-us/help/4010672
ds2
Enthusiast
Posts: 82
Liked: 19 times
Joined: Jul 16, 2015 6:31 am
Full Name: Rene Keller
Contact:

Re: Veeam Server 100 % CPU / RPC errors and backup failures

Post by ds2 » 1 person likes this post

DO NOT INSTALL THIS KB!!! It is only the Server-Version of KB3216755. It comes to the same issue as I reported here...

veeam-backup-replication-f2/after-the-r ... ml#p228297
Mike Resseler
Product Manager
Posts: 8044
Liked: 1263 times
Joined: Feb 08, 2013 3:08 pm
Full Name: Mike Resseler
Location: Belgium
Contact:

Re: Veeam Server 100 % CPU / RPC errors and backup failures

Post by Mike Resseler »

Hmmm,

I didn't know that. Since KB3216755 can also be installed on Server OS (which many did). But you are probably right and I checked the information on the system.data.dll and the versions are the same in both KB's which means the issue is still there.
Kas_Tigar
Influencer
Posts: 13
Liked: 2 times
Joined: Jan 29, 2015 9:18 am
Contact:

Re: 9.5/ReFS/Server 2016 Memory Consumption

Post by Kas_Tigar » 1 person likes this post

Gostev wrote:Critical bug is in System.Data.dll and it will cause its process on backup server to eventually consume all available memory regardless of SQL Server location, so I recommend you uninstall it ASAP from the backup server at least. It *should* be OK to keep on repository servers though.
We do not have any repository on Backup Server. This Hotfix is installed on repository servers only. So should not be a problem.
Mike Resseler wrote:There seems to be a replacement KB now for KB3216755 but I don't know if the bug is fixed in that one: https://support.microsoft.com/en-us/help/4010672
Yes, I installed this hotfix last week as well on all repository servers. I am not sure this hotfix address to solve this issue but after installing, no server hang was experienced any more.

I got another issue (compact job still failed but no server hang any more and the error is different) after installing this hotfix and working with veeam support to address it.
Mike Resseler
Product Manager
Posts: 8044
Liked: 1263 times
Joined: Feb 08, 2013 3:08 pm
Full Name: Mike Resseler
Location: Belgium
Contact:

Re: Veeam Server 100 % CPU / RPC errors and backup failures

Post by Mike Resseler »

Thanks Kas_Tigar for the updated information. Please keep us informed about the other issue also! Very much appreciated
mkretzer
Veeam Legend
Posts: 1140
Liked: 387 times
Joined: Dec 17, 2015 7:17 am
Contact:

Re: Veeam Server 100 % CPU / RPC errors and backup failures

Post by mkretzer » 2 people like this post

Sorry but this is kind of off-topic. The RPC issue has nothing to do with the REFS issues or the MS hotfix.

About the original RPC issue: Veeam support instructed us to create a FileCacheLimitPercent setting in Registry and set it to 0. Since then we had no issues. We are still not 100 % certain that it is finally solved but we are optimistic...
DaStivi
Service Provider
Posts: 250
Liked: 34 times
Joined: Jun 30, 2015 9:13 am
Full Name: Stephan Lang
Location: Austria
Contact:

Re: Veeam Server 100 % CPU / RPC errors and backup failures

Post by DaStivi »

anyone still having issues that veeam repository servers with refs used 100% CPU and sometimes freezes then??

as far as i understand every hotfix should be included in the latest cumulative hotfixes, as my server has build 14393.1358 it should have the fixes allready in?

it's a virtual proxy, and locked up all 8 vCPUs, even VMware console does only display blue screen, no logonscreen there...
Gostev
Chief Product Officer
Posts: 31460
Liked: 6648 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Veeam Server 100 % CPU / RPC errors and backup failures

Post by Gostev »

I don't believe any ReFS-related hot fixes have made it to Windows Update yet.
sunta
Lurker
Posts: 1
Liked: never
Joined: Jul 18, 2018 10:27 am
Contact:

Re: Veeam Server 100 % CPU / RPC errors and backup failures

Post by sunta »

no solution yet? saw this week in windows2012R2 environment
"Failed to perform backup file verification Error: Access is denied. RPC function call failed. Function name: [InvokerTestConnection]. Target machine: [10.0.n.n].""
Gostev
Chief Product Officer
Posts: 31460
Liked: 6648 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Veeam Server 100 % CPU / RPC errors and backup failures

Post by Gostev »

You should investigate the root cause with support, as Access Denied is a very generic error that can be caused by anything.
Post Reply

Who is online

Users browsing this forum: No registered users and 165 guests