Veeam Server 100 % CPU / RPC errors and backup failures

Availability for the Always-On Enterprise

Veeam Server 100 % CPU / RPC errors and backup failures

Veeam Logoby mkretzer » Tue Feb 07, 2017 10:25 pm

Hello,

case 02037727 and 02061258

since the upgrade to WIndows 2016 our application backups get RPC errors every 2-3 runs (Processing XXX Error: Failed to call RPC function 'StartAgent': Timed out requesting agent port for client sessions.)
Whats even a bigger problem is that when that happens CPU usage of our Veeam Server goes up to 100 % and stays like that for some time (sometimes 20 minutes, sometimes four hours).

This is really a problem for us. Increasing the port range might help. But doing that is not working for a long time because the port range for our Veeam Server gets reset to 2500 - 5000 after a short while...

Markus
mkretzer
Expert
 
Posts: 236
Liked: 54 times
Joined: Thu Dec 17, 2015 7:17 am

Re: Veeam Server 100 % CPU / RPC errors and backup failures

Veeam Logoby mkretzer » Tue Feb 07, 2017 11:35 pm

Port range increase did definately not help. Issue still happened.
mkretzer
Expert
 
Posts: 236
Liked: 54 times
Joined: Thu Dec 17, 2015 7:17 am

Re: Veeam Server 100 % CPU / RPC errors and backup failures

Veeam Logoby rendest » Thu Feb 09, 2017 2:53 pm

We are receiving an increase of the following errors, since using 2016 as proxies:

9/02/2017 15:17:04 :: Processing [redacted] Error: Access is denied.

RPC function call failed. Function name: [InvokerTestConnection]. Target machine: [10.3.228.93].
rendest
Influencer
 
Posts: 18
Liked: 5 times
Joined: Wed Feb 01, 2017 8:36 pm
Full Name: Stef

Re: Veeam Server 100 % CPU / RPC errors and backup failures

Veeam Logoby mkretzer » Fri Feb 10, 2017 1:30 pm

The issue still happens if we use a guest interaction proxy with windows server 2012R2 as it seems the issue is really on the Veeam server itself.

When the high cpu is in effect the problematic backups take forever to retry which leads me to believe this in really a Veeam issue.
Increasing the portrange to 10000 - 20000 with a script from support every 30 seconds does not help.

Still, support does not react (primary case is 02065285). How can we escalate such a case?

Markus
mkretzer
Expert
 
Posts: 236
Liked: 54 times
Joined: Thu Dec 17, 2015 7:17 am

Re: Veeam Server 100 % CPU / RPC errors and backup failures

Veeam Logoby mkretzer » Fri Feb 10, 2017 2:13 pm

mkretzer wrote:Still, support does not react (primary case is 02065285). How can we escalate such a case?

Support reacted. Sorry for panicing :-)
mkretzer
Expert
 
Posts: 236
Liked: 54 times
Joined: Thu Dec 17, 2015 7:17 am

Re: Veeam Server 100 % CPU / RPC errors and backup failures

Veeam Logoby mkretzer » Thu Feb 16, 2017 9:06 am 1 person likes this post

Just a quick update FYI:
We analyzed the issue with Veeam support and found Microsoft Network Teaming taking up alot of ressources when the issue happens.

We disabled the Network Teaming and updated Network Card Drivers. Right now it looks alot better.

Is someone else using Microsoft Network Teaming with Windows 2016 (the thing introduced in 2012)?

Markus
mkretzer
Expert
 
Posts: 236
Liked: 54 times
Joined: Thu Dec 17, 2015 7:17 am

Re: Veeam Server 100 % CPU / RPC errors and backup failures

Veeam Logoby jgrote » Sat Feb 18, 2017 1:15 am 1 person likes this post

We are seeing the same issue since upgrading to WIndows 2016 for our primary server to take advantage of REFS. Not seeing the CPU or memory problems, just the RPC StartAgent problem. A reboot seems to help but it comes back. Got our team opening a ticket on it, will let you know what we find.
jgrote
Novice
 
Posts: 7
Liked: 4 times
Joined: Tue Jul 13, 2010 12:14 am
Full Name: Justin Grote

Re: Veeam Server 100 % CPU / RPC errors and backup failures

Veeam Logoby mkretzer » Sat Feb 18, 2017 7:47 am

Finally! We thought it was fixed but then it happened again yesterday! CPU does not always go up to 100%, system is just quite of unresponsive for a few minutes but CPU is sometimes only at 70% - 80%.

We were quite desperate because nobody else seemed to have this.

Veeam support let us do performance recordings with xperf, i uploaded the files this night. I cannot find any driver causing this from my first look in the files...

@jgrote: Can you post the ticket number here? And are you sure that before the error message appears the CPU usage of the Veeam or Repository server does not go up?
mkretzer
Expert
 
Posts: 236
Liked: 54 times
Joined: Thu Dec 17, 2015 7:17 am

[MERGED] 9.5/ReFS/Server 2016 Memory Consumption

Veeam Logoby Kas_Tigar » Wed Feb 22, 2017 9:24 am

We have similar issue. Windows Server 2016 as repository with ReFS 3.1 volume. Hotfix KB3216755 already applied.
22.02.2017 02:20:06 :: Failed to compact full backup file Details: Agent: Failed to process method {Transform.CompileFIB}: Not enough storage is available to process this command.
Failed to write data to the file [.temp].
Volume does have 27TB free, Main backup file is 24.6TB
Temp size shown 1,86TB
[ID# 02076088]

Another Windows Server 2016 as repository with ReFS 3.1 volume. Hotfix KB3216755 already applied.
backup failed because repository server utilize 100% CPU (4 * CPU) and freeze. Error:
21.02.2017 19:22:13 :: Processing Error: bad allocation
Failed to upload disk.
Agent failed to process method {DataTransfer.SyncDisk}.
Because repository server does not answering backup server get error
22.02.2017 04:00:31 :: Processing Error: Failed to call RPC function 'StartAgent': Timed out requesting agent port for client sessions.
[ID# 02069864]

After a week struggling with Veeam 9.5 U1, Server 2016 and ReFS 3.1 have to say - at the moment it is useless!
Kas_Tigar
Influencer
 
Posts: 12
Liked: 2 times
Joined: Thu Jan 29, 2015 9:18 am

Re: 9.5/ReFS/Server 2016 Memory Consumption

Veeam Logoby Gostev » Wed Feb 22, 2017 3:32 pm 1 person likes this post

I am not sure I follow you here, issue similar issue to what? This thread is about memory consumption. Besides, KB3216755 should not be installed on Windows Server 2016 to start with - it is for Windows 10, and it has a known critical bug regardless...
Gostev
Veeam Software
 
Posts: 21039
Liked: 2266 times
Joined: Sun Jan 01, 2006 1:01 am
Full Name: Anton Gostev

Re: 9.5/ReFS/Server 2016 Memory Consumption

Veeam Logoby mkretzer » Wed Feb 22, 2017 4:14 pm

Kas_Tigar wrote:Because repository server does not answering backup server get error
22.02.2017 04:00:31 :: Processing Error: Failed to call RPC function 'StartAgent': Timed out requesting agent port for client sessions.
[ID# 02069864]

After a week struggling with Veeam 9.5 U1, Server 2016 and ReFS 3.1 have to say - at the moment it is useless!


Hey our RPC issue! Finally another user!
veeam-backup-replication-f2/veeam-server-100-cpu-rpc-errors-and-backup-failures-t40810.html

@Gostev: You still think this is an isolated issue? This is the third customer with this issue. And if you don't check logs or do guest integration backups you might never see that you have a RPC issue...
mkretzer
Expert
 
Posts: 236
Liked: 54 times
Joined: Thu Dec 17, 2015 7:17 am

Re: 9.5/ReFS/Server 2016 Memory Consumption

Veeam Logoby Kas_Tigar » Thu Feb 23, 2017 7:24 am

Gostev wrote:I am not sure I follow you here, issue similar issue to what? This thread is about memory consumption. Besides, KB3216755 should not be installed on Windows Server 2016 to start with - it is for Windows 10, and it has a known critical bug regardless...

I saw the same problem desciption in this thread that repository server becaume unresponsive and have to be reseted. This is my issue too.
Suggestion to install KB I got from Veeam support. Besides on MS article about veeam and refs problems says that this hotfix is for Server 2016 as well.
Critical bug is about SQL, I do not have SQL on Repository servers.
Kas_Tigar
Influencer
 
Posts: 12
Liked: 2 times
Joined: Thu Jan 29, 2015 9:18 am

Re: 9.5/ReFS/Server 2016 Memory Consumption

Veeam Logoby Gostev » Thu Feb 23, 2017 8:48 pm

Critical bug is in System.Data.dll and it will cause its process on backup server to eventually consume all available memory regardless of SQL Server location, so I recommend you uninstall it ASAP from the backup server at least. It *should* be OK to keep on repository servers though.
Gostev
Veeam Software
 
Posts: 21039
Liked: 2266 times
Joined: Sun Jan 01, 2006 1:01 am
Full Name: Anton Gostev

Re: Veeam Server 100 % CPU / RPC errors and backup failures

Veeam Logoby Mike Resseler » Fri Feb 24, 2017 9:11 am

There seems to be a replacement KB now for KB3216755 but I don't know if the bug is fixed in that one: https://support.microsoft.com/en-us/help/4010672
Mike Resseler
Veeam Software
 
Posts: 2604
Liked: 314 times
Joined: Fri Feb 08, 2013 3:08 pm
Location: Belgium, the land of the fries, the beer, the chocolate and the diamonds...
Full Name: Mike Resseler

Re: Veeam Server 100 % CPU / RPC errors and backup failures

Veeam Logoby ds2 » Fri Feb 24, 2017 11:01 am 1 person likes this post

DO NOT INSTALL THIS KB!!! It is only the Server-Version of KB3216755. It comes to the same issue as I reported here...

veeam-backup-replication-f2/after-the-refs-4k-horror-story-new-problem-t40720-30.html#p228297
ds2
Enthusiast
 
Posts: 69
Liked: 15 times
Joined: Thu Jul 16, 2015 6:31 am
Full Name: Rene Keller

Next

Return to Veeam Backup & Replication



Who is online

Users browsing this forum: Bing [Bot], Google [Bot], seananaya and 32 guests