High CPU load in idle state (#02222715)

Backup agent for Microsoft Windows servers and workstations (formerly Veeam Endpoint Backup FREE)

High CPU load in idle state (#02222715)

Veeam Logoby NickViz1 » Fri Jul 14, 2017 12:31 pm

Hello,

today I've noticed that Veeam.Endpoint.Service.exe together with the svchost.exe consumes 100% of a single core on my CPU. There is no any backup tasks running at the moment according to Veeam Control Panel.
The CPU load has been happenning for more than 1 hour and machine restart didn't help.

The only noticeable change I can remember - I've installed a new Intel USB3 C220 chipset driver ~10AM. I definitely had no such problem yesterday and I noticed it around 2PM.
Machine restart didn't help, CPU load goes 25% (quad-core Xeon) in 2-3 minutes after restart. It's quite annoying as machine becomes sometimes sluggish - actually that's how I've noticed this issue.

Nikolai
NickViz1
Enthusiast
 
Posts: 38
Liked: 5 times
Joined: Wed Jan 27, 2016 8:50 am
Full Name: Nikolai Vorontsov

Re: High CPU load in idle state (#02222715)

Veeam Logoby NickViz1 » Fri Jul 14, 2017 1:47 pm

Yeah, first line support is usually eh... cheap.
Support: Can you provide me with a screenshot of 100% CPU usage please?
Me: screenshot where Veeam.Endpoint.Service.exe takes 13% and svchost.exe 12%, did other required staff (sfc check, AV check).
Support: On the screenshots you've provided I can see that Veeam.Endpoint.Service.exe uses only 13% of CPU. Also I suggest re-installing Veeam Agent for Windows on this machine.
Me: On quad-core CPU a single looping thread can't use more than 25% of total CPU. And I see that Veeam + svchost (which is probably tickled by veeam) consumes together 25% - exactly one core 100%. Moreover - when I stopped Veeam service - both Veeam and svchost went to 0% CPU usage that confirms my idea that it's Veeam related issue.

Stay tuned, what else he asks me...
NickViz1
Enthusiast
 
Posts: 38
Liked: 5 times
Joined: Wed Jan 27, 2016 8:50 am
Full Name: Nikolai Vorontsov

Re: High CPU load in idle state (#02222715)

Veeam Logoby NickViz1 » Mon Jul 17, 2017 7:18 am

So, I came at work after the weekend and saw that the veeam service paced after 16 hours of work. Needless to say that after system reboot (just of curiosity) it again consumes 1 core.

I don't have any reply from the support yet, so I'm going to reinstall veeam.
NickViz1
Enthusiast
 
Posts: 38
Liked: 5 times
Joined: Wed Jan 27, 2016 8:50 am
Full Name: Nikolai Vorontsov

Re: High CPU load in idle state (#02222715)

Veeam Logoby NickViz1 » Mon Jul 17, 2017 7:50 am

Reinstallation didn't help :-(

Veeam support - any other suggestion?
NickViz1
Enthusiast
 
Posts: 38
Liked: 5 times
Joined: Wed Jan 27, 2016 8:50 am
Full Name: Nikolai Vorontsov

Re: High CPU load in idle state (#02222715)

Veeam Logoby Dima P. » Mon Jul 17, 2017 3:07 pm

Hi Nikolai,

Checking with support.

Edit: Checked your support case - can you please provide support team the requested information?
Dima P.
Veeam Software
 
Posts: 6490
Liked: 452 times
Joined: Mon Feb 04, 2013 2:07 pm
Location: SPb
Full Name: Dmitry Popov

Re: High CPU load in idle state (#02222715)

Veeam Logoby NickViz1 » Tue Jul 18, 2017 7:19 am

Hi Dima,

I've uploaded all requested info, but by my experience it's not enough. The best I did in such cases is to supply customer with instrumented product version with decent logging. From logs you usually get reason much faster..

BTW,
backup itself works.

Nikolai
NickViz1
Enthusiast
 
Posts: 38
Liked: 5 times
Joined: Wed Jan 27, 2016 8:50 am
Full Name: Nikolai Vorontsov

Re: High CPU load in idle state (#02222715)

Veeam Logoby NickViz1 » Tue Jul 18, 2017 1:35 pm

Well, support insists that Windows Update is guilty, personally I think that it's bullshit, for the following reasons:
1. If I stop Windows Update service - CPU load stays the same.
2. When I stop Veeam service - CPU load dropped.
3. There is no Windows Update in the list of that svchost services hosted.
4. CPU load begins after ~5 minutes of Veeam service start.

If I were support I would check with developers what task starts with 5 minutes delay after service start.
NickViz1
Enthusiast
 
Posts: 38
Liked: 5 times
Joined: Wed Jan 27, 2016 8:50 am
Full Name: Nikolai Vorontsov

Re: High CPU load in idle state (#02222715)

Veeam Logoby Dima P. » Mon Jul 24, 2017 1:46 pm

Hello Nikolai,

Sorry for keeping silence. I was in touch with support folks while they were working on your case and all investigation steps look valid. Moreover, the case was reviewed with our QA team.

I see that reinstalling / disabling the AV is not an option. If possible please ask you AV team to whitelist Veeam Agent for Windows in your antivirus as this might solve this issue completely. We received multiple cases with the similar symptoms and the AV was the root cause.
Dima P.
Veeam Software
 
Posts: 6490
Liked: 452 times
Joined: Mon Feb 04, 2013 2:07 pm
Location: SPb
Full Name: Dmitry Popov

Re: High CPU load in idle state (#02222715)

Veeam Logoby NickViz1 » Wed Jul 26, 2017 7:48 am

Hello Dima,

well, they did even more - the AV engine was completely changed :-) Now it's System Center Endpoint Protection. And... the problem still exists.

Personally I'm quite sure that the issue is somehow VSS related, one or another (not crucial) call to VSS is failed and by accident it's in the loop without sleep()/delay(). That leads to 100% single core usage. I guess I can catch the thread ID in Performance monitor and make a dump, so you can get the exact place. Unfortunately I've got proposal for a remote intervention on my machine which is way too far from what is allowed in the company.

As a workaround I'm starting Veeam service only for backup time interval and stopping it later. It also solved my problem that SQL server hangs in memory without reason.

Something like that...
NickViz1
Enthusiast
 
Posts: 38
Liked: 5 times
Joined: Wed Jan 27, 2016 8:50 am
Full Name: Nikolai Vorontsov

Re: High CPU load in idle state (#02222715)

Veeam Logoby Dima P. » Wed Jul 26, 2017 11:02 pm

Nikolai,

If you can please provide us with additional information that may help to investigate this issue further. Process dump works: I can pass it directly to QA and Dev guys. Can you PM me the download link to the file (or let me know if you need some ftp/googledrive space shared)? Thanks in advance!
Dima P.
Veeam Software
 
Posts: 6490
Liked: 452 times
Joined: Mon Feb 04, 2013 2:07 pm
Location: SPb
Full Name: Dmitry Popov

Re: High CPU load in idle state (#02222715)

Veeam Logoby NickViz1 » Thu Jul 27, 2017 8:36 am

Hello Dima,

you can find my previous dumps on your ftp (user ftp02137264). Also I've uploaded a new info. What I did:

1. Run Perf Mon and get thread(s) that consume(s) CPU in Veeam.EndPoint.Service.exe. It was not a single thread as I thought, but interleaved 4 threads - an educated guess is that it's a thread pool in your app.
2. I also got thread IDs for those threads. Actually I guess it's not very profitable as you already know where the thread pool in your app is located :-) What's actually interesting for us - what task(s) those threads are executing, but here I can't help much without version with extra logs.
3. I made dumps for Veeam.EndPoint.Service.exe and svchost.exe that consumes CPU.
4. Uploaded to your FTP.

Hope this helps,
Nikolai
NickViz1
Enthusiast
 
Posts: 38
Liked: 5 times
Joined: Wed Jan 27, 2016 8:50 am
Full Name: Nikolai Vorontsov

Re: High CPU load in idle state (#02222715)

Veeam Logoby Dima P. » Thu Jul 27, 2017 6:04 pm

Nikolai,

Thanks, shared the info with the team. I'll update this thread based on investigation results.
Dima P.
Veeam Software
 
Posts: 6490
Liked: 452 times
Joined: Mon Feb 04, 2013 2:07 pm
Location: SPb
Full Name: Dmitry Popov

Re: High CPU load in idle state (#02222715)

Veeam Logoby NickViz1 » Tue Aug 29, 2017 8:58 am

Hello Dima,

any (good) news? :-)

Nikolai
NickViz1
Enthusiast
 
Posts: 38
Liked: 5 times
Joined: Wed Jan 27, 2016 8:50 am
Full Name: Nikolai Vorontsov

Re: High CPU load in idle state (#02222715)

Veeam Logoby Dima P. » Wed Aug 30, 2017 5:50 pm

Nikolai,

Unfortunately, QA guy responsible for your issue is on a sick leave, so I cant share the latest status.
Dima P.
Veeam Software
 
Posts: 6490
Liked: 452 times
Joined: Mon Feb 04, 2013 2:07 pm
Location: SPb
Full Name: Dmitry Popov

Re: High CPU load in idle state (#02222715)

Veeam Logoby NickViz1 » Wed Sep 13, 2017 11:48 am

Hello Dima,

how is your QA guy health? Any progress? The problem is still reproducible on my machine...

Nikolai
NickViz1
Enthusiast
 
Posts: 38
Liked: 5 times
Joined: Wed Jan 27, 2016 8:50 am
Full Name: Nikolai Vorontsov

Next

Return to Veeam Agent for Windows



Who is online

Users browsing this forum: No registered users and 10 guests