Standalone backup agent for Microsoft Windows servers and workstations (formerly Veeam Endpoint Backup FREE)
Post Reply
NickViz1
Enthusiast
Posts: 41
Liked: 5 times
Joined: Jan 27, 2016 8:50 am
Full Name: Nikolai Vorontsov
Contact:

High CPU load in idle state (#02222715)

Post by NickViz1 »

Hello,

today I've noticed that Veeam.Endpoint.Service.exe together with the svchost.exe consumes 100% of a single core on my CPU. There is no any backup tasks running at the moment according to Veeam Control Panel.
The CPU load has been happenning for more than 1 hour and machine restart didn't help.

The only noticeable change I can remember - I've installed a new Intel USB3 C220 chipset driver ~10AM. I definitely had no such problem yesterday and I noticed it around 2PM.
Machine restart didn't help, CPU load goes 25% (quad-core Xeon) in 2-3 minutes after restart. It's quite annoying as machine becomes sometimes sluggish - actually that's how I've noticed this issue.

Nikolai
NickViz1
Enthusiast
Posts: 41
Liked: 5 times
Joined: Jan 27, 2016 8:50 am
Full Name: Nikolai Vorontsov
Contact:

Re: High CPU load in idle state (#02222715)

Post by NickViz1 »

Yeah, first line support is usually eh... cheap.
Support: Can you provide me with a screenshot of 100% CPU usage please?
Me: screenshot where Veeam.Endpoint.Service.exe takes 13% and svchost.exe 12%, did other required staff (sfc check, AV check).
Support: On the screenshots you've provided I can see that Veeam.Endpoint.Service.exe uses only 13% of CPU. Also I suggest re-installing Veeam Agent for Windows on this machine.
Me: On quad-core CPU a single looping thread can't use more than 25% of total CPU. And I see that Veeam + svchost (which is probably tickled by veeam) consumes together 25% - exactly one core 100%. Moreover - when I stopped Veeam service - both Veeam and svchost went to 0% CPU usage that confirms my idea that it's Veeam related issue.

Stay tuned, what else he asks me...
NickViz1
Enthusiast
Posts: 41
Liked: 5 times
Joined: Jan 27, 2016 8:50 am
Full Name: Nikolai Vorontsov
Contact:

Re: High CPU load in idle state (#02222715)

Post by NickViz1 »

So, I came at work after the weekend and saw that the veeam service paced after 16 hours of work. Needless to say that after system reboot (just of curiosity) it again consumes 1 core.

I don't have any reply from the support yet, so I'm going to reinstall veeam.
NickViz1
Enthusiast
Posts: 41
Liked: 5 times
Joined: Jan 27, 2016 8:50 am
Full Name: Nikolai Vorontsov
Contact:

Re: High CPU load in idle state (#02222715)

Post by NickViz1 »

Reinstallation didn't help :-(

Veeam support - any other suggestion?
Dima P.
Product Manager
Posts: 14396
Liked: 1568 times
Joined: Feb 04, 2013 2:07 pm
Full Name: Dmitry Popov
Location: Prague
Contact:

Re: High CPU load in idle state (#02222715)

Post by Dima P. »

Hi Nikolai,

Checking with support.

Edit: Checked your support case - can you please provide support team the requested information?
NickViz1
Enthusiast
Posts: 41
Liked: 5 times
Joined: Jan 27, 2016 8:50 am
Full Name: Nikolai Vorontsov
Contact:

Re: High CPU load in idle state (#02222715)

Post by NickViz1 »

Hi Dima,

I've uploaded all requested info, but by my experience it's not enough. The best I did in such cases is to supply customer with instrumented product version with decent logging. From logs you usually get reason much faster..

BTW,
backup itself works.

Nikolai
NickViz1
Enthusiast
Posts: 41
Liked: 5 times
Joined: Jan 27, 2016 8:50 am
Full Name: Nikolai Vorontsov
Contact:

Re: High CPU load in idle state (#02222715)

Post by NickViz1 »

Well, support insists that Windows Update is guilty, personally I think that it's bullshit, for the following reasons:
1. If I stop Windows Update service - CPU load stays the same.
2. When I stop Veeam service - CPU load dropped.
3. There is no Windows Update in the list of that svchost services hosted.
4. CPU load begins after ~5 minutes of Veeam service start.

If I were support I would check with developers what task starts with 5 minutes delay after service start.
Dima P.
Product Manager
Posts: 14396
Liked: 1568 times
Joined: Feb 04, 2013 2:07 pm
Full Name: Dmitry Popov
Location: Prague
Contact:

Re: High CPU load in idle state (#02222715)

Post by Dima P. »

Hello Nikolai,

Sorry for keeping silence. I was in touch with support folks while they were working on your case and all investigation steps look valid. Moreover, the case was reviewed with our QA team.

I see that reinstalling / disabling the AV is not an option. If possible please ask you AV team to whitelist Veeam Agent for Windows in your antivirus as this might solve this issue completely. We received multiple cases with the similar symptoms and the AV was the root cause.
NickViz1
Enthusiast
Posts: 41
Liked: 5 times
Joined: Jan 27, 2016 8:50 am
Full Name: Nikolai Vorontsov
Contact:

Re: High CPU load in idle state (#02222715)

Post by NickViz1 »

Hello Dima,

well, they did even more - the AV engine was completely changed :-) Now it's System Center Endpoint Protection. And... the problem still exists.

Personally I'm quite sure that the issue is somehow VSS related, one or another (not crucial) call to VSS is failed and by accident it's in the loop without sleep()/delay(). That leads to 100% single core usage. I guess I can catch the thread ID in Performance monitor and make a dump, so you can get the exact place. Unfortunately I've got proposal for a remote intervention on my machine which is way too far from what is allowed in the company.

As a workaround I'm starting Veeam service only for backup time interval and stopping it later. It also solved my problem that SQL server hangs in memory without reason.

Something like that...
Dima P.
Product Manager
Posts: 14396
Liked: 1568 times
Joined: Feb 04, 2013 2:07 pm
Full Name: Dmitry Popov
Location: Prague
Contact:

Re: High CPU load in idle state (#02222715)

Post by Dima P. »

Nikolai,

If you can please provide us with additional information that may help to investigate this issue further. Process dump works: I can pass it directly to QA and Dev guys. Can you PM me the download link to the file (or let me know if you need some ftp/googledrive space shared)? Thanks in advance!
NickViz1
Enthusiast
Posts: 41
Liked: 5 times
Joined: Jan 27, 2016 8:50 am
Full Name: Nikolai Vorontsov
Contact:

Re: High CPU load in idle state (#02222715)

Post by NickViz1 »

Hello Dima,

you can find my previous dumps on your ftp (user ftp02137264). Also I've uploaded a new info. What I did:

1. Run Perf Mon and get thread(s) that consume(s) CPU in Veeam.EndPoint.Service.exe. It was not a single thread as I thought, but interleaved 4 threads - an educated guess is that it's a thread pool in your app.
2. I also got thread IDs for those threads. Actually I guess it's not very profitable as you already know where the thread pool in your app is located :-) What's actually interesting for us - what task(s) those threads are executing, but here I can't help much without version with extra logs.
3. I made dumps for Veeam.EndPoint.Service.exe and svchost.exe that consumes CPU.
4. Uploaded to your FTP.

Hope this helps,
Nikolai
Dima P.
Product Manager
Posts: 14396
Liked: 1568 times
Joined: Feb 04, 2013 2:07 pm
Full Name: Dmitry Popov
Location: Prague
Contact:

Re: High CPU load in idle state (#02222715)

Post by Dima P. »

Nikolai,

Thanks, shared the info with the team. I'll update this thread based on investigation results.
NickViz1
Enthusiast
Posts: 41
Liked: 5 times
Joined: Jan 27, 2016 8:50 am
Full Name: Nikolai Vorontsov
Contact:

Re: High CPU load in idle state (#02222715)

Post by NickViz1 »

Hello Dima,

any (good) news? :-)

Nikolai
Dima P.
Product Manager
Posts: 14396
Liked: 1568 times
Joined: Feb 04, 2013 2:07 pm
Full Name: Dmitry Popov
Location: Prague
Contact:

Re: High CPU load in idle state (#02222715)

Post by Dima P. »

Nikolai,

Unfortunately, QA guy responsible for your issue is on a sick leave, so I cant share the latest status.
NickViz1
Enthusiast
Posts: 41
Liked: 5 times
Joined: Jan 27, 2016 8:50 am
Full Name: Nikolai Vorontsov
Contact:

Re: High CPU load in idle state (#02222715)

Post by NickViz1 »

Hello Dima,

how is your QA guy health? Any progress? The problem is still reproducible on my machine...

Nikolai
Dima P.
Product Manager
Posts: 14396
Liked: 1568 times
Joined: Feb 04, 2013 2:07 pm
Full Name: Dmitry Popov
Location: Prague
Contact:

Re: High CPU load in idle state (#02222715)

Post by Dima P. »

He is good now, thanks for asking! We still suspect the Windows Background Intelligent Transfer Service (BITS) as a root cause. In order to check whether it's true or not please disable ‘Automatically check and notify me on available updates’ option (this is the place where VAW leverages BITS) in the Control Panel > About and reboot computer. Let me know how it goes after this option was disabled.
NickViz1
Enthusiast
Posts: 41
Liked: 5 times
Joined: Jan 27, 2016 8:50 am
Full Name: Nikolai Vorontsov
Contact:

Re: High CPU load in idle state (#02222715)

Post by NickViz1 »

Hi Dima,

indeed I can confirm that it depends on the BITS service. When I shut it down - the CPU usage back to 0. Funny enough - it looks that the problem pops up only after the restart - if I stop/start the BITS service - VAW doesn't consume CPU anymore.

The ‘Automatically check and notify me on available updates’ was unchecked. I'll try to check it and reset back explicitly to see if it helps.

Update: enabled ‘Automatically check and notify me on available updates’ and restart - same issue (100% load). Restarted BITS - 0% CPU. So looks like kind of race during the services start..
Dima P.
Product Manager
Posts: 14396
Liked: 1568 times
Joined: Feb 04, 2013 2:07 pm
Full Name: Dmitry Popov
Location: Prague
Contact:

Re: High CPU load in idle state (#02222715)

Post by Dima P. »

Nikolai,

Thanks! Try to run sfc /scannow - theoretically it may fix BITS.
NickViz1
Enthusiast
Posts: 41
Liked: 5 times
Joined: Jan 27, 2016 8:50 am
Full Name: Nikolai Vorontsov
Contact:

Re: High CPU load in idle state (#02222715)

Post by NickViz1 »

C:\windows\system32>sfc /scannow

Beginning system scan. This process will take some time.

Beginning verification phase of system scan.
Verification 100% complete.

Windows Resource Protection did not find any integrity violations.
NickViz1
Enthusiast
Posts: 41
Liked: 5 times
Joined: Jan 27, 2016 8:50 am
Full Name: Nikolai Vorontsov
Contact:

Re: High CPU load in idle state (#02222715)

Post by NickViz1 »

so, as a workaround I've added startup script that restarts BITS after a while. That helps. The bug is still in VAW, but now it's Veeam's responsibility.
modenet
Lurker
Posts: 2
Liked: 2 times
Joined: Sep 19, 2016 5:36 pm
Contact:

Re: High CPU load in idle state (#02222715)

Post by modenet »

same problem on veeam agent 2.2.0.589, any resolution from veeam?
Dima P.
Product Manager
Posts: 14396
Liked: 1568 times
Joined: Feb 04, 2013 2:07 pm
Full Name: Dmitry Popov
Location: Prague
Contact:

Re: High CPU load in idle state (#02222715)

Post by Dima P. »

modenet,

Can you please update your agent to the latest version and check is the issue still persists? We've released v3 several days ago and it contains lots of improvements and fixes, you can get it via our website Thanks!
Dolphin
Enthusiast
Posts: 31
Liked: 4 times
Joined: Mar 06, 2019 10:38 am
Contact:

[MERGED] Veeam Windows Agent CPU on idle

Post by Dolphin »

Hello everybody, running Veeam Agent Windows v.3, seems using cpu when system idle, although no jobs are running. Is this by design?
Offending process are:

Veeam.EndPoint.Service.exe
Veeam.EndPoint.Tray.exe
PTide
Product Manager
Posts: 6408
Liked: 724 times
Joined: May 19, 2015 1:46 pm
Contact:

Re: High CPU load in idle state (#02222715)

Post by PTide »

Hi,
Is this by design?
The answer depends on how high the observed load is. Sure the processes will use some CPU even in idle state, however it the load does not shrink when the machine is under load, or causes performance problems, then you should open a support case as we've seen similar behaviour before. Please post your case ID when you have one.

Thanks!
Post Reply

Who is online

Users browsing this forum: No registered users and 25 guests