Monitoring and reporting for Veeam Data Platform
joespirit88
Enthusiast
Posts: 39
Liked: 5 times
Joined: Jul 04, 2017 12:53 pm
Full Name: Joe Spirit
Contact:

9.5 update 4 High resource usage on Backup servers

Post by joespirit88 »

Hi,

I have upgraded my veeam one to the latest update but have noticed it seems to be causing my backup and replication servers to lock up constantly.

Looking at task manager it seems to be related to Veeam.backup.WMIserver which from what i understand is started by Veeam One.
This process is spiking to nearly the 30% cpu constantly which seems overly unnecessary for a wmi query, the server is a vm with 6 cpus and 24 GB ram so its not under resourced either.

Is anyone else seeing this behaviour after the update?
Shestakov
Veteran
Posts: 7328
Liked: 781 times
Joined: May 21, 2014 11:03 am
Full Name: Nikita Shestakov
Location: Prague
Contact:

Re: 9.5 update 4 High resource usage on Backup servers

Post by Shestakov »

Hi Joe,
You are the first one reporting it. Could you contact Veeam technical support to investigate the behavior?
Thanks!
joespirit88
Enthusiast
Posts: 39
Liked: 5 times
Joined: Jul 04, 2017 12:53 pm
Full Name: Joe Spirit
Contact:

Re: 9.5 update 4 High resource usage on Backup servers

Post by joespirit88 »

Sure I'll post back my case number and any findings
joespirit88
Enthusiast
Posts: 39
Liked: 5 times
Joined: Jul 04, 2017 12:53 pm
Full Name: Joe Spirit
Contact:

Re: 9.5 update 4 High resource usage on Backup servers

Post by joespirit88 »

case number 03389004
Shestakov
Veteran
Posts: 7328
Liked: 781 times
Joined: May 21, 2014 11:03 am
Full Name: Nikita Shestakov
Location: Prague
Contact:

Re: 9.5 update 4 High resource usage on Backup servers

Post by Shestakov »

Thank you for sharing the case number, Joe!
I'll keep track of your case as we expected better performance after upgrade.
Once assistance is needed, let me know.
nickcasa3
Novice
Posts: 7
Liked: never
Joined: Jul 22, 2016 3:01 pm
Full Name: nick casagrande
Contact:

[MERGED] Veeam One server is pounding Veeam B&R Server

Post by nickcasa3 »

I installed VeeamOne trial on a new vm, all is working well, however my B&R server is at 100% cpu and has been for about 12 hours, why would this be happening? I have a small environment, 4 hosts and about 40 vm's being backed up between 2 jobs. Nothing special honestly. I had to shut down veeam one for now to get the b&r server to come back down to a normal 10% cpu level.
nickcasa3
Novice
Posts: 7
Liked: never
Joined: Jul 22, 2016 3:01 pm
Full Name: nick casagrande
Contact:

Re: 9.5 update 4 High resource usage on Backup servers

Post by nickcasa3 »

I have this same issue and just posted a topic about it as well. VeeamOne is pounding my b&r server to the point where I had to shutdown veeam one.
Shestakov
Veteran
Posts: 7328
Liked: 781 times
Joined: May 21, 2014 11:03 am
Full Name: Nikita Shestakov
Location: Prague
Contact:

Re: 9.5 update 4 High resource usage on Backup servers

Post by Shestakov »

Hello Nick and welcome to the community!
Could you also provide a support case number?
Thanks!
nickcasa3
Novice
Posts: 7
Liked: never
Joined: Jul 22, 2016 3:01 pm
Full Name: nick casagrande
Contact:

Re: 9.5 update 4 High resource usage on Backup servers

Post by nickcasa3 »

I have not logged a case yet, I just came here to see if others were having the same issue and apparently they are. I am running 9.5u4 for veeam one and 9.5.0.1922 for b&r
Shestakov
Veteran
Posts: 7328
Liked: 781 times
Joined: May 21, 2014 11:03 am
Full Name: Nikita Shestakov
Location: Prague
Contact:

Re: 9.5 update 4 High resource usage on Backup servers

Post by Shestakov » 1 person likes this post

As was said the behavior is not expected, so it worth contacting the support team. Once we have a fix, support engineers will provide it to you.
joespirit88
Enthusiast
Posts: 39
Liked: 5 times
Joined: Jul 04, 2017 12:53 pm
Full Name: Joe Spirit
Contact:

Re: 9.5 update 4 High resource usage on Backup servers

Post by joespirit88 »

Had a webex this morning with an engineer, they have collected some logs and are speaking to the dev team
Shestakov
Veteran
Posts: 7328
Liked: 781 times
Joined: May 21, 2014 11:03 am
Full Name: Nikita Shestakov
Location: Prague
Contact:

Re: 9.5 update 4 High resource usage on Backup servers

Post by Shestakov »

Thanks for the update,
I've also talked to developers and it turned out that after upgrade to u4 Veeam ONE needs to recollect historical performance data from VBR and since the new collector is much powerful, it may cause CPU usage on VBR side for 15-25%.
How many CPUs do have assigned to VBR server?
joespirit88
Enthusiast
Posts: 39
Liked: 5 times
Joined: Jul 04, 2017 12:53 pm
Full Name: Joe Spirit
Contact:

Re: 9.5 update 4 High resource usage on Backup servers

Post by joespirit88 »

We have 6 vCPU's for the backup server, we were seeing the CPU spiking above 40% for the Veeam WMI Server exe.
If it is collecting historical data how long should we expect it to take, can we skip the historical data import?
It seems a bit overkill that the monitoring server can take out the backup server
Shestakov
Veteran
Posts: 7328
Liked: 781 times
Joined: May 21, 2014 11:03 am
Full Name: Nikita Shestakov
Location: Prague
Contact:

Re: 9.5 update 4 High resource usage on Backup servers

Post by Shestakov »

Duration depends on infrastructure size, but approximately will take 1-3 days, after that the load should be back to normal low values. It cannot be skipped in GUI, but you may ask support if it's an option.
Is the load distributed among all CPUs or it's 100% for one of the units?
joespirit88
Enthusiast
Posts: 39
Liked: 5 times
Joined: Jul 04, 2017 12:53 pm
Full Name: Joe Spirit
Contact:

Re: 9.5 update 4 High resource usage on Backup servers

Post by joespirit88 »

I have currently stopped the veeam one service, however I might re enable it over the weekend whilst I won't be accessing the console and see what the performance is like monday
Shestakov
Veteran
Posts: 7328
Liked: 781 times
Joined: May 21, 2014 11:03 am
Full Name: Nikita Shestakov
Location: Prague
Contact:

Re: 9.5 update 4 High resource usage on Backup servers

Post by Shestakov »

I see. Please let us know how it works once it's on again.
Thanks!
nickcasa3
Novice
Posts: 7
Liked: never
Joined: Jul 22, 2016 3:01 pm
Full Name: nick casagrande
Contact:

Re: 9.5 update 4 High resource usage on Backup servers

Post by nickcasa3 »

I've let my VBR and VeeamOne server run for the past 5 days with 4vcpu each, i have a small 4 host environment with only a few jobs and my VBR server is still spiking quite a bit. Doesn't seem normal honestly. Ticket# is 03399451
Shestakov
Veteran
Posts: 7328
Liked: 781 times
Joined: May 21, 2014 11:03 am
Full Name: Nikita Shestakov
Location: Prague
Contact:

Re: 9.5 update 4 High resource usage on Backup servers

Post by Shestakov »

Thanks for the update, Nick.
Indeed 5 days should be enough to recollect data from 4 host environment.
Will see what support can find out.
joespirit88
Enthusiast
Posts: 39
Liked: 5 times
Joined: Jul 04, 2017 12:53 pm
Full Name: Joe Spirit
Contact:

Re: 9.5 update 4 High resource usage on Backup servers

Post by joespirit88 »

So small update,
They think its an issue if you are running veeam one 9.5 update 4 with backup and replication 9.5 update 3.
I recieved a hotfix from support which i tested today for me it did drop the Veeam WMI Service usage however it seemed to cause the windows WMI Host provider service to spike its' cpu instead.
I'm just organising another webex with support currently
Shestakov
Veteran
Posts: 7328
Liked: 781 times
Joined: May 21, 2014 11:03 am
Full Name: Nikita Shestakov
Location: Prague
Contact:

Re: 9.5 update 4 High resource usage on Backup servers

Post by Shestakov »

Thanks for the update, Joe!
Just curious, why haven't you upgraded backup & replication 9.5 to update 4?
joespirit88
Enthusiast
Posts: 39
Liked: 5 times
Joined: Jul 04, 2017 12:53 pm
Full Name: Joe Spirit
Contact:

Re: 9.5 update 4 High resource usage on Backup servers

Post by joespirit88 »

We wanted to give it a little bit of time to see if any other issues come out before upgrading.

Also we are currently going through parity intitialisation(painfully slow) after adding another 100TB to our backup drive which has had an impact on performance, we are waiting until this has finished so we can clearly see any issues and confirm that the 2 aren't related after an upgrade.
Shestakov
Veteran
Posts: 7328
Liked: 781 times
Joined: May 21, 2014 11:03 am
Full Name: Nikita Shestakov
Location: Prague
Contact:

Re: 9.5 update 4 High resource usage on Backup servers

Post by Shestakov »

I've got it.
We have upgraded Veeam ONE-VBR interaction in update 4, so it will work much better when they are equally updated.
Thanks!
kritzinger
Service Provider
Posts: 7
Liked: 1 time
Joined: Feb 23, 2018 9:33 am
Full Name: Thomas Kritzinger
Contact:

Re: 9.5 update 4 High resource usage on Backup servers

Post by kritzinger »

i have the same Issue.

I needed to Install VEEAM ONE 9.5 U4 because of Hyper-V CSV Data Collection Bug.
VEEAM Cloud Connect ist still at 9.5 U3a. There are too many known issues on update 4 for productive environment.

I had to remove the the Veeam CloudConnect Server from VEEAM ONE and lost all data collected from CloudConnect.
After this CPU utilization went down from 100% to 16% on CloudConnect Server.
Shestakov
Veteran
Posts: 7328
Liked: 781 times
Joined: May 21, 2014 11:03 am
Full Name: Nikita Shestakov
Location: Prague
Contact:

Re: 9.5 update 4 High resource usage on Backup servers

Post by Shestakov »

Thanks for the feedback, Thomas.
As stated above higher CPU usage is expected for the first time after the upgrade.You can contact technical support for the hotfix.
nickcasa3
Novice
Posts: 7
Liked: never
Joined: Jul 22, 2016 3:01 pm
Full Name: nick casagrande
Contact:

Re: 9.5 update 4 High resource usage on Backup servers

Post by nickcasa3 »

There's a hot fix for high cpu?? i have an open ticket on this, please advise.
Shestakov
Veteran
Posts: 7328
Liked: 781 times
Joined: May 21, 2014 11:03 am
Full Name: Nikita Shestakov
Location: Prague
Contact:

Re: 9.5 update 4 High resource usage on Backup servers

Post by Shestakov »

Hi Nick,
Yes, there is a hotfix for the issue. It doesn't resolve the problem completely, but helps decrease CPU usage.
PawelCzopowik
Lurker
Posts: 1
Liked: never
Joined: Sep 08, 2016 7:45 pm
Full Name: Pawel Czopowik
Contact:

Re: 9.5 update 4 High resource usage on Backup servers

Post by PawelCzopowik »

The hotfix doesn't help, the reduction in CPU was probably just form the reboot i performed. A few days later, the server is very slow and sluggish.
Gostev
Chief Product Officer
Posts: 31456
Liked: 6647 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: 9.5 update 4 High resource usage on Backup servers

Post by Gostev »

Do you have Update 4a installed?
pablo.gutierrez
Service Provider
Posts: 4
Liked: 1 time
Joined: May 27, 2015 9:54 pm
Full Name: Pablo Gutierrez
Contact:

Re: 9.5 update 4 High resource usage on Backup servers

Post by pablo.gutierrez »

Same problem:

Veeam ONE 9.5 Update 4 (9.5.4.4566) and Veeam Backup & Replication Update 3a (9.5.0.1922) in the same server, CPU at 100 %; WMI Provider Host (WmiPrvSE.exe); shutdown Veeam ONE Monitor Server decrease CPU usage to 10 - 20 %. Any fix.

Is neccesary update B & R to Update 4a?
wishr
Veteran
Posts: 3077
Liked: 453 times
Joined: Aug 07, 2018 3:11 pm
Full Name: Fedor Maslov
Contact:

Re: 9.5 update 4 High resource usage on Backup servers

Post by wishr »

Hi Pablo,

Yes, please update B&R instance to U4a as recommended in our Top Issues Tracker for Veeam ONE (point #5) and reach out to our support team directly if the situation does not change.

Regards,
Fedor
Post Reply

Who is online

Users browsing this forum: No registered users and 9 guests