Monitoring and reporting for Veeam Backup & Replication, VMware vSphere and Microsoft Hyper-V
joespirit88
Enthusiast
Posts: 36
Liked: 5 times
Joined: Jul 04, 2017 12:53 pm
Full Name: Joe Spirit
Contact:

9.5 update 4 High resource usage on Backup servers

Post by joespirit88 » Jan 30, 2019 11:02 am

Hi,

I have upgraded my veeam one to the latest update but have noticed it seems to be causing my backup and replication servers to lock up constantly.

Looking at task manager it seems to be related to Veeam.backup.WMIserver which from what i understand is started by Veeam One.
This process is spiking to nearly the 30% cpu constantly which seems overly unnecessary for a wmi query, the server is a vm with 6 cpus and 24 GB ram so its not under resourced either.

Is anyone else seeing this behaviour after the update?

Shestakov
Veeam Software
Posts: 6907
Liked: 700 times
Joined: May 21, 2014 11:03 am
Full Name: Nikita Shestakov
Location: Prague
Contact:

Re: 9.5 update 4 High resource usage on Backup servers

Post by Shestakov » Jan 30, 2019 12:15 pm

Hi Joe,
You are the first one reporting it. Could you contact Veeam technical support to investigate the behavior?
Thanks!

joespirit88
Enthusiast
Posts: 36
Liked: 5 times
Joined: Jul 04, 2017 12:53 pm
Full Name: Joe Spirit
Contact:

Re: 9.5 update 4 High resource usage on Backup servers

Post by joespirit88 » Jan 30, 2019 12:29 pm

Sure I'll post back my case number and any findings

joespirit88
Enthusiast
Posts: 36
Liked: 5 times
Joined: Jul 04, 2017 12:53 pm
Full Name: Joe Spirit
Contact:

Re: 9.5 update 4 High resource usage on Backup servers

Post by joespirit88 » Jan 30, 2019 12:41 pm

case number 03389004

Shestakov
Veeam Software
Posts: 6907
Liked: 700 times
Joined: May 21, 2014 11:03 am
Full Name: Nikita Shestakov
Location: Prague
Contact:

Re: 9.5 update 4 High resource usage on Backup servers

Post by Shestakov » Jan 30, 2019 1:50 pm

Thank you for sharing the case number, Joe!
I'll keep track of your case as we expected better performance after upgrade.
Once assistance is needed, let me know.

nickcasa3
Novice
Posts: 7
Liked: never
Joined: Jul 22, 2016 3:01 pm
Full Name: nick casagrande
Contact:

[MERGED] Veeam One server is pounding Veeam B&R Server

Post by nickcasa3 » Jan 30, 2019 2:17 pm

I installed VeeamOne trial on a new vm, all is working well, however my B&R server is at 100% cpu and has been for about 12 hours, why would this be happening? I have a small environment, 4 hosts and about 40 vm's being backed up between 2 jobs. Nothing special honestly. I had to shut down veeam one for now to get the b&r server to come back down to a normal 10% cpu level.

nickcasa3
Novice
Posts: 7
Liked: never
Joined: Jul 22, 2016 3:01 pm
Full Name: nick casagrande
Contact:

Re: 9.5 update 4 High resource usage on Backup servers

Post by nickcasa3 » Jan 30, 2019 2:18 pm

I have this same issue and just posted a topic about it as well. VeeamOne is pounding my b&r server to the point where I had to shutdown veeam one.

Shestakov
Veeam Software
Posts: 6907
Liked: 700 times
Joined: May 21, 2014 11:03 am
Full Name: Nikita Shestakov
Location: Prague
Contact:

Re: 9.5 update 4 High resource usage on Backup servers

Post by Shestakov » Jan 30, 2019 2:22 pm

Hello Nick and welcome to the community!
Could you also provide a support case number?
Thanks!

nickcasa3
Novice
Posts: 7
Liked: never
Joined: Jul 22, 2016 3:01 pm
Full Name: nick casagrande
Contact:

Re: 9.5 update 4 High resource usage on Backup servers

Post by nickcasa3 » Jan 30, 2019 2:26 pm

I have not logged a case yet, I just came here to see if others were having the same issue and apparently they are. I am running 9.5u4 for veeam one and 9.5.0.1922 for b&r

Shestakov
Veeam Software
Posts: 6907
Liked: 700 times
Joined: May 21, 2014 11:03 am
Full Name: Nikita Shestakov
Location: Prague
Contact:

Re: 9.5 update 4 High resource usage on Backup servers

Post by Shestakov » Jan 30, 2019 2:28 pm 1 person likes this post

As was said the behavior is not expected, so it worth contacting the support team. Once we have a fix, support engineers will provide it to you.

joespirit88
Enthusiast
Posts: 36
Liked: 5 times
Joined: Jul 04, 2017 12:53 pm
Full Name: Joe Spirit
Contact:

Re: 9.5 update 4 High resource usage on Backup servers

Post by joespirit88 » Jan 31, 2019 2:07 pm

Had a webex this morning with an engineer, they have collected some logs and are speaking to the dev team

Shestakov
Veeam Software
Posts: 6907
Liked: 700 times
Joined: May 21, 2014 11:03 am
Full Name: Nikita Shestakov
Location: Prague
Contact:

Re: 9.5 update 4 High resource usage on Backup servers

Post by Shestakov » Jan 31, 2019 2:33 pm

Thanks for the update,
I've also talked to developers and it turned out that after upgrade to u4 Veeam ONE needs to recollect historical performance data from VBR and since the new collector is much powerful, it may cause CPU usage on VBR side for 15-25%.
How many CPUs do have assigned to VBR server?

joespirit88
Enthusiast
Posts: 36
Liked: 5 times
Joined: Jul 04, 2017 12:53 pm
Full Name: Joe Spirit
Contact:

Re: 9.5 update 4 High resource usage on Backup servers

Post by joespirit88 » Jan 31, 2019 3:00 pm

We have 6 vCPU's for the backup server, we were seeing the CPU spiking above 40% for the Veeam WMI Server exe.
If it is collecting historical data how long should we expect it to take, can we skip the historical data import?
It seems a bit overkill that the monitoring server can take out the backup server

Shestakov
Veeam Software
Posts: 6907
Liked: 700 times
Joined: May 21, 2014 11:03 am
Full Name: Nikita Shestakov
Location: Prague
Contact:

Re: 9.5 update 4 High resource usage on Backup servers

Post by Shestakov » Jan 31, 2019 3:05 pm

Duration depends on infrastructure size, but approximately will take 1-3 days, after that the load should be back to normal low values. It cannot be skipped in GUI, but you may ask support if it's an option.
Is the load distributed among all CPUs or it's 100% for one of the units?

joespirit88
Enthusiast
Posts: 36
Liked: 5 times
Joined: Jul 04, 2017 12:53 pm
Full Name: Joe Spirit
Contact:

Re: 9.5 update 4 High resource usage on Backup servers

Post by joespirit88 » Jan 31, 2019 3:48 pm

I have currently stopped the veeam one service, however I might re enable it over the weekend whilst I won't be accessing the console and see what the performance is like monday

Shestakov
Veeam Software
Posts: 6907
Liked: 700 times
Joined: May 21, 2014 11:03 am
Full Name: Nikita Shestakov
Location: Prague
Contact:

Re: 9.5 update 4 High resource usage on Backup servers

Post by Shestakov » Feb 01, 2019 11:39 am

I see. Please let us know how it works once it's on again.
Thanks!

nickcasa3
Novice
Posts: 7
Liked: never
Joined: Jul 22, 2016 3:01 pm
Full Name: nick casagrande
Contact:

Re: 9.5 update 4 High resource usage on Backup servers

Post by nickcasa3 » Feb 05, 2019 7:30 pm

I've let my VBR and VeeamOne server run for the past 5 days with 4vcpu each, i have a small 4 host environment with only a few jobs and my VBR server is still spiking quite a bit. Doesn't seem normal honestly. Ticket# is 03399451

Shestakov
Veeam Software
Posts: 6907
Liked: 700 times
Joined: May 21, 2014 11:03 am
Full Name: Nikita Shestakov
Location: Prague
Contact:

Re: 9.5 update 4 High resource usage on Backup servers

Post by Shestakov » Feb 06, 2019 12:09 pm

Thanks for the update, Nick.
Indeed 5 days should be enough to recollect data from 4 host environment.
Will see what support can find out.

joespirit88
Enthusiast
Posts: 36
Liked: 5 times
Joined: Jul 04, 2017 12:53 pm
Full Name: Joe Spirit
Contact:

Re: 9.5 update 4 High resource usage on Backup servers

Post by joespirit88 » Feb 13, 2019 3:27 pm

So small update,
They think its an issue if you are running veeam one 9.5 update 4 with backup and replication 9.5 update 3.
I recieved a hotfix from support which i tested today for me it did drop the Veeam WMI Service usage however it seemed to cause the windows WMI Host provider service to spike its' cpu instead.
I'm just organising another webex with support currently

Shestakov
Veeam Software
Posts: 6907
Liked: 700 times
Joined: May 21, 2014 11:03 am
Full Name: Nikita Shestakov
Location: Prague
Contact:

Re: 9.5 update 4 High resource usage on Backup servers

Post by Shestakov » Feb 13, 2019 3:36 pm

Thanks for the update, Joe!
Just curious, why haven't you upgraded backup & replication 9.5 to update 4?

joespirit88
Enthusiast
Posts: 36
Liked: 5 times
Joined: Jul 04, 2017 12:53 pm
Full Name: Joe Spirit
Contact:

Re: 9.5 update 4 High resource usage on Backup servers

Post by joespirit88 » Feb 13, 2019 4:40 pm

We wanted to give it a little bit of time to see if any other issues come out before upgrading.

Also we are currently going through parity intitialisation(painfully slow) after adding another 100TB to our backup drive which has had an impact on performance, we are waiting until this has finished so we can clearly see any issues and confirm that the 2 aren't related after an upgrade.

Shestakov
Veeam Software
Posts: 6907
Liked: 700 times
Joined: May 21, 2014 11:03 am
Full Name: Nikita Shestakov
Location: Prague
Contact:

Re: 9.5 update 4 High resource usage on Backup servers

Post by Shestakov » Feb 15, 2019 3:39 pm

I've got it.
We have upgraded Veeam ONE-VBR interaction in update 4, so it will work much better when they are equally updated.
Thanks!

kritzinger
Service Provider
Posts: 7
Liked: 1 time
Joined: Feb 23, 2018 9:33 am
Full Name: Thomas Kritzinger
Contact:

Re: 9.5 update 4 High resource usage on Backup servers

Post by kritzinger » Feb 19, 2019 11:01 am

i have the same Issue.

I needed to Install VEEAM ONE 9.5 U4 because of Hyper-V CSV Data Collection Bug.
VEEAM Cloud Connect ist still at 9.5 U3a. There are too many known issues on update 4 for productive environment.

I had to remove the the Veeam CloudConnect Server from VEEAM ONE and lost all data collected from CloudConnect.
After this CPU utilization went down from 100% to 16% on CloudConnect Server.

Shestakov
Veeam Software
Posts: 6907
Liked: 700 times
Joined: May 21, 2014 11:03 am
Full Name: Nikita Shestakov
Location: Prague
Contact:

Re: 9.5 update 4 High resource usage on Backup servers

Post by Shestakov » Feb 19, 2019 11:38 am

Thanks for the feedback, Thomas.
As stated above higher CPU usage is expected for the first time after the upgrade.You can contact technical support for the hotfix.

nickcasa3
Novice
Posts: 7
Liked: never
Joined: Jul 22, 2016 3:01 pm
Full Name: nick casagrande
Contact:

Re: 9.5 update 4 High resource usage on Backup servers

Post by nickcasa3 » Feb 25, 2019 1:12 am

There's a hot fix for high cpu?? i have an open ticket on this, please advise.

Shestakov
Veeam Software
Posts: 6907
Liked: 700 times
Joined: May 21, 2014 11:03 am
Full Name: Nikita Shestakov
Location: Prague
Contact:

Re: 9.5 update 4 High resource usage on Backup servers

Post by Shestakov » Feb 25, 2019 1:36 pm

Hi Nick,
Yes, there is a hotfix for the issue. It doesn't resolve the problem completely, but helps decrease CPU usage.

PawelCzopowik
Lurker
Posts: 1
Liked: never
Joined: Sep 08, 2016 7:45 pm
Full Name: Pawel Czopowik
Contact:

Re: 9.5 update 4 High resource usage on Backup servers

Post by PawelCzopowik » Apr 02, 2019 6:40 pm

The hotfix doesn't help, the reduction in CPU was probably just form the reboot i performed. A few days later, the server is very slow and sluggish.

Gostev
SVP, Product Management
Posts: 24612
Liked: 3463 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: 9.5 update 4 High resource usage on Backup servers

Post by Gostev » Apr 02, 2019 6:58 pm

Do you have Update 4a installed?

pablo.gutierrez
Lurker
Posts: 2
Liked: 1 time
Joined: May 27, 2015 9:54 pm
Full Name: Pablo Gutierrez
Contact:

Re: 9.5 update 4 High resource usage on Backup servers

Post by pablo.gutierrez » Apr 11, 2019 7:38 pm

Same problem:

Veeam ONE 9.5 Update 4 (9.5.4.4566) and Veeam Backup & Replication Update 3a (9.5.0.1922) in the same server, CPU at 100 %; WMI Provider Host (WmiPrvSE.exe); shutdown Veeam ONE Monitor Server decrease CPU usage to 10 - 20 %. Any fix.

Is neccesary update B & R to Update 4a?

wishr
Veeam Software
Posts: 1076
Liked: 109 times
Joined: Aug 07, 2018 3:11 pm
Full Name: Fedor Maslov
Contact:

Re: 9.5 update 4 High resource usage on Backup servers

Post by wishr » Apr 12, 2019 10:02 am

Hi Pablo,

Yes, please update B&R instance to U4a as recommended in our Top Issues Tracker for Veeam ONE (point #5) and reach out to our support team directly if the situation does not change.

Regards,
Fedor

Post Reply

Who is online

Users browsing this forum: No registered users and 2 guests