9.5 update 4 High resource usage on Backup servers

[joespirit88]

Hi,

I have upgraded my veeam one to the latest update but have noticed it seems to be causing my backup and replication servers to lock up constantly.

Looking at task manager it seems to be related to Veeam.backup.WMIserver which from what i understand is started by Veeam One.
This process is spiking to nearly the 30% cpu constantly which seems overly unnecessary for a wmi query, the server is a vm with 6 cpus and 24 GB ram so its not under resourced either.

Is anyone else seeing this behaviour after the update?

[Shestakov]

Hi Joe,
You are the first one reporting it. Could you contact Veeam technical support to investigate the behavior?
Thanks!

[joespirit88]

Sure I'll post back my case number and any findings

[joespirit88]

case number 03389004

[Shestakov]

Thank you for sharing the case number, Joe!
I'll keep track of your case as we expected better performance after upgrade.
Once assistance is needed, let me know.

[nickcasa3]

I installed VeeamOne trial on a new vm, all is working well, however my B&R server is at 100% cpu and has been for about 12 hours, why would this be happening? I have a small environment, 4 hosts and about 40 vm's being backed up between 2 jobs. Nothing special honestly. I had to shut down veeam one for now to get the b&r server to come back down to a normal 10% cpu level.

[nickcasa3]

I have this same issue and just posted a topic about it as well. VeeamOne is pounding my b&r server to the point where I had to shutdown veeam one.

[Shestakov]

Hello Nick and welcome to the community!
Could you also provide a support case number?
Thanks!

[nickcasa3]

I have not logged a case yet, I just came here to see if others were having the same issue and apparently they are. I am running 9.5u4 for veeam one and 9.5.0.1922 for b&r

[Shestakov]

As was said the behavior is not expected, so it worth contacting the support team. Once we have a fix, support engineers will provide it to you.

[joespirit88]

Had a webex this morning with an engineer, they have collected some logs and are speaking to the dev team

[Shestakov]

Thanks for the update,
I've also talked to developers and it turned out that after upgrade to u4 Veeam ONE needs to recollect historical performance data from VBR and since the new collector is much powerful, it may cause CPU usage on VBR side for 15-25%.
How many CPUs do have assigned to VBR server?

[joespirit88]

We have 6 vCPU's for the backup server, we were seeing the CPU spiking above 40% for the Veeam WMI Server exe.
If it is collecting historical data how long should we expect it to take, can we skip the historical data import?
It seems a bit overkill that the monitoring server can take out the backup server

[Shestakov]

Duration depends on infrastructure size, but approximately will take 1-3 days, after that the load should be back to normal low values. It cannot be skipped in GUI, but you may ask support if it's an option.
Is the load distributed among all CPUs or it's 100% for one of the units?

[joespirit88]

I have currently stopped the veeam one service, however I might re enable it over the weekend whilst I won't be accessing the console and see what the performance is like monday

[Shestakov]

I see. Please let us know how it works once it's on again.
Thanks!

[nickcasa3]

I've let my VBR and VeeamOne server run for the past 5 days with 4vcpu each, i have a small 4 host environment with only a few jobs and my VBR server is still spiking quite a bit. Doesn't seem normal honestly. Ticket# is 03399451

[Shestakov]

Thanks for the update, Nick.
Indeed 5 days should be enough to recollect data from 4 host environment.
Will see what support can find out.

[joespirit88]

So small update,
They think its an issue if you are running veeam one 9.5 update 4 with backup and replication 9.5 update 3.
I recieved a hotfix from support which i tested today for me it did drop the Veeam WMI Service usage however it seemed to cause the windows WMI Host provider service to spike its' cpu instead.
I'm just organising another webex with support currently

[Shestakov]

Thanks for the update, Joe!
Just curious, why haven't you upgraded backup & replication 9.5 to update 4?

[joespirit88]

We wanted to give it a little bit of time to see if any other issues come out before upgrading.

Also we are currently going through parity intitialisation(painfully slow) after adding another 100TB to our backup drive which has had an impact on performance, we are waiting until this has finished so we can clearly see any issues and confirm that the 2 aren't related after an upgrade.

[Shestakov]

I've got it.
We have upgraded Veeam ONE-VBR interaction in update 4, so it will work much better when they are equally updated.
Thanks!

[kritzinger]

i have the same Issue.

I needed to Install VEEAM ONE 9.5 U4 because of Hyper-V CSV Data Collection Bug.
VEEAM Cloud Connect ist still at 9.5 U3a. There are too many known issues on update 4 for productive environment.

I had to remove the the Veeam CloudConnect Server from VEEAM ONE and lost all data collected from CloudConnect.
After this CPU utilization went down from 100% to 16% on CloudConnect Server.

[Shestakov]

Thanks for the feedback, Thomas.
As stated above higher CPU usage is expected for the first time after the upgrade.You can contact technical support for the hotfix.

[nickcasa3]

There's a hot fix for high cpu?? i have an open ticket on this, please advise.

[Shestakov]

Hi Nick,
Yes, there is a hotfix for the issue. It doesn't resolve the problem completely, but helps decrease CPU usage.

[PawelCzopowik]

The hotfix doesn't help, the reduction in CPU was probably just form the reboot i performed. A few days later, the server is very slow and sluggish.

[Gostev]

Do you have Update 4a installed?

[pablo.gutierrez]

Same problem:

Veeam ONE 9.5 Update 4 (9.5.4.4566) and Veeam Backup & Replication Update 3a (9.5.0.1922) in the same server, CPU at 100 %; WMI Provider Host (WmiPrvSE.exe); shutdown Veeam ONE Monitor Server decrease CPU usage to 10 - 20 %. Any fix.

Is neccesary update B & R to Update 4a?

[wishr]

Hi Pablo,

Yes, please update B&R instance to U4a as recommended in our Top Issues Tracker for Veeam ONE (point #5) and reach out to our support team directly if the situation does not change.

Regards,
Fedor