V6 high cpu

[FrancWest]

Hi,

Case #05324183. After installing V6 in our lab on Windows Server 2022, we are experiencing continuous high cpu usage by the Veeam.Archiver.Service. Cpu usage for this process is always around 50% even when there are no jobs configured or running. Restarting the server doesn’t help. As soon as you stop and start the Veeam.Archiver.Service service, cpu usage for this process goes to 50% and stays there.

Anyone else experiencing this?

Regards,
Franc

[Mike Resseler]

Franc,

We haven't seen this in our labs. Did you do an upgrade in your lab or a fresh install?

[FrancWest]

Hi,

It is a fresh install.

Regards,
Franc

[Polina]

Franc,

Kindly, create a support case and share its ID here. I'll pass it to our RnD for investigation.
Thanks!

[FrancWest]

Hi, see first sentence in my opening post.

[Polina]

Wooops, sorry, missed it )

[MILJW002]

I'm having the same experience, however mine was an in-place upgrade. Rolling back to V5 until this can be resolved.

[Gr1pen]

Any updates?

I am seeing exact same issue after upgrading from v5 to v6.

[kevlahau]

FYI not seeing this running W2K19 standard with in place upgrade

[MILJW002]

Hi,

I thought I should perhaps share a little more of my experience as it may be more helpful to others.

My current setup is V5 (all the latest patches), running on Windows Server 2022 (Core). I'm using the Community Education as this is a home lab, and only has 5 users to back up.

The first attempt at an in place upgrade resulted in a failure. After some investigation and reading the forums here, I realised it was a REST API service certificate issue. The service wouldn't start (as part of the upgrade) and it said it was because the certificate couldn't be found in the local cert store. So, while still running V5 I renewed the certificate (issued by my local CA). The thumbprint then matched the details in the config file, and it was in the local cert store.

The upgrade then worked successfully, however... the archiver service was running 100% CPU. The proxy service etc were all idle. No jobs were running yet, and I hand't triggered any repository upgrades etc. After looking at the various service log files, I couldn't see anything obviously wrong. The logs didn't (at least to me) jump out as something.

At this stage I'd spent a few hours, so wanted to get back to a known working stage. I restored my previous backup, and things are mostly working.

The only issue I've got at the moment, is the AWS hosted S3 backup repositories are showing as invalid. I assume because there was a change as part of the upgrade to V6. I've temporarily disabled those jobs and hoping when a fix is realised for V6 I can then upgrade and get things back in sync.

James

[Mike Resseler]

FYI, our engineers are investigating the case from OP. However, (unless I am wrong Franc, then let me know) it seems that you are not running the minimum requirements. In our testing with the minimum requirements on WS2022 we see a max of 12% CPU usage.

That said, I requested to do some more digging.

Also, if you have already enabled the self-service portal, then keep in mind you will need more requirements

[lharmer]

Hi, we are also seeing relatively high CPU on a community edition lab upgrade. WS2019, 16GB memory but only 4 core CPU (it's actually just a re-purposed desktop for lab purposes).
Worked fine on v5 but since v6 Veeam.Archiver.Service idles at 40% CPU.

It does seem a little unusual as no jobs are running, no GUI is open so we would expect very little CPU on that process? But I see the minimum requirements have increased to 16GB/8cores in this version so guessing it just idles a bit hotter nowadays.

In production we have quite a large environment backed up to s3 as well, so would be good to know if this is going to increase load on those servers prior to upgrading!

[MILJW002]

I can totally respect and understand with the new capabilities (e.g. self service portal) that the minimum requirements would increase.

What doesn’t make sense to me is why the Veeam.Archiver.Service process would be using 100% of the CPU when idle. No jobs running, not upgrading the repositories yet, haven’t turned on the additional functions (e.g, self service). This is just after installing the update before even connecting the console to the service.

It’s looking like some kind of bug, to go from V5 idling at 1% to 2% to idling at 100%.

[Polina]

Hi James,

As Mike said, RnD is now investigating this case. We'll post an update when there's any news.

[MILJW002]

Hi Polina,

Sorry if it came across as tone from me, it wasn't my intent! The risk of posting to a forum from an iPad when half asleep on the couch on a long weekend.

I've just run up a clean/vanilla VM on my MacBook Pro with VMware Fusion. It's Server 2022 (no updates, but a base install), and I've just performed a base install of VBO v6 (zero config).

It appears to have a similar behaviour. It's sitting around 50% CPU, and that's before I've even opened the console. It looks like Veeam.Archiver.Service is using 100% of one of the cores (I configured the VM with 2 virtual cores). This is vanilla, no configured organisation, no jobs etc.

If I open the console, that console seems reasonably responsive. I'm just looked in the various log files, and they look really similar the the in-place upgrade I tried.

Again, totally understand RnD are investigating, I just wanted to double check if it was something with my install, but a clean install (no config done yet) and it seems to be really similar.

Looking forward to hearing the good news from RnD.

James

[FrancWest]

FYI, our engineers are investigating the case from OP. However, (unless I am wrong Franc, then let me know) it seems that you are not running the minimum requirements. In our testing with the minimum requirements on WS2022 we see a max of 12% CPU usage.

Correct, since my testing was in a lab machine, it doesn’t meet the requirements. However, if an application is idle, even with no job configured at all, it shouldn’t use any cpu. With v5 this wasn’t an issue, with v5 it is 0% idle.

When temporarily adding the correct amount of resources I also see cpu usage going down to 12% at idle, but still it shouldn’t be that high when doing nothing. You state you also see it at 12%, is that also when being idle?

[Mike Resseler]

To be honest, not sure if that 12% comes from your environment or one of ours. Would need to verify that. I do saw that they requested memory dumps and hopefully that will learn us something. Obviously each version comes with more demands as we add new functionality, but I still would like to learn what is causing it in your environment and why there is such a difference between v5 and 6 (especially on w2022)

[Polina]

James, no worries )
I just didn't have any better response at this point rather than saying that our engineers are looking into it. I'll forward this info to the team as well, and update you with the news.

[MILJW002]

Hi Polina,

I have a theory, which I've just tested.

With 1 x vCPU it sits at 100% load idling.
With 2 x vCPU it sits at 50% load idling.
With 4 x vCPU it sis at 25% load idling.
With 8 x vCPU it sits at 12.49% load idling.




It's only a basic test, but it looks like the Veeam.Archiver.Service is consistently using 100% of 1 CPU core.

I understand the minimum requirements have increased, but this may be masking the actual issue. It is starting to look to me like there is a thread within Veeam.Archiver.Service that consumes 100% of 1 core, no matter what is actually happening.

James

[Polina]

Configurations with less than 4 cores were not supported even before v6, but still, it doesn't make the constant consumption in the idle state clear.
I'll forward your findings to the RnD team.

Thanks!

[Regnor]

I can confirm the same behaviour for this service, both for my lab which has been upgraded and a fresh installation.

[Polina]

A small update: the hotfix is ready and is now in final testing. Soon it should be available via support.

[FrancWest]

Great, then I’ll be hearing from support soon to test it out.

[kavaa]

A small update: the hotfix is ready and is now in final testing. Soon it should be available via support.

Can you send it to us as well?
Still running V5

Veeam Archiver Shell with 12 Cores is not even using 1% of the CPU at idle...

Starting to test on a VM with same specs, and indeed also had more CPU.

[stevetNOVUS]

I can confirm the same issue, physical server with 12 cores, running server 2019. This was a clean install.
The Veeam Archiver Service is running at a constant 8% CPU (1 core at 100%)

@Polina : Do you know when the patch will be made public?

[wishr]

Hi Guys,

As usual, to stay up to date on the hotfix availability and for actually getting it, we recommend contacting our customer support team directly by opening a case.

Thanks

[KOPFteam]

Hi wishr,

same here. 25% on a 4 vCPU VM. I can't open a support case, because I'm using the community edition of VBO365. How can I get the patch?

Thanks,
Florian

[Mildur]

Hi Florian

If I remember correctly, using hotfixes requires an active and paid maintenance on the license. I know that this is true for the vbr product.
If it‘s the same for VBO/VBM365, you must wait for an public CU Patch.

[wishr]

Hi Florian,

I'm afraid hotfixes are only available for customers with an active maintenance contract. In such situation I would suggest re-deploying the previous version or wait for the next public update.

Thanks

[FrancWest]

I just received the private fix from support and everything is fine now. CPU usage for the Veeam.Archiver.Service.exe is down to 0% when idle.