Guest Catalog Service is going nuts post 12.1 Upgrade

[davemann]

Hello -

I was wondering if anyone else has seen similar behavior's after upgrading to v12.1...

The other night during our backup window, I noticed that my VBR console was very sluggish, and that jobs were not progressing as quickly as normal. I then noticed in vCenter that my VBR server was 100% pinned on CPU, consuming ~26Ghz worth of CPU (ouch!). I logged into the system to observe that the Guest Catalog service was causing 100% CPU utilization, and showed no signs of stopping.

I restarted the service, and things calmed down - for a bit, but then it started all over again. For now I seem to have to keep this service stopped in order for things to process correctly, and I don't know the best way to deal with this, so I'll put in a support ticket.

Just checking to see if this is known?

[Gostev]

Hello, it is likely a one-time analysis of all existing guest indexes by the malware detection functionality, but I will let the corresponding PM confirm and share any additional details.

[davemann]

Thanks! I've submitted a case with logs from the VBR server.

For now, I've disabled the suspicious activity detection functionality and I've started the service back up, and hopefully I can get through tonight's window fine.

[Gostev]

So apparently I was wrong about the "all existing" part, only newly created indexes are analyzed. As such this is unexpected and needs to be investigated.

[Dima P.]

Hello Dave,

If by any chance you open a support case, please, do not forget to share the case ID with us! Thank you!

[lmoglie]

Hello,
I have exactly the same problem.
I solved it temporarily, setting the CPU affinity for the process, but I'll for sure open a case to investigate.

[Mildur]

Hi Lorenzo

We have a hotfix ready for this issue. Please reference to Bug ID 623177 in your case.
veeam-backup-replication-f2/top-issues- ... 91449.html

#4: High CPU usage by CatalogDataService
Scope: Rare condition that depends on a number of parameters, more likely to happen in larger environments.
Symptoms: CatalogDataService consumes all the CPU cycles it can get.
Cause: Resource scheduling bug in the index scan engine of the malware detection functionality.
Status: Contact support for a hotfix referencing Bug ID 623177.

Best,
Fabian

[lmoglie]

Thanks a lot Fabian

[Nodnarb]

Hi all. I'm also having an issue with high CPU usage on the CatalogDataService process if "Enable file system activity analysis" is checked. The process continues to use high CPU even after jobs have completed.

I contacted Support (case 07081739) and applied the hotfix, but I am still having the high CPU usage issue. I'll update here once we've resolved it.

[Gaba2001]

I have same problem.

I opened a support case and got a private fix.
veeam_backup_12.1.0.2131_PrivateFix_TF623294.exe

I installed that.

Now the service appears is in normal condition. No high CPU and memory use.

[Dima P.]

Hello Gabor,

Thank you for sharing the updates and glad that your issue was addressed by the hotfix!

[Nodnarb]

Hi all. I'm also having an issue with high CPU usage on the CatalogDataService process if "Enable file system activity analysis" is checked. The process continues to use high CPU even after jobs have completed.

I contacted Support (case 07081739) and applied the hotfix, but I am still having the high CPU usage issue. I'll update here once we've resolved it.

This never quite got resolved...We were given a registry key to reduce the number of threads the process uses, but usage still remained pretty constant. I eventually turned the feature off. I'll try it again in future updates.

[Gostev]

This issue was fixed back in January, it is specifically called out in 12.1.1 release notes.

[popjls]

This issue was fixed back in January, it is specifically called out in 12.1.1 release notes.

I found this thread because I'm having this exact issue and I'm on 12.1.1.56. I'll submit a support case as it's only been an issue since upgrading to 12.1 for me.