Guest Catalog Service is going nuts post 12.1 Upgrade

[davemann]

Hello -

I was wondering if anyone else has seen similar behavior's after upgrading to v12.1...

The other night during our backup window, I noticed that my VBR console was very sluggish, and that jobs were not progressing as quickly as normal. I then noticed in vCenter that my VBR server was 100% pinned on CPU, consuming ~26Ghz worth of CPU (ouch!). I logged into the system to observe that the Guest Catalog service was causing 100% CPU utilization, and showed no signs of stopping.

I restarted the service, and things calmed down - for a bit, but then it started all over again. For now I seem to have to keep this service stopped in order for things to process correctly, and I don't know the best way to deal with this, so I'll put in a support ticket.

Just checking to see if this is known?

[Gostev]

Hello, it is likely a one-time analysis of all existing guest indexes by the malware detection functionality, but I will let the corresponding PM confirm and share any additional details.

[davemann]

Thanks! I've submitted a case with logs from the VBR server.

For now, I've disabled the suspicious activity detection functionality and I've started the service back up, and hopefully I can get through tonight's window fine.

[Gostev]

So apparently I was wrong about the "all existing" part, only newly created indexes are analyzed. As such this is unexpected and needs to be investigated.

[Dima P.]

Hello Dave,

If by any chance you open a support case, please, do not forget to share the case ID with us! Thank you!

[lmoglie]

Hello,
I have exactly the same problem.
I solved it temporarily, setting the CPU affinity for the process, but I'll for sure open a case to investigate.

[Mildur]

Hi Lorenzo

We have a hotfix ready for this issue. Please reference to Bug ID 623177 in your case.
veeam-backup-replication-f2/top-issues- ... 91449.html

#4: High CPU usage by CatalogDataService
Scope: Rare condition that depends on a number of parameters, more likely to happen in larger environments.
Symptoms: CatalogDataService consumes all the CPU cycles it can get.
Cause: Resource scheduling bug in the index scan engine of the malware detection functionality.
Status: Contact support for a hotfix referencing Bug ID 623177.

Best,
Fabian

[lmoglie]

Thanks a lot Fabian

[Nodnarb]

Hi all. I'm also having an issue with high CPU usage on the CatalogDataService process if "Enable file system activity analysis" is checked. The process continues to use high CPU even after jobs have completed.

I contacted Support (case 07081739) and applied the hotfix, but I am still having the high CPU usage issue. I'll update here once we've resolved it.

[Gaba2001]

I have same problem.

I opened a support case and got a private fix.
veeam_backup_12.1.0.2131_PrivateFix_TF623294.exe

I installed that.

Now the service appears is in normal condition. No high CPU and memory use.

[Dima P.]

Hello Gabor,

Thank you for sharing the updates and glad that your issue was addressed by the hotfix!

[Nodnarb]

Hi all. I'm also having an issue with high CPU usage on the CatalogDataService process if "Enable file system activity analysis" is checked. The process continues to use high CPU even after jobs have completed.

I contacted Support (case 07081739) and applied the hotfix, but I am still having the high CPU usage issue. I'll update here once we've resolved it.

This never quite got resolved...We were given a registry key to reduce the number of threads the process uses, but usage still remained pretty constant. I eventually turned the feature off. I'll try it again in future updates.

[Gostev]

This issue was fixed back in January, it is specifically called out in 12.1.1 release notes.

[popjls]

This issue was fixed back in January, it is specifically called out in 12.1.1 release notes.

I found this thread because I'm having this exact issue and I'm on 12.1.1.56. I'll submit a support case as it's only been an issue since upgrading to 12.1 for me.

[LU86]

Encountered the same issues after a routine Windows Update maintanance window.
VBR version on this server is 12.1.1.56 since 2/13/2024 and never had the CPU usage issue before.

I noticed that the following Windows Update was installed yesterday:
KB5036343 - Description of the security update for SQL Server 2022 CU12: April 9, 2024

Maybe that triggered some kind of "catalog refresh" that took several hours and un-zipped the content of the archived data of the VM fileserver that has file indexing active.
The un-zipping of those files (drive:\VBRCatalog\Index\Machines\...) almost filled up the entire Disk because it un-zipped those files back to: 03/19/2024.


I'll definately sumbit a support case on that matter

[Mildur]

Hi Lukas

What's your case number?

Best,
Fabian

[LU86]

Hi Fabian,

Case #07263318


Regards,
Lukas.

[iDeNt_5]

Hi all,

Are there any news about this issue?
It seems like I'm facing the same with a Customer with the VBR 12.1.2.172 version.

Many thanks.

[Dima P.]

Hello Matteo,

Do you have a case ID to share? Thank you!

[iDeNt_5]

Hi Dima,

Sure, that's 0729047.

I opened this case for an issue related to the VBR recurring hang (with all annected consequences), but after further analysis I discovered that one of the issues I'm facing is reconducible to the topic of this thread.

Thanks.

[Dima P.]

Hello Matteo,

Thank you for the case details! The case you are referring is 07290477 the one you shared is account number, so I've erased it from your post. Can you clarify if you are running malware detection with guest file indexing analysis enabled or not? The issue appeared after upgrade or you've seen it before? Thank you!

[iDeNt_5]

Hi Dmitry,

Sorry for my mistake, too many hours of work these days made me wrong

Yes, until this morning I worked with Suspicious activity detection and from somedays also the Encryption detection enabled (the issue has presented in both cases), so today I decided to completely disable both of them and after some time I noted the Veeam Guest Catalog service started to require less CPU cycles, but at least for now I decided to still keeping the related process able to use only a CPU, by manually setting the Affinity Rule.

I also confirm the issue is started to happen (and became worse to worst in the sequential days) since I upgraded the VBR from V2.0 to V12.1.2.

Thanks.

[Dima P.]

Hello Matteo,

Indeed guest file index analysis requires additional resources when enabled, so it can lead to backup catalog consuming additional resources. Support team can propose further troubleshooting steps. Let us know how investigation goes, thank you!

[iDeNt_5]

Hi Dmitry,

I agree with you with that statement, but here it seems like the more resources that service can use the more it actually use.

Just for example purpose, this VBR previously had 14 cores and it runs on a slower ESXi host than now, so as a test one time I tried to give it 24 cores (more powerful than the previous ones, since this time they were based on better HW CPUs), but the result has been the same, the VBR hanged and stopped to respond after a short period of time from the boot up.

I'm awaiting for the Veeam support help from a couple of days, so I'll let you know what we'll find about that.

[iDeNt_5]

We are continuing to work on this case with the support team.

Now the issue changed from CPU usage 100% by the Veeam Guest Catalog Service process to RAM continuous increase since the boot of the VBR until the infinite, that is potentially causing the VBR outage for the out of memory condition (and it already happened once in recent days).

[karsten123]

is it fixed in v12.2?

[iDeNt_5]

I guess not, since the Veeam R&D team is still working on a private fix, in order to solve this issue that in my case it seems to be caused principally by a computer that has about 5000 restore points (and most of them are marked as suspicious).

[Dima P.]

Hello guys,

Some improvements in this area were included into 12.2 for sure as well as ability to limit the size of the catalog by setting up a retention for index shorter than the actually backup job retention. Matteo case is, indeed, still under investigation, I'll check the most recent state with the team. Stay tuned.

[Gostev]

Yeah I remember writing in the What's New document for 12.2 about the resource consumption reduction for this functionality.

[iDeNt_5]

Hi all,

I read this into the what's new document:
Malware detection index improvements — the CPU consumption during the guest file system index
analysis has been lowered significantly to reduce the backup server load. Further, you can now reduce
the index retention period if your guest catalog storage is running out of space with the IndexRetentionDays
(DWORD) registry value under the HKLM\SOFTWARE\Veeam\Veeam Backup and Replication key on
the backup server with your preferred retention duration in days.

The point is that now our first issue related to this feature has become RAM usage.
If the malware detection features of VBR are enabled, after a few days the VBR RAM goes into out of memory condition.

Has anything been done about this?

Thanks.