Comprehensive data protection for all workloads
davemann
Influencer
Posts: 11
Liked: 4 times
Joined: Jul 15, 2016 8:40 pm
Full Name: Dave Reichert
Contact:

Guest Catalog Service is going nuts post 12.1 Upgrade

Post by davemann »

Hello -

I was wondering if anyone else has seen similar behavior's after upgrading to v12.1...

The other night during our backup window, I noticed that my VBR console was very sluggish, and that jobs were not progressing as quickly as normal. I then noticed in vCenter that my VBR server was 100% pinned on CPU, consuming ~26Ghz worth of CPU (ouch!). I logged into the system to observe that the Guest Catalog service was causing 100% CPU utilization, and showed no signs of stopping.

I restarted the service, and things calmed down - for a bit, but then it started all over again. For now I seem to have to keep this service stopped in order for things to process correctly, and I don't know the best way to deal with this, so I'll put in a support ticket.

Just checking to see if this is known?
Gostev
Chief Product Officer
Posts: 31690
Liked: 7201 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Guest Catalog Service is going nuts post 12.1 Upgrade

Post by Gostev »

Hello, it is likely a one-time analysis of all existing guest indexes by the malware detection functionality, but I will let the corresponding PM confirm and share any additional details.
davemann
Influencer
Posts: 11
Liked: 4 times
Joined: Jul 15, 2016 8:40 pm
Full Name: Dave Reichert
Contact:

Re: Guest Catalog Service is going nuts post 12.1 Upgrade

Post by davemann »

Thanks! I've submitted a case with logs from the VBR server.

For now, I've disabled the suspicious activity detection functionality and I've started the service back up, and hopefully I can get through tonight's window fine.
Gostev
Chief Product Officer
Posts: 31690
Liked: 7201 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Guest Catalog Service is going nuts post 12.1 Upgrade

Post by Gostev »

So apparently I was wrong about the "all existing" part, only newly created indexes are analyzed. As such this is unexpected and needs to be investigated.
Dima P.
Product Manager
Posts: 14643
Liked: 1675 times
Joined: Feb 04, 2013 2:07 pm
Full Name: Dmitry Popov
Location: Prague
Contact:

Re: Guest Catalog Service is going nuts post 12.1 Upgrade

Post by Dima P. »

Hello Dave,

If by any chance you open a support case, please, do not forget to share the case ID with us! Thank you!
lmoglie
Lurker
Posts: 2
Liked: 2 times
Joined: Dec 15, 2023 1:56 pm
Full Name: Lorenzo Moglie
Contact:

Re: Guest Catalog Service is going nuts post 12.1 Upgrade

Post by lmoglie » 1 person likes this post

Hello,
I have exactly the same problem.
I solved it temporarily, setting the CPU affinity for the process, but I'll for sure open a case to investigate.
Mildur
Product Manager
Posts: 9689
Liked: 2562 times
Joined: May 13, 2017 4:51 pm
Full Name: Fabian K.
Location: Switzerland
Contact:

Re: Guest Catalog Service is going nuts post 12.1 Upgrade

Post by Mildur » 1 person likes this post

Hi Lorenzo

We have a hotfix ready for this issue. Please reference to Bug ID 623177 in your case.
veeam-backup-replication-f2/top-issues- ... 91449.html
#4: High CPU usage by CatalogDataService
Scope: Rare condition that depends on a number of parameters, more likely to happen in larger environments.
Symptoms: CatalogDataService consumes all the CPU cycles it can get.
Cause: Resource scheduling bug in the index scan engine of the malware detection functionality.
Status: Contact support for a hotfix referencing Bug ID 623177.
Best,
Fabian
Product Management Analyst @ Veeam Software
lmoglie
Lurker
Posts: 2
Liked: 2 times
Joined: Dec 15, 2023 1:56 pm
Full Name: Lorenzo Moglie
Contact:

Re: Guest Catalog Service is going nuts post 12.1 Upgrade

Post by lmoglie » 1 person likes this post

Thanks a lot Fabian
Nodnarb
Service Provider
Posts: 14
Liked: 2 times
Joined: Dec 19, 2018 3:44 pm
Full Name: Brandon
Contact:

Re: Guest Catalog Service is going nuts post 12.1 Upgrade

Post by Nodnarb »

Hi all. I'm also having an issue with high CPU usage on the CatalogDataService process if "Enable file system activity analysis" is checked. The process continues to use high CPU even after jobs have completed.

I contacted Support (case 07081739) and applied the hotfix, but I am still having the high CPU usage issue. I'll update here once we've resolved it.
Gaba2001
Lurker
Posts: 1
Liked: 1 time
Joined: Jan 16, 2024 8:35 am
Full Name: Gabor Batari
Contact:

Re: Guest Catalog Service is going nuts post 12.1 Upgrade

Post by Gaba2001 » 1 person likes this post

I have same problem.

I opened a support case and got a private fix.
veeam_backup_12.1.0.2131_PrivateFix_TF623294.exe

I installed that.

Now the service appears is in normal condition. No high CPU and memory use.
Dima P.
Product Manager
Posts: 14643
Liked: 1675 times
Joined: Feb 04, 2013 2:07 pm
Full Name: Dmitry Popov
Location: Prague
Contact:

Re: Guest Catalog Service is going nuts post 12.1 Upgrade

Post by Dima P. »

Hello Gabor,

Thank you for sharing the updates and glad that your issue was addressed by the hotfix!
Nodnarb
Service Provider
Posts: 14
Liked: 2 times
Joined: Dec 19, 2018 3:44 pm
Full Name: Brandon
Contact:

Re: Guest Catalog Service is going nuts post 12.1 Upgrade

Post by Nodnarb »

Nodnarb wrote: Jan 10, 2024 2:48 pm Hi all. I'm also having an issue with high CPU usage on the CatalogDataService process if "Enable file system activity analysis" is checked. The process continues to use high CPU even after jobs have completed.

I contacted Support (case 07081739) and applied the hotfix, but I am still having the high CPU usage issue. I'll update here once we've resolved it.
This never quite got resolved...We were given a registry key to reduce the number of threads the process uses, but usage still remained pretty constant. I eventually turned the feature off. I'll try it again in future updates.
Gostev
Chief Product Officer
Posts: 31690
Liked: 7201 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Guest Catalog Service is going nuts post 12.1 Upgrade

Post by Gostev »

This issue was fixed back in January, it is specifically called out in 12.1.1 release notes.
popjls
Enthusiast
Posts: 57
Liked: 5 times
Joined: Jun 25, 2018 3:41 am
Contact:

Re: Guest Catalog Service is going nuts post 12.1 Upgrade

Post by popjls »

This issue was fixed back in January, it is specifically called out in 12.1.1 release notes.
I found this thread because I'm having this exact issue and I'm on 12.1.1.56. I'll submit a support case as it's only been an issue since upgrading to 12.1 for me.
LU86
Lurker
Posts: 2
Liked: never
Joined: Mar 10, 2022 1:51 pm
Full Name: Lukas Schleinzer
Contact:

Re: Guest Catalog Service is going nuts post 12.1 Upgrade

Post by LU86 »

Encountered the same issues after a routine Windows Update maintanance window.
VBR version on this server is 12.1.1.56 since 2/13/2024 and never had the CPU usage issue before.

I noticed that the following Windows Update was installed yesterday:
KB5036343 - Description of the security update for SQL Server 2022 CU12: April 9, 2024

Maybe that triggered some kind of "catalog refresh" that took several hours and un-zipped the content of the archived data of the VM fileserver that has file indexing active.
The un-zipping of those files (drive:\VBRCatalog\Index\Machines\...) almost filled up the entire Disk because it un-zipped those files back to: 03/19/2024.


I'll definately sumbit a support case on that matter :)
Mildur
Product Manager
Posts: 9689
Liked: 2562 times
Joined: May 13, 2017 4:51 pm
Full Name: Fabian K.
Location: Switzerland
Contact:

Re: Guest Catalog Service is going nuts post 12.1 Upgrade

Post by Mildur »

Hi Lukas

What's your case number?

Best,
Fabian
Product Management Analyst @ Veeam Software
LU86
Lurker
Posts: 2
Liked: never
Joined: Mar 10, 2022 1:51 pm
Full Name: Lukas Schleinzer
Contact:

Re: Guest Catalog Service is going nuts post 12.1 Upgrade

Post by LU86 »

Hi Fabian,

Case #07263318


Regards,
Lukas.
iDeNt_5
Influencer
Posts: 15
Liked: 4 times
Joined: Feb 09, 2024 5:34 pm
Full Name: Matteo Fringuelli
Contact:

Re: Guest Catalog Service is going nuts post 12.1 Upgrade

Post by iDeNt_5 »

Hi all,

Are there any news about this issue?
It seems like I'm facing the same with a Customer with the VBR 12.1.2.172 version.

Many thanks.
Dima P.
Product Manager
Posts: 14643
Liked: 1675 times
Joined: Feb 04, 2013 2:07 pm
Full Name: Dmitry Popov
Location: Prague
Contact:

Re: Guest Catalog Service is going nuts post 12.1 Upgrade

Post by Dima P. »

Hello Matteo,

Do you have a case ID to share? Thank you!
iDeNt_5
Influencer
Posts: 15
Liked: 4 times
Joined: Feb 09, 2024 5:34 pm
Full Name: Matteo Fringuelli
Contact:

Re: Guest Catalog Service is going nuts post 12.1 Upgrade

Post by iDeNt_5 » 1 person likes this post

Hi Dima,

Sure, that's 0729047.

I opened this case for an issue related to the VBR recurring hang (with all annected consequences), but after further analysis I discovered that one of the issues I'm facing is reconducible to the topic of this thread.

Thanks.
Dima P.
Product Manager
Posts: 14643
Liked: 1675 times
Joined: Feb 04, 2013 2:07 pm
Full Name: Dmitry Popov
Location: Prague
Contact:

Re: Guest Catalog Service is going nuts post 12.1 Upgrade

Post by Dima P. »

Hello Matteo,

Thank you for the case details! The case you are referring is 07290477 the one you shared is account number, so I've erased it from your post. Can you clarify if you are running malware detection with guest file indexing analysis enabled or not? The issue appeared after upgrade or you've seen it before? Thank you!
iDeNt_5
Influencer
Posts: 15
Liked: 4 times
Joined: Feb 09, 2024 5:34 pm
Full Name: Matteo Fringuelli
Contact:

Re: Guest Catalog Service is going nuts post 12.1 Upgrade

Post by iDeNt_5 »

Hi Dmitry,

Sorry for my mistake, too many hours of work these days made me wrong :-)

Yes, until this morning I worked with Suspicious activity detection and from somedays also the Encryption detection enabled (the issue has presented in both cases), so today I decided to completely disable both of them and after some time I noted the Veeam Guest Catalog service started to require less CPU cycles, but at least for now I decided to still keeping the related process able to use only a CPU, by manually setting the Affinity Rule.

I also confirm the issue is started to happen (and became worse to worst in the sequential days) since I upgraded the VBR from V2.0 to V12.1.2.

Thanks.
Dima P.
Product Manager
Posts: 14643
Liked: 1675 times
Joined: Feb 04, 2013 2:07 pm
Full Name: Dmitry Popov
Location: Prague
Contact:

Re: Guest Catalog Service is going nuts post 12.1 Upgrade

Post by Dima P. »

Hello Matteo,

Indeed guest file index analysis requires additional resources when enabled, so it can lead to backup catalog consuming additional resources. Support team can propose further troubleshooting steps. Let us know how investigation goes, thank you!
iDeNt_5
Influencer
Posts: 15
Liked: 4 times
Joined: Feb 09, 2024 5:34 pm
Full Name: Matteo Fringuelli
Contact:

Re: Guest Catalog Service is going nuts post 12.1 Upgrade

Post by iDeNt_5 » 1 person likes this post

Hi Dmitry,

I agree with you with that statement, but here it seems like the more resources that service can use the more it actually use.

Just for example purpose, this VBR previously had 14 cores and it runs on a slower ESXi host than now, so as a test one time I tried to give it 24 cores (more powerful than the previous ones, since this time they were based on better HW CPUs), but the result has been the same, the VBR hanged and stopped to respond after a short period of time from the boot up.

I'm awaiting for the Veeam support help from a couple of days, so I'll let you know what we'll find about that.
iDeNt_5
Influencer
Posts: 15
Liked: 4 times
Joined: Feb 09, 2024 5:34 pm
Full Name: Matteo Fringuelli
Contact:

Re: Guest Catalog Service is going nuts post 12.1 Upgrade

Post by iDeNt_5 »

We are continuing to work on this case with the support team.

Now the issue changed from CPU usage 100% by the Veeam Guest Catalog Service process to RAM continuous increase since the boot of the VBR until the infinite, that is potentially causing the VBR outage for the out of memory condition (and it already happened once in recent days).
karsten123
Service Provider
Posts: 467
Liked: 119 times
Joined: Apr 03, 2019 6:53 am
Full Name: Karsten Meja
Contact:

Re: Guest Catalog Service is going nuts post 12.1 Upgrade

Post by karsten123 »

is it fixed in v12.2?
iDeNt_5
Influencer
Posts: 15
Liked: 4 times
Joined: Feb 09, 2024 5:34 pm
Full Name: Matteo Fringuelli
Contact:

Re: Guest Catalog Service is going nuts post 12.1 Upgrade

Post by iDeNt_5 »

I guess not, since the Veeam R&D team is still working on a private fix, in order to solve this issue that in my case it seems to be caused principally by a computer that has about 5000 restore points (and most of them are marked as suspicious).
Dima P.
Product Manager
Posts: 14643
Liked: 1675 times
Joined: Feb 04, 2013 2:07 pm
Full Name: Dmitry Popov
Location: Prague
Contact:

Re: Guest Catalog Service is going nuts post 12.1 Upgrade

Post by Dima P. »

Hello guys,

Some improvements in this area were included into 12.2 for sure as well as ability to limit the size of the catalog by setting up a retention for index shorter than the actually backup job retention. Matteo case is, indeed, still under investigation, I'll check the most recent state with the team. Stay tuned.
Gostev
Chief Product Officer
Posts: 31690
Liked: 7201 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Guest Catalog Service is going nuts post 12.1 Upgrade

Post by Gostev »

Yeah I remember writing in the What's New document for 12.2 about the resource consumption reduction for this functionality.
iDeNt_5
Influencer
Posts: 15
Liked: 4 times
Joined: Feb 09, 2024 5:34 pm
Full Name: Matteo Fringuelli
Contact:

Re: Guest Catalog Service is going nuts post 12.1 Upgrade

Post by iDeNt_5 »

Hi all,

I read this into the what's new document:
Malware detection index improvements — the CPU consumption during the guest file system index
analysis has been lowered significantly to reduce the backup server load. Further, you can now reduce
the index retention period if your guest catalog storage is running out of space with the IndexRetentionDays
(DWORD) registry value under the HKLM\SOFTWARE\Veeam\Veeam Backup and Replication key on
the backup server with your preferred retention duration in days.

The point is that now our first issue related to this feature has become RAM usage.
If the malware detection features of VBR are enabled, after a few days the VBR RAM goes into out of memory condition.

Has anything been done about this?

Thanks.
Post Reply

Who is online

Users browsing this forum: ma3011 and 37 guests