-
- Influencer
- Posts: 11
- Liked: 4 times
- Joined: Jul 15, 2016 8:40 pm
- Full Name: Dave Reichert
- Contact:
Guest Catalog Service is going nuts post 12.1 Upgrade
Hello -
I was wondering if anyone else has seen similar behavior's after upgrading to v12.1...
The other night during our backup window, I noticed that my VBR console was very sluggish, and that jobs were not progressing as quickly as normal. I then noticed in vCenter that my VBR server was 100% pinned on CPU, consuming ~26Ghz worth of CPU (ouch!). I logged into the system to observe that the Guest Catalog service was causing 100% CPU utilization, and showed no signs of stopping.
I restarted the service, and things calmed down - for a bit, but then it started all over again. For now I seem to have to keep this service stopped in order for things to process correctly, and I don't know the best way to deal with this, so I'll put in a support ticket.
Just checking to see if this is known?
I was wondering if anyone else has seen similar behavior's after upgrading to v12.1...
The other night during our backup window, I noticed that my VBR console was very sluggish, and that jobs were not progressing as quickly as normal. I then noticed in vCenter that my VBR server was 100% pinned on CPU, consuming ~26Ghz worth of CPU (ouch!). I logged into the system to observe that the Guest Catalog service was causing 100% CPU utilization, and showed no signs of stopping.
I restarted the service, and things calmed down - for a bit, but then it started all over again. For now I seem to have to keep this service stopped in order for things to process correctly, and I don't know the best way to deal with this, so I'll put in a support ticket.
Just checking to see if this is known?
-
- Chief Product Officer
- Posts: 31690
- Liked: 7201 times
- Joined: Jan 01, 2006 1:01 am
- Location: Baar, Switzerland
- Contact:
Re: Guest Catalog Service is going nuts post 12.1 Upgrade
Hello, it is likely a one-time analysis of all existing guest indexes by the malware detection functionality, but I will let the corresponding PM confirm and share any additional details.
-
- Influencer
- Posts: 11
- Liked: 4 times
- Joined: Jul 15, 2016 8:40 pm
- Full Name: Dave Reichert
- Contact:
Re: Guest Catalog Service is going nuts post 12.1 Upgrade
Thanks! I've submitted a case with logs from the VBR server.
For now, I've disabled the suspicious activity detection functionality and I've started the service back up, and hopefully I can get through tonight's window fine.
For now, I've disabled the suspicious activity detection functionality and I've started the service back up, and hopefully I can get through tonight's window fine.
-
- Chief Product Officer
- Posts: 31690
- Liked: 7201 times
- Joined: Jan 01, 2006 1:01 am
- Location: Baar, Switzerland
- Contact:
Re: Guest Catalog Service is going nuts post 12.1 Upgrade
So apparently I was wrong about the "all existing" part, only newly created indexes are analyzed. As such this is unexpected and needs to be investigated.
-
- Product Manager
- Posts: 14643
- Liked: 1675 times
- Joined: Feb 04, 2013 2:07 pm
- Full Name: Dmitry Popov
- Location: Prague
- Contact:
Re: Guest Catalog Service is going nuts post 12.1 Upgrade
Hello Dave,
If by any chance you open a support case, please, do not forget to share the case ID with us! Thank you!
If by any chance you open a support case, please, do not forget to share the case ID with us! Thank you!
-
- Lurker
- Posts: 2
- Liked: 2 times
- Joined: Dec 15, 2023 1:56 pm
- Full Name: Lorenzo Moglie
- Contact:
Re: Guest Catalog Service is going nuts post 12.1 Upgrade
Hello,
I have exactly the same problem.
I solved it temporarily, setting the CPU affinity for the process, but I'll for sure open a case to investigate.
I have exactly the same problem.
I solved it temporarily, setting the CPU affinity for the process, but I'll for sure open a case to investigate.
-
- Product Manager
- Posts: 9689
- Liked: 2562 times
- Joined: May 13, 2017 4:51 pm
- Full Name: Fabian K.
- Location: Switzerland
- Contact:
Re: Guest Catalog Service is going nuts post 12.1 Upgrade
Hi Lorenzo
We have a hotfix ready for this issue. Please reference to Bug ID 623177 in your case.
veeam-backup-replication-f2/top-issues- ... 91449.html
Fabian
We have a hotfix ready for this issue. Please reference to Bug ID 623177 in your case.
veeam-backup-replication-f2/top-issues- ... 91449.html
Best,#4: High CPU usage by CatalogDataService
Scope: Rare condition that depends on a number of parameters, more likely to happen in larger environments.
Symptoms: CatalogDataService consumes all the CPU cycles it can get.
Cause: Resource scheduling bug in the index scan engine of the malware detection functionality.
Status: Contact support for a hotfix referencing Bug ID 623177.
Fabian
Product Management Analyst @ Veeam Software
-
- Lurker
- Posts: 2
- Liked: 2 times
- Joined: Dec 15, 2023 1:56 pm
- Full Name: Lorenzo Moglie
- Contact:
Re: Guest Catalog Service is going nuts post 12.1 Upgrade
Thanks a lot Fabian
-
- Service Provider
- Posts: 14
- Liked: 2 times
- Joined: Dec 19, 2018 3:44 pm
- Full Name: Brandon
- Contact:
Re: Guest Catalog Service is going nuts post 12.1 Upgrade
Hi all. I'm also having an issue with high CPU usage on the CatalogDataService process if "Enable file system activity analysis" is checked. The process continues to use high CPU even after jobs have completed.
I contacted Support (case 07081739) and applied the hotfix, but I am still having the high CPU usage issue. I'll update here once we've resolved it.
I contacted Support (case 07081739) and applied the hotfix, but I am still having the high CPU usage issue. I'll update here once we've resolved it.
-
- Lurker
- Posts: 1
- Liked: 1 time
- Joined: Jan 16, 2024 8:35 am
- Full Name: Gabor Batari
- Contact:
Re: Guest Catalog Service is going nuts post 12.1 Upgrade
I have same problem.
I opened a support case and got a private fix.
veeam_backup_12.1.0.2131_PrivateFix_TF623294.exe
I installed that.
Now the service appears is in normal condition. No high CPU and memory use.
I opened a support case and got a private fix.
veeam_backup_12.1.0.2131_PrivateFix_TF623294.exe
I installed that.
Now the service appears is in normal condition. No high CPU and memory use.
-
- Product Manager
- Posts: 14643
- Liked: 1675 times
- Joined: Feb 04, 2013 2:07 pm
- Full Name: Dmitry Popov
- Location: Prague
- Contact:
Re: Guest Catalog Service is going nuts post 12.1 Upgrade
Hello Gabor,
Thank you for sharing the updates and glad that your issue was addressed by the hotfix!
Thank you for sharing the updates and glad that your issue was addressed by the hotfix!
-
- Service Provider
- Posts: 14
- Liked: 2 times
- Joined: Dec 19, 2018 3:44 pm
- Full Name: Brandon
- Contact:
Re: Guest Catalog Service is going nuts post 12.1 Upgrade
This never quite got resolved...We were given a registry key to reduce the number of threads the process uses, but usage still remained pretty constant. I eventually turned the feature off. I'll try it again in future updates.Nodnarb wrote: ↑Jan 10, 2024 2:48 pm Hi all. I'm also having an issue with high CPU usage on the CatalogDataService process if "Enable file system activity analysis" is checked. The process continues to use high CPU even after jobs have completed.
I contacted Support (case 07081739) and applied the hotfix, but I am still having the high CPU usage issue. I'll update here once we've resolved it.
-
- Chief Product Officer
- Posts: 31690
- Liked: 7201 times
- Joined: Jan 01, 2006 1:01 am
- Location: Baar, Switzerland
- Contact:
Re: Guest Catalog Service is going nuts post 12.1 Upgrade
This issue was fixed back in January, it is specifically called out in 12.1.1 release notes.
-
- Enthusiast
- Posts: 57
- Liked: 5 times
- Joined: Jun 25, 2018 3:41 am
- Contact:
Re: Guest Catalog Service is going nuts post 12.1 Upgrade
I found this thread because I'm having this exact issue and I'm on 12.1.1.56. I'll submit a support case as it's only been an issue since upgrading to 12.1 for me.This issue was fixed back in January, it is specifically called out in 12.1.1 release notes.
-
- Lurker
- Posts: 2
- Liked: never
- Joined: Mar 10, 2022 1:51 pm
- Full Name: Lukas Schleinzer
- Contact:
Re: Guest Catalog Service is going nuts post 12.1 Upgrade
Encountered the same issues after a routine Windows Update maintanance window.
VBR version on this server is 12.1.1.56 since 2/13/2024 and never had the CPU usage issue before.
I noticed that the following Windows Update was installed yesterday:
KB5036343 - Description of the security update for SQL Server 2022 CU12: April 9, 2024
Maybe that triggered some kind of "catalog refresh" that took several hours and un-zipped the content of the archived data of the VM fileserver that has file indexing active.
The un-zipping of those files (drive:\VBRCatalog\Index\Machines\...) almost filled up the entire Disk because it un-zipped those files back to: 03/19/2024.
I'll definately sumbit a support case on that matter
VBR version on this server is 12.1.1.56 since 2/13/2024 and never had the CPU usage issue before.
I noticed that the following Windows Update was installed yesterday:
KB5036343 - Description of the security update for SQL Server 2022 CU12: April 9, 2024
Maybe that triggered some kind of "catalog refresh" that took several hours and un-zipped the content of the archived data of the VM fileserver that has file indexing active.
The un-zipping of those files (drive:\VBRCatalog\Index\Machines\...) almost filled up the entire Disk because it un-zipped those files back to: 03/19/2024.
I'll definately sumbit a support case on that matter
-
- Product Manager
- Posts: 9689
- Liked: 2562 times
- Joined: May 13, 2017 4:51 pm
- Full Name: Fabian K.
- Location: Switzerland
- Contact:
Re: Guest Catalog Service is going nuts post 12.1 Upgrade
Hi Lukas
What's your case number?
Best,
Fabian
What's your case number?
Best,
Fabian
Product Management Analyst @ Veeam Software
-
- Lurker
- Posts: 2
- Liked: never
- Joined: Mar 10, 2022 1:51 pm
- Full Name: Lukas Schleinzer
- Contact:
Re: Guest Catalog Service is going nuts post 12.1 Upgrade
Hi Fabian,
Case #07263318
Regards,
Lukas.
Case #07263318
Regards,
Lukas.
-
- Influencer
- Posts: 15
- Liked: 4 times
- Joined: Feb 09, 2024 5:34 pm
- Full Name: Matteo Fringuelli
- Contact:
Re: Guest Catalog Service is going nuts post 12.1 Upgrade
Hi all,
Are there any news about this issue?
It seems like I'm facing the same with a Customer with the VBR 12.1.2.172 version.
Many thanks.
Are there any news about this issue?
It seems like I'm facing the same with a Customer with the VBR 12.1.2.172 version.
Many thanks.
-
- Product Manager
- Posts: 14643
- Liked: 1675 times
- Joined: Feb 04, 2013 2:07 pm
- Full Name: Dmitry Popov
- Location: Prague
- Contact:
Re: Guest Catalog Service is going nuts post 12.1 Upgrade
Hello Matteo,
Do you have a case ID to share? Thank you!
Do you have a case ID to share? Thank you!
-
- Influencer
- Posts: 15
- Liked: 4 times
- Joined: Feb 09, 2024 5:34 pm
- Full Name: Matteo Fringuelli
- Contact:
Re: Guest Catalog Service is going nuts post 12.1 Upgrade
Hi Dima,
Sure, that's 0729047.
I opened this case for an issue related to the VBR recurring hang (with all annected consequences), but after further analysis I discovered that one of the issues I'm facing is reconducible to the topic of this thread.
Thanks.
Sure, that's 0729047.
I opened this case for an issue related to the VBR recurring hang (with all annected consequences), but after further analysis I discovered that one of the issues I'm facing is reconducible to the topic of this thread.
Thanks.
-
- Product Manager
- Posts: 14643
- Liked: 1675 times
- Joined: Feb 04, 2013 2:07 pm
- Full Name: Dmitry Popov
- Location: Prague
- Contact:
Re: Guest Catalog Service is going nuts post 12.1 Upgrade
Hello Matteo,
Thank you for the case details! The case you are referring is 07290477 the one you shared is account number, so I've erased it from your post. Can you clarify if you are running malware detection with guest file indexing analysis enabled or not? The issue appeared after upgrade or you've seen it before? Thank you!
Thank you for the case details! The case you are referring is 07290477 the one you shared is account number, so I've erased it from your post. Can you clarify if you are running malware detection with guest file indexing analysis enabled or not? The issue appeared after upgrade or you've seen it before? Thank you!
-
- Influencer
- Posts: 15
- Liked: 4 times
- Joined: Feb 09, 2024 5:34 pm
- Full Name: Matteo Fringuelli
- Contact:
Re: Guest Catalog Service is going nuts post 12.1 Upgrade
Hi Dmitry,
Sorry for my mistake, too many hours of work these days made me wrong
Yes, until this morning I worked with Suspicious activity detection and from somedays also the Encryption detection enabled (the issue has presented in both cases), so today I decided to completely disable both of them and after some time I noted the Veeam Guest Catalog service started to require less CPU cycles, but at least for now I decided to still keeping the related process able to use only a CPU, by manually setting the Affinity Rule.
I also confirm the issue is started to happen (and became worse to worst in the sequential days) since I upgraded the VBR from V2.0 to V12.1.2.
Thanks.
Sorry for my mistake, too many hours of work these days made me wrong
Yes, until this morning I worked with Suspicious activity detection and from somedays also the Encryption detection enabled (the issue has presented in both cases), so today I decided to completely disable both of them and after some time I noted the Veeam Guest Catalog service started to require less CPU cycles, but at least for now I decided to still keeping the related process able to use only a CPU, by manually setting the Affinity Rule.
I also confirm the issue is started to happen (and became worse to worst in the sequential days) since I upgraded the VBR from V2.0 to V12.1.2.
Thanks.
-
- Product Manager
- Posts: 14643
- Liked: 1675 times
- Joined: Feb 04, 2013 2:07 pm
- Full Name: Dmitry Popov
- Location: Prague
- Contact:
Re: Guest Catalog Service is going nuts post 12.1 Upgrade
Hello Matteo,
Indeed guest file index analysis requires additional resources when enabled, so it can lead to backup catalog consuming additional resources. Support team can propose further troubleshooting steps. Let us know how investigation goes, thank you!
Indeed guest file index analysis requires additional resources when enabled, so it can lead to backup catalog consuming additional resources. Support team can propose further troubleshooting steps. Let us know how investigation goes, thank you!
-
- Influencer
- Posts: 15
- Liked: 4 times
- Joined: Feb 09, 2024 5:34 pm
- Full Name: Matteo Fringuelli
- Contact:
Re: Guest Catalog Service is going nuts post 12.1 Upgrade
Hi Dmitry,
I agree with you with that statement, but here it seems like the more resources that service can use the more it actually use.
Just for example purpose, this VBR previously had 14 cores and it runs on a slower ESXi host than now, so as a test one time I tried to give it 24 cores (more powerful than the previous ones, since this time they were based on better HW CPUs), but the result has been the same, the VBR hanged and stopped to respond after a short period of time from the boot up.
I'm awaiting for the Veeam support help from a couple of days, so I'll let you know what we'll find about that.
I agree with you with that statement, but here it seems like the more resources that service can use the more it actually use.
Just for example purpose, this VBR previously had 14 cores and it runs on a slower ESXi host than now, so as a test one time I tried to give it 24 cores (more powerful than the previous ones, since this time they were based on better HW CPUs), but the result has been the same, the VBR hanged and stopped to respond after a short period of time from the boot up.
I'm awaiting for the Veeam support help from a couple of days, so I'll let you know what we'll find about that.
-
- Influencer
- Posts: 15
- Liked: 4 times
- Joined: Feb 09, 2024 5:34 pm
- Full Name: Matteo Fringuelli
- Contact:
Re: Guest Catalog Service is going nuts post 12.1 Upgrade
We are continuing to work on this case with the support team.
Now the issue changed from CPU usage 100% by the Veeam Guest Catalog Service process to RAM continuous increase since the boot of the VBR until the infinite, that is potentially causing the VBR outage for the out of memory condition (and it already happened once in recent days).
Now the issue changed from CPU usage 100% by the Veeam Guest Catalog Service process to RAM continuous increase since the boot of the VBR until the infinite, that is potentially causing the VBR outage for the out of memory condition (and it already happened once in recent days).
-
- Service Provider
- Posts: 467
- Liked: 119 times
- Joined: Apr 03, 2019 6:53 am
- Full Name: Karsten Meja
- Contact:
Re: Guest Catalog Service is going nuts post 12.1 Upgrade
is it fixed in v12.2?
-
- Influencer
- Posts: 15
- Liked: 4 times
- Joined: Feb 09, 2024 5:34 pm
- Full Name: Matteo Fringuelli
- Contact:
Re: Guest Catalog Service is going nuts post 12.1 Upgrade
I guess not, since the Veeam R&D team is still working on a private fix, in order to solve this issue that in my case it seems to be caused principally by a computer that has about 5000 restore points (and most of them are marked as suspicious).
-
- Product Manager
- Posts: 14643
- Liked: 1675 times
- Joined: Feb 04, 2013 2:07 pm
- Full Name: Dmitry Popov
- Location: Prague
- Contact:
Re: Guest Catalog Service is going nuts post 12.1 Upgrade
Hello guys,
Some improvements in this area were included into 12.2 for sure as well as ability to limit the size of the catalog by setting up a retention for index shorter than the actually backup job retention. Matteo case is, indeed, still under investigation, I'll check the most recent state with the team. Stay tuned.
Some improvements in this area were included into 12.2 for sure as well as ability to limit the size of the catalog by setting up a retention for index shorter than the actually backup job retention. Matteo case is, indeed, still under investigation, I'll check the most recent state with the team. Stay tuned.
-
- Chief Product Officer
- Posts: 31690
- Liked: 7201 times
- Joined: Jan 01, 2006 1:01 am
- Location: Baar, Switzerland
- Contact:
Re: Guest Catalog Service is going nuts post 12.1 Upgrade
Yeah I remember writing in the What's New document for 12.2 about the resource consumption reduction for this functionality.
-
- Influencer
- Posts: 15
- Liked: 4 times
- Joined: Feb 09, 2024 5:34 pm
- Full Name: Matteo Fringuelli
- Contact:
Re: Guest Catalog Service is going nuts post 12.1 Upgrade
Hi all,
I read this into the what's new document:
Malware detection index improvements — the CPU consumption during the guest file system index
analysis has been lowered significantly to reduce the backup server load. Further, you can now reduce
the index retention period if your guest catalog storage is running out of space with the IndexRetentionDays
(DWORD) registry value under the HKLM\SOFTWARE\Veeam\Veeam Backup and Replication key on
the backup server with your preferred retention duration in days.
The point is that now our first issue related to this feature has become RAM usage.
If the malware detection features of VBR are enabled, after a few days the VBR RAM goes into out of memory condition.
Has anything been done about this?
Thanks.
I read this into the what's new document:
Malware detection index improvements — the CPU consumption during the guest file system index
analysis has been lowered significantly to reduce the backup server load. Further, you can now reduce
the index retention period if your guest catalog storage is running out of space with the IndexRetentionDays
(DWORD) registry value under the HKLM\SOFTWARE\Veeam\Veeam Backup and Replication key on
the backup server with your preferred retention duration in days.
The point is that now our first issue related to this feature has become RAM usage.
If the malware detection features of VBR are enabled, after a few days the VBR RAM goes into out of memory condition.
Has anything been done about this?
Thanks.
Who is online
Users browsing this forum: ma3011 and 37 guests