Monitoring and reporting for Veeam Data Platform
Post Reply
omegagx
Enthusiast
Posts: 68
Liked: 4 times
Joined: May 09, 2017 6:33 pm
Full Name: Mike G
Contact:

SQL usage goes to 100% immediately after upgrading to 11a version

Post by omegagx »

after upgrading Veeam ONE yesterday to version 11a, the SQL server was stuck at 100% of CPU with the following query causing the issue with the Veeam ONE database:

delete #IDs
where ID not in
( select ID from #IDS_Current )

I had to stop the Veeam ONE service to be able to free up SQL server CPU. Starting it again and starting the collection caused the issue to occur again about 8 minutes after the collection started. Looks like it's a bug.
Vitaliy S.
VP, Product Management
Posts: 27377
Liked: 2800 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: SQL usage goes to 100% immediately after upgrading to 11a version

Post by Vitaliy S. »

Hi Mike,

Please open the support case and let our technical team review the issue.

Thanks!
omegagx
Enthusiast
Posts: 68
Liked: 4 times
Joined: May 09, 2017 6:33 pm
Full Name: Mike G
Contact:

Re: SQL usage goes to 100% immediately after upgrading to 11a version

Post by omegagx »

I already did open the case #05078863 yesterday but so far 2 support engineers couldn't help and even though it's severity 2, my Veeam ONE is currently down.
Vitaliy S.
VP, Product Management
Posts: 27377
Liked: 2800 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: SQL usage goes to 100% immediately after upgrading to 11a version

Post by Vitaliy S. »

Feel free to ask for escalation, our RnD team should be able to help once we have this ticket in our queue.
omegagx
Enthusiast
Posts: 68
Liked: 4 times
Joined: May 09, 2017 6:33 pm
Full Name: Mike G
Contact:

Re: SQL usage goes to 100% immediately after upgrading to 11a version

Post by omegagx »

Just asked for the escalation - hopefully it's in your queue now.
wishr
Veteran
Posts: 3077
Liked: 455 times
Joined: Aug 07, 2018 3:11 pm
Full Name: Fedor Maslov
Contact:

Re: SQL usage goes to 100% immediately after upgrading to 11a version

Post by wishr »

Hi Mike,

As far as I see the case has been escalated and the support team is waiting for additional logs from you.

Thanks
omegagx
Enthusiast
Posts: 68
Liked: 4 times
Joined: May 09, 2017 6:33 pm
Full Name: Mike G
Contact:

Re: SQL usage goes to 100% immediately after upgrading to 11a version

Post by omegagx »

logs uploaded
MFranzen
Lurker
Posts: 1
Liked: never
Joined: Nov 03, 2021 7:20 am
Contact:

Re: SQL usage goes to 100% immediately after upgrading to 11a version

Post by MFranzen »

Hi,

we got the same issue, but we dont use Veeam One that much.
Stopping and disabling the Service "Veeam One Reporting Service" immediately stopped
the hight CPU usage.

It's not a permanent or perfect solution but it solves the bigger problem for now.
Vitaliy S.
VP, Product Management
Posts: 27377
Liked: 2800 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: SQL usage goes to 100% immediately after upgrading to 11a version

Post by Vitaliy S. »

Hi MFranzen,

Thanks for your feedback, but if you want to resolve it, please open a support case as additional details might be required for review.

Thanks!
omegagx
Enthusiast
Posts: 68
Liked: 4 times
Joined: May 09, 2017 6:33 pm
Full Name: Mike G
Contact:

Re: SQL usage goes to 100% immediately after upgrading to 11a version

Post by omegagx » 1 person likes this post

We did open case 05078863 but haven't had any progress at all. I've been asking for updates for a week already.
Vitaliy S.
VP, Product Management
Posts: 27377
Liked: 2800 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: SQL usage goes to 100% immediately after upgrading to 11a version

Post by Vitaliy S. »

I see that your case has been escalated to a higher-tier engineer. Let's see what he can find out.
omegagx
Enthusiast
Posts: 68
Liked: 4 times
Joined: May 09, 2017 6:33 pm
Full Name: Mike G
Contact:

Re: SQL usage goes to 100% immediately after upgrading to 11a version

Post by omegagx »

so far they weren't able to find out anything and it's been 3 weeks.
wishr
Veteran
Posts: 3077
Liked: 455 times
Joined: Aug 07, 2018 3:11 pm
Full Name: Fedor Maslov
Contact:

Re: SQL usage goes to 100% immediately after upgrading to 11a version

Post by wishr »

Hi Mike,

As far as I see from the case notes our tech. support engineers are analyzing the data you provided. Stay tuned for updates.

Thanks
omegagx
Enthusiast
Posts: 68
Liked: 4 times
Joined: May 09, 2017 6:33 pm
Full Name: Mike G
Contact:

Re: SQL usage goes to 100% immediately after upgrading to 11a version

Post by omegagx »

How long do you think it will take to get a fix? Any idea why this wasn't caught in QA?
wishr
Veteran
Posts: 3077
Liked: 455 times
Joined: Aug 07, 2018 3:11 pm
Full Name: Fedor Maslov
Contact:

Re: SQL usage goes to 100% immediately after upgrading to 11a version

Post by wishr » 1 person likes this post

Hi Mike,

The fix is ready and currently being verified by our QA team. The engineer assigned to the case will let you know once it's ready to be distributed. We are also planning to include this issue in the Top Issues Tracker forum post once get all the necessary information.

I cannot say yet why the issue was not discovered during our regular QC cycle as the data about it is still being formalized, but we'll make sure to get to the bottom of that.

Thanks
stfconsulting
Service Provider
Posts: 33
Liked: 12 times
Joined: Jan 31, 2015 9:17 pm
Full Name: S Furman
Contact:

Re: SQL usage goes to 100% immediately after upgrading to 11a version

Post by stfconsulting »

I am going through this right now. Case 05233717 When you stop the reporting service it calms down immediately. I provided the most expensive queries to support and all the logs.
wishr
Veteran
Posts: 3077
Liked: 455 times
Joined: Aug 07, 2018 3:11 pm
Full Name: Fedor Maslov
Contact:

Re: SQL usage goes to 100% immediately after upgrading to 11a version

Post by wishr »

Hi Stfconsulting,

It sounds like issue #4 from the aforementioned list of known issues. You can get a hotfix for it through our support team by referencing the specified issue ID.

Upd: I've pinged our support management and made sure the engineers are proposing the correct fix.

Thanks
dali@iae.nl
Enthusiast
Posts: 82
Liked: 21 times
Joined: Jan 17, 2022 10:31 am
Full Name: Da Li
Contact:

Re: SQL usage goes to 100% immediately after upgrading to 11a version

Post by dali@iae.nl »

Same high CPU issue for us when Veeam ONE Reporting Service is running, Case #05771747
jorgedlcruz
Veeam Software
Posts: 1494
Liked: 655 times
Joined: Jul 17, 2015 6:54 pm
Full Name: Jorge de la Cruz
Contact:

Re: SQL usage goes to 100% immediately after upgrading to 11a version

Post by jorgedlcruz »

Hello Da Li,
I can see that our Support Engineers are actively sending you a few scripts, and things to try. Please keep working with them wherever possible.

I have added myself to the ticket so any update, I will be informed.

Thanks a lot!
Jorge de la Cruz
Senior Product Manager | Veeam ONE @ Veeam Software

@jorgedlcruz
https://www.jorgedelacruz.es / https://jorgedelacruz.uk
vExpert 2014-2024 / InfluxAce / Grafana Champion
dali@iae.nl
Enthusiast
Posts: 82
Liked: 21 times
Joined: Jan 17, 2022 10:31 am
Full Name: Da Li
Contact:

Re: SQL usage goes to 100% immediately after upgrading to 11a version

Post by dali@iae.nl » 1 person likes this post

Today had a Webex with Veeam on Case #05771747 to analyze Veeam ONE database server 100% CPU.
Got the Fix_330950_4004cac594.zip and added a registry ParallelWidgetDataDbRequestCount. After that CPU Usage went from 100% to below 10%. Thanks Veeam Support!
Post Reply

Who is online

Users browsing this forum: No registered users and 6 guests