LocalDB constantly using CPU ressources(15-80%)

[hoFFy]

My LocalDB installation (which came with Veeam Endpoint) under Win8.1 is constantly using some CPU power... starting from 15% up to 80% of my 2-core PC.
I'm wondering if someone else made this experience!?

[Gostev]

This has not been reported before. We would definitely want to investigate this while you are having this issue, because this is very wrong considering absolute minimal usage of the configuration database on our end. But there is always a possibility that some newly added code polls some table constantly, or some other bug like this. Thanks for catching this!

[Dima P.]

Sebastian,
In addition, could you share the application logs by creating the support case and let me know the case ID, please.

[hoFFy]

Your support case number is 00830485

[Dima P.]

Thanks Sebastian!

I’ve shared your issue and case with QA team. Let’s see what they will find.

[jeddytier4]

Sorry to revive an old thread, but I'm having this same issue. On a server 2012 R2 box, the LocalDB instance is constantly using 10-11% of a 3.2 Ghz AMD Opteron. The server in question runs very high cpu and disk io for other processes, and we definitely need the extra 10% when not running backups.

[Dima P.]

Hi jeddytier4.

Please kill Veeam Agent for Windows process, tray app, stop agent service and check the resource consumption from a pure SQL instance.

[jeddytier4]

What I have done for now, is stop the service (this reduces the usage to 0%) and I have a script that starts it 10 minutes before the backup schedule and stops it afterwards.

I did also try a rebuild of the localdb database using https://tinkertry.com/veeam-endpoint-ba ... base-reset, it did not resolve the issue after the rebuild.

[dkleeman]

I know this is old, but I believe that I have exactly this problem on a recent Windows 10 1903 with Veeam Endpoint installed. When the problem arises started Veeam Endpoint does not work and I am referred to the KB article https://www.veeam.com/kb1959. Veeam version is 3.0.1.1039 Free Edition.

The command that is creating the high CPU is:

Code: Select all

"C:\Program Files\Microsoft SQL Server\110\LocalDB\Binn\\sqlservr.exe"   -c -SMSSQL11E.LOCALDB -sLOCALDB#71XXXXXX -d"C:\Windows\System32\config\systemprofile\AppData\Local\Microsoft\Microsoft SQL Server Local DB\Instances\VeeamEndPoint\master.mdf" -l"C:\Windows\System32\config\systemprofile\AppData\Local\Microsoft\Microsoft SQL Server Local DB\Instances\VeeamEndPoint\mastlog.ldf" -e"C:\Windows\System32\config\systemprofile\AppData\Local\Microsoft\Microsoft SQL Server Local DB\Instances\VeeamEndPoint\error.log"

The files in the folder referenced are only accessible to the I don't believe that this process, which is not run as a service, will ever complete without a reboot. It consumes between 50% and 100% CPU.

The error.log referenced contains repeated entries like these two:

Code: Select all

2019-08-04 17:23:52.10 Server      Resource Monitor (0x1ef4) Worker 0x000000044E684160 appears to be non-yielding on Node 0. Memory freed: 0 KB. Last wait: MISCELLANEOUS. Last clerk: type MEMORYCLERK_SOSNODE, name SOS_Node. Approx CPU Used: kernel 61593 ms, user 191840843 ms, Interval: 1626768867.
2019-08-04 17:23:57.20 Server      Process 0:0:0 (0x3a0) Worker 0x00000004431CE160 appears to be non-yielding on Scheduler 6. Thread creation time: 13207739674218. Approx Thread CPU Used: kernel 60171 ms, user 191375937 ms. Process Utilization 14%. System Idle 83%. Interval: 1625930696 ms.

[pufferdude]

I don't know if it's the same issue as immediately above, but I installed Veeam Agent for Windows a week ago on Win10 1903 and now my machine runs far more sluggishly, and when I look at Task Manager "SQL Server Windows NT - 64bit" is almost always using CPU... anywhere from 10% to 60%. I only have the agent set to back up ONCE a day at noon and it completes in minutes, so I have no idea why SQL Server is constantly trying to "do something" on my machine.

Getting ready to uninstall it :-/

[Dima P.]

Hello Jim,

Please raise a support case as we are investigating this issue with RnD folks and your logs will help us to identify the root cause. Thank you in advance!

[dkleeman]

I don't know if it's the same issue as immediately above

I used Process Explorer from Sysinternals to find out more about what had launched the process. See https://docs.microsoft.com/en-us/sysint ... s-explorer

[cathym]

What was the resolution of this issue? I have experienced the same on Windows 10 running VAW and am in the process of a complete uninstall/reinstall to see if it continues to reoccur.

[Dima P.]

Hello Cathy,

We need a support case to investigate the root cause of this behavior. Can you please raise a support ticket and then share the case with us? Thanks!

[einhirn]

This is a thread where someone posted a workaround for high CPU load caused by SQL-Server process:

veeam-agent-for-windows-f33/sql-server- ... 19-30.html

[Dima P.]

Thank you for sharing. This issue is being investigated and we've made some progress with optimizing the SQL behavior. The plan is to include all the necessary tuning in the upcoming major version after it's signed by QA folks. Cheers!

[MaximillianC]

Hello,

Unfortunately, it looks like this issue has resurfaced, years after this thread was opened. My last Veeam backup finished successfully 5 hours ago, but "SQL Server Windows NT - 64 Bit" (Local DB) is sitting around 15% constantly, until I stop the "Veeam Agent for Microsoft Windows" service... in which case, after the service finishes stopping, CPU usage on the SQL process drops. As soon as I start the service again, the SQL CPU usage jumps back up. I have an Intel Core i9-9900K CPU, with 64GB of RAM, so no dearth of CPU power or RAM. I didn't notice this until recently, but I haven't been paying a lot of attention to CPU hogs since upgrading to this more-powerful CPU a few months ago. I haven't upgraded my Veeam Agent in a while, but it says it's up to date, as of 23 hours ago. I'm on Veeam Agent for Microsoft Windows 4.0.1.2169 - Free Edition.

I used the info in the other thread above (posted by einhirn) to connect to my Veeam DB through SSMS. I used SQL Profiler to find out what SQL commands it was running, and it seems to be running certain stored procedures repeatedly, which have a CPU cost of about 1100 each time they're run (according to SQL Profiler). The main CPU-costly command/SP I see running over and over (multiple times a second) is:

exec ReportEpTaskSession @top_count=7,@job_id='FBCE0F42-7BCA-49A2-8650-CDF9BF29FD46',@usn=1366206

The SP is just a single SQL query. When I manually executed it, I could see it takes almost 2 seconds to execute, so it is fairly costly. Looking at the SQL execution plan, the CPU-costly portions of the query are the two XML .query functions being run in the SELECT statement:

isnull(s.[bJobAdvanced].query('/stat/RepoWaitStartTime').value('.','bigint'), 0) as [start_wait_repo],
isnull(s.[bJobAdvanced].query('/stat/RepoWaitEndTime').value('.','bigint'), 0) as [end_wait_repo],

Veeam team, can you please escalate this issue to your dev/QA team, and find out whether or not:

a) there's a more-efficient way to get the start_wait_repo and end_wait_repo values in this query, and
b) that SP is supposed to be running repeatedly, multiple times per second, even when there's no backup job running?

XML queries in SQL in general tend to be fairly costly CPU-wise, so I can't imagine they'd be deliberately running a SP that calls two XML queries this frequently. Hopefully the info I provided above is helpful in getting this issue resolved quickly.

Regards,
Maximillian R. Carper

[Gostev]

Hi Maximillian, can you upgrade to the latest version 5 and see if it fixes the issue? You can download if from our web-site. Thanks!

[MaximillianC]

Thanks for the quick reply! I will do that now, and write back.

BTW, I submitted Case #04731203 for this issue, as well. So I take it the automatic update checking doesn't work for major versions, since it is saying my version is up-to-date? I hadn't even checked, as I was trusting the app to tell me if there was a newer version. Does the update checking work for major versions with this new version?

[Mildur]

It's possible, that the update server are not updated for the new Major version until Veeam decides to do that.
I think, they wait some weeks for that, until the versions has proven itself in many production environments and no software breaking bug has appeared

[MaximillianC]

Doh! I wish the v4.0 auto update checking would've worked, or I would've just double-checked for a newer version. I just spent a fair amount of time troubleshooting that issue, when I could've just upgraded to the new version, which doesn't seem to have that problem! (headbonk) Thanks, Gostev, for responding so quickly, and letting me know!

[lucius_the]

Dear Gostev,
I've been having these issues too, they surface every now and then on some of our servers. Didn't report here before, as others have already done that. Tried most suggestions and it did help a bit on some servers, but on other server, every now and then I get one of them using resources like crazy and the machine hogs, needs a restart of veeam service to get back to normal - sometimes even a complete machine reboot is needed, if it becomes too unresponsive because I didn't act on system monitoring alerts quickly enough. The culprit is always the LocalDB SQL instance that Veeam agent service uses. On some servers it doesn't happen, on others it's a regular thing, especially newer Windows Server OS-es.

I would like to try and suggest something, as a professional developer, if I may. While I understand these problems are SQL Server related and acknowledge it's not Veeams' fault that LocalDB behaves the way it does, Veeam still gets to choose what technology to use in a product. Now, thinking about what Veeam Agent for Windows needs from a database system (I'd say - not too much) and knowing LocalDB is used just as an in-process database for the Veeam agent service in this case, makes me wonder why you didn't choose something like sqlite instead. Or firebird embedded, or whatever, something that is better suited for such the needs of the agent. Using SQL Server Express LocalDB here looks... well, at the very least it's an overkill. It's like using a rocket to kill a flee. In spite of what Microsoft likes us all to believe, that it's well suited as an in-process database, SQL Server LocalDB is really a behemoth. It's not at all suited as an embedded/in-process database. They had this hole in their offering so they put SQL server engine in the place and named it differently and that's about it. I'm not saying it's bad - for a particularly large and/or complex database, yes it may well behave better then something else in it's place. But for a small in-process database, something that Veeam agent would need just to keep data about it's backup schedules and logs, it's quite an overkill. And also a resource hog.

So, instead of these workarounds, it would be really nice if you consider swapping SQL Server Express LocalDB with something more appropriate for the job, like sqlite. The end result I believe would make everybody happy. CPU and RAM usage should drop significantly. Which would make Veeam agent even more awesome than it already is - and lot more friendly to the system it runs on. I found sqlite to be very robust, that would be my first choice so I would recommend that specifically. I use it on many important production systems, about a thousand machines more for than a year now, have yet to see one crashed/corrupted database. Zero issues. It really is well designed. Also has a very free licence enabling you to use it anywhere (but also has other options if you need it). But honestly, any other purposely designed in-process DB would probably do the job better than LocalDB here.

So it's just a suggestion. Might not be that hard to switch, if you ask your developers to consider what it would actually take. It might look scary, maybe even crazy on first consideration, but in reality it's not such a big deal to migrate a relatively small and not hugely complex DB to different engine. Especially if your code is properly modeled and you have data access layer already in place (which I believe you have). This could in fact prove to be not too complex to realize, and would do a lot in terms of resource usage - plus it would definitely solve all the problems described above that some of us are experiencing.

If this was not Veeam I wouldn't waste my time even recommend this. But I hope that you will at least take it into consideration.

With kind regards,
David

P.S. I purposely didn't get into little details like SQL server has a VSS writer (means it's snapshot aware), knowing that could play a role in choosing it for Veeam agent - I leave this one to you to consider if you really need that functionality for the agent DB - or you could actually work around it and survive quite happily without that detail.

[lucius_the]

After posting and searched around a little more,
and found this: veeam-agent-for-windows-f33/veb-config- ... 33872.html
and this: veeam-agent-for-windows-f33/feature-req ... 41900.html
and above all THIS: https://helpcenter.veeam.com/docs/agent ... tml?ver=50

You're already using sqlite for the Linux version of the agent ! Well, great choice !
You didn't go with LocalDB on Linux now... there is LocalDB for Linux, AFAIK. How come ? might be developers on the other team are more though

Well. I'm hoping for a reconsideration on the Veeam agent for Windows, then

No, really, I understand if you don't do it right away, or ever, I am grateful for what I have. It'd be great if you do it at some point, though. It really bothers me to see what resources the agent wastes on machine just because LocalDB was the chosen way back then (ok valid reasons, but that's way back in time now) but even more so is that choice causes issues on some machines. So now that's you're already using sqlite, you're using it for the agent... c'mon, switch to it on windows as well. It's the proper way to go ! As you already know

Kind regards,
David

[Gostev]

Thanks David, yes I think you are right - it's probably the time for us to give up on LocalDB and switch to SQLite.

[lucius_the]

Thank you Gostev,
This is great ! What can I say. I wish more people like you existed in my line of business...

[Dotmax]

+1 for SQLite (and +1 for Gostev !!!)
I also have some machines with intermittent CPU spikes caused by SQL LocalDB.
Especially on <=4gb ram

MAX

[mts75]

Same here ... Esxi-Free-Server with multipe Windows Instances ... multiple Veeam-Agents installed ... All Instances together make the Server really slow all the time, not only when Backups are runnig!!

[arsagera]

Hi, this is great to hear about switching to sqlite. But how soon can we expect it? I just found out about the release of a new version 5.0, but it is of 24 february so I guess there is no new feature yet...

[Mildur]

Veeam doesn‘t gave you an ETA, until they are sure that it will be released on time

Veeam releases new version yearly. I expect it in Agent v6.0 or v7.0 (2022 or 2023).
It depends on what they have already planned for the next version and if they have free deployement ressources for this change. This is not something for a simple hotfix or patch, i think.

[Gerald]

Please get rid of the MSSQL server as soon as you can. Even when it works normally it consumes to much CPU:
For example on my machine it currently has consumed 6h25min of CPU time, but the machine has only been up 6 1/2 days.
If I divide the 6h25min CPU time by the total CPU time available (6 1/2 days * 24 hours * 12 threads = 1872h) this comes out to "only" 0,35% cpu load an average.
(such a CPU usage is normal for the sql server of a veeam agent installation, I've observed this on all systems, this is not specific to a problem or bug)

0,35% is not a huge amount - but for WHAT? sqlservr.exe has also read ~500MB and written 2.7GB to my SSD in there last 6 1/2 days. While doing nothing. Something is seriously wrong here.



Veeam Agent just sat in the tray idling, not a single backup was performed in those 6 1/2 days and yet mssql managed to "calculate" something for 6h25min of CPU time. Please try to get rid of this garbage as soon as you can. I for one welcome our new sqlite overlords.