Memory leak in Veeam agent?

[danaos]

I've reported this to support as case 04671660, however as we are using the free agent rather than licensed (we have a support contract for Backup&Replication, but don't need the licensed features of the agent at present) I'm not expecting a response, but wanted to report it in case it's a bug in Veeam rather than something related to our network setup.

We've noticed on some Windows 10 (mostly 1903 and 1909), and Windows Server (2012R2 and 2016) where we are doing agent based backups to a B&R repository (these are mostly physical systems, not virtual machines), that on some of them where they are not rebooted for long periods (a few months, so making it hard to replicate the issue!) that Windows become sluggish, and in a couple of cases has become so slow that we had to cold boot the systems to even log in to them as nothing would respond. Comparing these machines we found one thing in common - the non-paged pool in Windows becomes so large that there's very little memory available, not enough for the system to operate correctly. Using task manager, RAMMap, poolmon, and performance monitor we've narrowed the source of this non-paged pool usage to the Veeam.Endpoint.Service.exe process - and more specifically filtmgr.sys.

I've tried adding the Veeam application folder to the AV exclusion list (most systems are running Trend WFBS, but some are just using Windows Defender as built in to Windows as they are development test systems), but the RAM keeps on growing. If I stop and start the Veeam Agent for Microsoft Windows service, all the non-paged pool memory is released and the system is back to running normally. I'm a software developer but not an expert by any means, but this looks like a leak caused by not releasing resources correctly.

I've got performance monitor running on one of the system logging data about the Veeam process long term, sampling every 1800 seconds and set up to run over 14 days, so far I'm 24 hours in and the non-paged pool line on the graph is a consistent linear growth. As this server is doing a backup once a night I'd expect there to be a spike at that point, but if Veeam is constantly monitoring the changes on the disk I guess that this might be the reason why it's linear? In the past 18 hours this particular Windows 2016 server (it's actually a virtual machine on Hyper-V 2016, but not covered by our B&R VM license so I'm using VAW to do a nightly backup as it's not a critical server, just for soak testing software in our development environment) has seen the non-paged pool RAM in Veeam grow from 291k to 419k - it might not sound a lot, but after a month or two that'll be causing pressure, and a month or so later it'll be cold reboot time.

Most of our Windows laptops and desktops are unaffected because we force a reboot once a week on those used by employees. And the virtual machines backed up direct from B&R on our Hyper-V 2016 system don't have an issue either, as they're not using VAW.

Am I the only one seeing this issue? Or is it just that most Veeam users are using B&R jobs to backup up virtual machines for servers that are running for long periods and nobody is using VAW in the way I am?

Regards,

Dan

[Dima P.]

Hello Dan,

Thanks for the detailed description of your issue. The only similar issue I am aware of - localbd process memory consumption in idle state. I've asked support team to review the logs and start the investigation. Cheers!

[danaos]

Hi Dima,

If you need anything else let me know. I've got a performance monitor chart adding points every 30 minutes, and while it's not showing much as it's been running just under 24 hours there's a very clear linear growth of Pool Nonpaged Bytes, but nothing else I'm tracking is growing - handles, threads, pool paged bytes, working set, and virtual bytes all look flat is, when looking at line of best fit.

[wong1tal]

Joined to say I had this happen to me today on two of my servers that are running Server 2012 R2, so your're not the only one. Non-paged pool usage was high in RAMMap and poolmon showed drivers using the tags Irp and FMic with the highest usage. After some digging narrowed it down to Veeam endpoint service. Just as danaos, stopping the Veeam service frees the memory.

[Vitaliy S.]

wong1tal, can you please contact our support team and share your findings with them? That would help our RnD team a lot.

[danaos]

Just wanted to update that I've had a reply from Veeam support, apparently this is a known issue in version 4 and has been fixed in version 5. I can't confirm this is the case yet as I now need to plan an update to B&R 11 as VAW 5 does not support B&R 10, and will have to roll this out on our backup infrastructure and test it first. For now I'm setting up a scheduled job on all of the servers and workstations that do not get rebooted often to restart the Veeam endpoint service on a Sunday.

[sanch]

Hi guys, just wanted to give a feedback that I faced the same issue. Windows Server 2016 VM with 8 GB of RAM, 6 GB in non-pagined pool. System nearly unusable. Found with poolmon the tags FMic and IRP mainly consuming bytes. After browsing the internet for a moment I found this thread and I tried to restart all the Veeam services and TADAA, like magic, my RAM went down to 2.4 GB Thanks guys !

[Dima P.]

Hello sanch,

Thank you for updating this thread! Mind me asking what agent version you are currently running?