100% RAM when Backup with Agent

[GianlucaCroci]

We are using a 9.5U4b on Windows Server 2016 standard, but tested on v10 on Windows Server 2019 too.

I want to save a 4 File Server Cluster via Agent because the File Server disks are ISCSI and aren't part of the VM (VMDK).

I installed the Agents and created a Job.
Backing up everything is fine in the first hours, but then the RAM of the Proxy server starts to rise up to 100% with the consequent blocking of the Server.

Looking with the "Resource Manager" the VeeamAgent occupies a lot.
But although the memory is 100%, the physical memory used was 5%.

I've already opened a case # 03997571.


As said I also tried with the v10. The server is a Windows Server 2019 and has 20 CPUs, 20 GB of RAM and 26 GB of PageFile.

[PetrM]

Hi Gianluca,

I'd suggest to check storage optimization settings and make sure that 4 Mb block size is selected - Local target (large blocks).
It's recommended for backup files that are larger than 16 Tb. Veeam will produce large deduplication metadata table if you use small blocks for a large backup file.

It might be a reason of the issue with RAM on a repository when backup file is opened on read/write operations.

Thanks!

[GianlucaCroci]

Thanks a lot Petr.

After I used "Compression level" = "Dedupe-friendly" and "Storage optimization" = "Local target". Now I've modified the Storage with "Local target (large blocks)". I'll try only with this...

The block size on the Storage we can't change at the moment.

THANKS

[PetrM]

By the way, you must create an active full backup to apply new storage optimization settings, please make sure you have enough free space on the backup repository.

Thanks!

[GianlucaCroci]

yes, of course. I've deleted the previous backup, because uncompleted, and restart a FULL Backup.
At the moment all well, but last time the first 2.5H was good. Connected again after 7H it was blocked why the RAM completely used.

Thanks.

[GianlucaCroci]

Yesterday done a new Test.
The last time I looked at yesterday 6: 30H had passed and everything was going well (RAM used 8.7 GB). He had previously saved 3 TB of data. The servers had been saved from 43 to 63% (4 servers in different percentage).
This morning after 17H he had saved 4.2 TB and the RAM was 100%.

Here the print screen of yesterday evening and that of this morning.
YESTERDAY EVENING https://drive.google.com/file/d/1fGPAKb ... sp=sharing
THIS MORNING https://drive.google.com/file/d/1Oo7jOf ... sp=sharing

On the Backup Job, since Yesterday the Storage is configured as:
Compression level: Dedupe-friendly
Storage optimization: Local target (large blocks)

Thanks

[PetrM]

Hi Gianluca,

My last idea is to decrease the number of parallel activities going to the repository, for example you can set 2 concurrent tasks in the repository options.

Anyway, I'd suggest to let our support team to figure out the root cause, looks like deep technical analysis is required.

Thanks!

[GianlucaCroci]

Hi Petr,
thanks again for your reply.

I changed it as you suggested, reducing the number of "Concurrent Tasks" on the Repository to "2".

I guess it's not a simple thing. In fact, the Case al Support has been open for 3 weeks, and it's a long time with you, because normally I get a solution in a shorter time. But I understand that it is not easy.

Thanks again
Kind regards

[PetrM]

You are welcome!

That's true, this one is sophisticated, there are many different factors which can make the issue to appear.
Let's wait for conclusion of our support team.

Thanks!

[GianlucaCroci]

with "Concurrent Tasks" set at 2, the problem not solved.

Here a PDF with few Print Screen about the job of Yesterday.
https://drive.google.com/file/d/1sp2jhq ... sp=sharing

Kind regards

[PetrM]

Hi Gianluca,

Thanks for information, please continue working with our support team to find the solution.

Thanks!

[GianlucaCroci]

Hi Petr,

thanks to you too for your help.

Kind regards

[GianlucaCroci]

To solve the problem it was enough to add two entries in the registry, and restart the Veeam server.

Before inserting the two keys, it's better to speak with Veeam support.

HKEY_LOCAL_MACHINE\SOFTWARE\Veeam\Veeam Backup and Replication

1) FileCacheLimitPercent
Type: REG_DWORD
Default value: 25 set to a value lower than 25, any, starting from a small modification

2) AgentReadOnlyCache
Type: REG_DWORD
Default value: 2 set to 0

I hope it will help

[rm@kompetera.dk]

Hi
We have a similar issue with a SQL server where the Veeam Agent is maxing Ram out so the server is not working and as soon as we restart the Veeam agent service the ram usage goes down from 100% and the server begins to work again. We see this issue with an interval of a few weeks on different servers and as soon as we restart the Veeam agent service it is solved.

So just wondering if this solved your problem and also where the registry change should be made because it looks like you do it on the Veeam server and then restart it but does that have effekt on the servers where the Veeam agent is installed where the problem actually is?

best regards
Rune

[Dima P.]

Hello Rune,

You can raise a support case, we already have several workaround at the support side for the idle state CPU usage issue. The issue is currently being investigated by RnD team, so additional troubleshooting information will help us as well. Please share the support case once you have it. Cheers!

[rm@kompetera.dk]

Hi Dima
This is my case and I already got help and have rebuilded the database on the server with the problem as there was error logs present and then time will tell I guess if it worked or not as we don't see it every week but we have the problem on a handfull of servers.

This is the response I got from Support and I have only made the first Part as there where error logs present.

First of all I would like you to check the folder bellow:
C:WindowsSystem32configsystemprofileAppDataLocalMicrosoftMicrosoft SQL Server Local DBInstancesVeeamEndPoint

Please let me know if there are any Error log files in there. In case there are, it may be helpful to rebuild the Veeam Agent database as this would fix a Corrupted Database (but you may need to reconfigure the Veeam Agent backup job):
https://www.veeam.com/kb2335

In case there are no error files, you may limit the occurrence of this behavior by changing the SQL server max server memory allocation. This can be achieved in the following way:
1. Download and install Microsoft SQL Server 2012 Native Client and Microsoft SQL Server 2012 Command Line Utilites from the official site.

Expand Install Instructions and find there Microsoft® SQL Server® 2012 Native Client and Microsoft SQL Server 2012 Command Line Utilites and choose x86 or x64 correspondingly;

2. Make sure that the Veeam Agent for Microsoft Windows service is running and after that open up the command prompt admin and type 'sqlcmd -S SqlInstancePipeName –U VeeamLogin –P SqlPassword'.

The cursor will become 1>

SqlInstancePipeName and SqlPassword values might be found in HKEY_LOCAL_MACHINESOFTWAREVeeamVeeam Endpoint Backup

3. next print the commands below and press enter after each string:
sp_configure ‘show advanced options’,1;
GO
RECONFIGURE;
GO
sp_configure ‘max server memory’,400;
GO
RECONFIGURE;
GO
Exit

4. Restart Veeam Agent for Microsoft Windows service and verify the results. (it may simply just be better to restart the entire machine)

Hope it can help others as well.
Best regards
Rune

[Dima P.]

Thanks for sharing the update Rune! We are checking the mentioned solution with QA folks (and few additional tweaks for SQL), the plan is to make the adjustments in the upcoming major version of Windows agent. Cheers!

[ChristineAlexa]

Here's how I set it (I wrap it in an invoke command against many machines at once) - Powershell
$parent = Get-ItemProperty "HKLM:\SOFTWARE\Veeam\Veeam EndPoint Backup\DbConfiguration"

sqlcmd -S $parent.SqlInstancePipeName -U $parent.SqlLogin -P $parent.SqlPassword -Q "sp_configure 'show advanced options',1 ;
reconfigure with override;
go
sp_configure 'max server memory',400;
reconfigure with override;
go
sp_configure 'show advanced options',0 ;
reconfigure with override;
go"

Hope that helps for people trying to figure out how to automate it. This assume the machine has sql server tools installs of some form (sqlcmd) installed and available in the path.