Veeam.Archiver.Proxy.exe high RAM usage

[HDClown]

v4.0.0.2459

As of this post, I have a job that has been running 8 hours and transferred 221 GB. The job will probably run another 8 hours until it finishes baed current object counts.

Veeam.Archiver.Proxy.exe is currently using 12.5GB of RAM. The server itself has 18GB of RAM total. Is this amount of utilization normal?

[Mike Resseler]

Hi @HDClown

It is certainly not abnormal that it uses that high amount of RAM. Especially during large jobs such as this one. Is this a full backup? Or did you add a lot of new resources?

[HDClown]

This is a combination of adding SharePoint/OneDrive to a prior e-mail only backup, and there also being 10 days since the last e-mail only back ran (I was moving storage and to a new server). There's a lot of new data in the set from importing some large PST's and moving a bunch of users to OneDrive for their user-level data.

I never paid any attention to RAM utilization in general when I was running this as an email only job, because the server I was running VB365 on also ran VBR and VMware vCenter 5.5 on Server 2008 R2. I'm on a dedicated 2019 host now that only runs VBR v10 P1 and VBO365 4.0.0.2549. I've not seen where the RAM utiliation has caused any kind of performance impact on my VBR jobs. I have no way to relate the RAM use to prior server on the VB0365 side given it now has to backup a lot more objects, and these initial jobs are moving a lot more data.

The job that was running when I made this post when into a retry due to 6 objects having failed. It's completed 6 of 9 of those objects in past 5 hrs with 97GB backed up. Veeam.Archiver.Proxy.exe RAM utilization is currently at 11GB, so it went down slightly.

[HDClown]

@Mike Resseler

After my job was finished, Veeam.Archiver.Proxy.exe did not release the 11GB of memory it was consuming, even 13 hours after the job had stopped.

I rebooted the server and started the job again. I've been monitoring RAM utilization of Veeam.Archiver.Proxy.exe based on job runtime/objects/transferred and here are a few reference points.

- 3.5GB RAM utilized @ 7 min,40 of 180 objects, 800MB transferred
- 5.3GB RAM utilized @ 16 min, 93 of 516 objects, 2.0GB total transferred
- 5.3GB RAM utilized @ 30 min, 158 of 1031 objects, 4.0GB total transferred
- 6.8GB RAM utilized @ 60 min, 328 of 1031 objects, 8.2GB total transferred
- 8.1GB RAM utilized @ 90 min, 490 of 1031 objects, 12.5GB total transferred
- 8.8GB RAM utilized @ 120 min, 697 of 1031 objects, 15.7GB total transferred
- 9.2GB RAM utilizied @ 180 min, 1025 of 1031 objects, 16.4GB total transferred

At this point I was down to 2 actively processing objects, 1 failed with warning, and 3 failed with 401 Unauthorized. The 2 active jobs seemed to stall for about hours but then picked up again.

At the 7 3/4 hr mark, RAM utilization was up to 11GB with 32GB as total transferred and the job finished. The job went into retry and the few objects that failed in initial run all completed successfully

While job was running CPU utilization on Veeam.Archiver.Proxy.exe bounces around with every update. Lowest I've observed is 10%, highest 54%.

With a job 100% completed successfully, Veeam.Archiver.Proxy.exe is now at 10.6GB RAM utilization and it hasn't released it. Not releasing this memory is my primary concern now. I can understand the utilization during the job, but why is it keeping that memory locked up?

[Polina]

HDClown,

To investigate the issue, we need to study your logs. Please open a support case and share its ID here for a followup.

Thanks!

[Mike Resseler]

@HDClown

Not releasing the memory is not normal. Using it is OK . So please follow Polina's advice.

PS: Those 3 objects with 401 authorized. I assume they are either SharePoint objects or OneDrive 4 Business files. Can you check those in O365 itself? 90% or more of the time it means they are seen as infected by the O365 service, and they are locked so we cannot download them. So they might be malware or seen as it.

[HDClown]

Case # 04147290 has been created.

The 401 authorized failures were backed up successfully on the next job. I attributed most of the errors I received over past 2 days to be tied to how long backups were taking and a lot of objects being first time backup. I had 2 jobs complete yesterday on the first attempt with zero object issues. Neither of those jobs were from a full 24 hr cycle, but I expect the job that runs tonight to run without issue.

[Matt.Sharpe]

We're seeing similar issues in relation to RAM usage being high and not being released. I recently saw RAM utilization climb to 10's of GB when no jobs were running.

Can anyone confirm what exactly is stored in memory Veeam 365 wise, does it still actively store/access it when no restore/backup jobs are running ?

[HDClown]

In v4, the default configuration is to allocate up to 50% of the available RAM for JET cache, which is documented here: https://vbo.veeambp.com/guide/design/ma ... #fnref:2:1

You can set a hard limit on the cache use by modifying the following in C:\ProgramData\Veeam\Backup365\Proxy.xml

Code: Select all

<Veeam>
  <Archiver>
    <Storage>
        <Params MaxCacheLimit = "4096" />
    </Storage>
  </Archiver>
</Veeam>

The MaxCacheLimit is a value in Megabytes, and support recommended 4096 as the minimum. Restart the Proxy service after making this change.

I tried to get an explanation from support as to why the memory needs to stay utilized after a job finishes. I did numerous tests where I restarted the Proxy service between jobs and found zero performance benefit for a job run when the memory was held in allocation vs. release from the service restart. I did not try to run any restore performance comparisons.

In the case of my server, I did find that the Proxy service never went above 50% of the total RAM of the server. As my server has 24GB of RAM, the Proxy service using 12GB isn't a big deal. The server also runs B&R v10 but that is not particularly RAM hungry either, so I always had plenty of RAM headroom, so I opted to not set a MaxCacheLimit.

However, I did set a schedule task to restart the Proxy service every day before the bulk of my backup jobs run. I did this as an extra precaution to make sure I didn't run into a RAM contention issue during my backup windows. I also found that this resolved random JET I/O errors (causing job warnings and failures) I was getting, which I asked support about as well. I am using an SMB repository and support said the errors indicated the repository was unavailable when jobs ran but I knew that was not the case. When I found restarting the Proxy service pretty much eliminated the I/O errors, I decided to not bother investigating any further with support. That being said, it sure seems to me like there may be other JET related issues that can/do occur because of the Proxy service holding memory.

[pesos]

Hi, I saw the archiver was taking up 144gb of ram on our 256gb proxy/host. I set the value mentioned above to 32768 and restarted the proxy. This morning I see it has crept up to almost 60gb again.

Is this value not respected in newer versions? How can I prevent the mem usage from getting out of control?

Thank you!

[infused]

Any answer to this for v5?

[Polina]

There are multiple optimizations in v5, but the above-mentioned requirement for Jet cache is still relevant.
If you see some specific issues which potentially could be caused by memory utilization, we'd certainly want to investigate those.