VBR RAM for 2PB SOBR

[matthewingram]

We deployed Veeam a few months ago. This week backups jobs started randomly failing. It required a resyc of the repo. One of the repo's is 2.2PB with about 1.7PB used. As soon as the resync started one of the Veeam backup manager process consumed all 20GB of RAM and filled the page file on the VBR. Anyone else running 1PB+ SOBR's have any insight on sizing the VBR? Its the VBR only, SQL is external as is all other functions. I can't find any guidance other than 4GB +500MB per active job. Support basically said just throw more resources at it without any real guidance.

[nmdange]

The recommendation I've seen is 1GB of RAM for every 1TB of disk space, though I don't think that applies at the high end. If you really only have 20gb allocated, I would go to 128gb minimum. That's what I have on my ReFS repo which is only a couple hundred TB in size.

[DaveWatkins]

The recommendation I've seen is 1GB of RAM for every 1TB of disk space

That's a ReFS guideline and I wouldn't expect that to be true for NTFS at all. It's not mentioned what he FS is though so who knows. 20GB of RAM does seem fairly low for that total disk size. I've got 96GB happily running ~200TB of ReFS volumes. It crashed hard with only 32GB but that was back when ReFS was still horribly unstable

[jmmarton]

For ReFS, the 1 GB RAM per 1 TB of storage becomes less linear once you hit around 128 TB of storage, and at 200 TB of storage the RAM requirement typically tops out at 192-256 GB RAM.

So for a 2 PB repo, 20 GB of RAM is sorely undersized. I would up that to at least 192 GB RAM. Again this is assuming it's ReFS.

Joe

[matthewingram]

I probably want clear. The ask is for the VBR server not the proxy/repo. The proxy/repo has 1TB of RAM. The VBR server is separate VM. It’s just a management server.

[tsightler]

The amount of data on the repo shouldn't matter much for the VBR server, however, with that amount of data, I'm assuming you may have quite a few jobs. Can you tell us a little more about the setup, how many backup jobs+backup copy jobs, etc. I've never seen a single backup manager process use so much RAM so that's my first concern. Do you know what job that specific backup manager was associated with (you can determine this by looking at GUID part of the process string and matching it to the job GUID).

[Gostev]

Yep, this behavior is definitely not normal: normally, backup manager process should consume less than 1GB of RAM. Perhaps you're running in some bug, or there's a 3rd party software conflict (like antivirus with real-time protection). Please open a support case, upload the debug logs package and let us know the case number. Thanks!

[matthewingram]

The case # was 03524751. IT was a part of that case. What I noticed was the memoery was being consumed after a manual rescan of a 2.4P B SOBR was initiated. There was about 1.8PB used. With 22GB the VBR server ran out of physical memory and filled the page file. The system went non responsive. I moved the memory up incrementally until I hit 64GB. At that point the veeam backup manager process peaked at 40GB used. Most of the memory was released about 2 hours later when the rescan was complete. Normal backuo operations I see about 20GB peak used, but now I have to keep 64GB on the VBR server encase a rescan is needed.