Comprehensive data protection for all workloads
Post Reply
matthewingram
Influencer
Posts: 18
Liked: never
Joined: Oct 03, 2018 10:13 pm
Contact:

VBR RAM for 2PB SOBR

Post by matthewingram » Apr 26, 2019 5:45 pm

We deployed Veeam a few months ago. This week backups jobs started randomly failing. It required a resyc of the repo. One of the repo's is 2.2PB with about 1.7PB used. As soon as the resync started one of the Veeam backup manager process consumed all 20GB of RAM and filled the page file on the VBR. Anyone else running 1PB+ SOBR's have any insight on sizing the VBR? Its the VBR only, SQL is external as is all other functions. I can't find any guidance other than 4GB +500MB per active job. Support basically said just throw more resources at it without any real guidance.

nmdange
Expert
Posts: 479
Liked: 120 times
Joined: Aug 20, 2015 9:30 pm
Contact:

Re: VBR RAM for 2PB SOBR

Post by nmdange » Apr 26, 2019 9:46 pm

The recommendation I've seen is 1GB of RAM for every 1TB of disk space, though I don't think that applies at the high end. If you really only have 20gb allocated, I would go to 128gb minimum. That's what I have on my ReFS repo which is only a couple hundred TB in size.

DaveWatkins
Expert
Posts: 349
Liked: 94 times
Joined: Dec 13, 2015 11:33 pm
Contact:

Re: VBR RAM for 2PB SOBR

Post by DaveWatkins » Apr 26, 2019 9:52 pm

The recommendation I've seen is 1GB of RAM for every 1TB of disk space
That's a ReFS guideline and I wouldn't expect that to be true for NTFS at all. It's not mentioned what he FS is though so who knows. 20GB of RAM does seem fairly low for that total disk size. I've got 96GB happily running ~200TB of ReFS volumes. It crashed hard with only 32GB but that was back when ReFS was still horribly unstable

jmmarton
Veeam Software
Posts: 1767
Liked: 245 times
Joined: Nov 17, 2015 2:38 am
Full Name: Joe Marton
Location: Chicago, IL
Contact:

Re: VBR RAM for 2PB SOBR

Post by jmmarton » May 01, 2019 4:28 am

For ReFS, the 1 GB RAM per 1 TB of storage becomes less linear once you hit around 128 TB of storage, and at 200 TB of storage the RAM requirement typically tops out at 192-256 GB RAM.

So for a 2 PB repo, 20 GB of RAM is sorely undersized. I would up that to at least 192 GB RAM. Again this is assuming it's ReFS.

Joe

matthewingram
Influencer
Posts: 18
Liked: never
Joined: Oct 03, 2018 10:13 pm
Contact:

Re: VBR RAM for 2PB SOBR

Post by matthewingram » May 02, 2019 7:24 pm

I probably want clear. The ask is for the VBR server not the proxy/repo. The proxy/repo has 1TB of RAM. The VBR server is separate VM. It’s just a management server.

tsightler
VP, Product Management
Posts: 5476
Liked: 2294 times
Joined: Jun 05, 2009 12:57 pm
Full Name: Tom Sightler
Contact:

Re: VBR RAM for 2PB SOBR

Post by tsightler » May 03, 2019 2:30 am

The amount of data on the repo shouldn't matter much for the VBR server, however, with that amount of data, I'm assuming you may have quite a few jobs. Can you tell us a little more about the setup, how many backup jobs+backup copy jobs, etc. I've never seen a single backup manager process use so much RAM so that's my first concern. Do you know what job that specific backup manager was associated with (you can determine this by looking at GUID part of the process string and matching it to the job GUID).

Gostev
SVP, Product Management
Posts: 25145
Liked: 3698 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: VBR RAM for 2PB SOBR

Post by Gostev » May 05, 2019 9:15 pm

Yep, this behavior is definitely not normal: normally, backup manager process should consume less than 1GB of RAM. Perhaps you're running in some bug, or there's a 3rd party software conflict (like antivirus with real-time protection). Please open a support case, upload the debug logs package and let us know the case number. Thanks!

matthewingram
Influencer
Posts: 18
Liked: never
Joined: Oct 03, 2018 10:13 pm
Contact:

Re: VBR RAM for 2PB SOBR

Post by matthewingram » May 11, 2019 5:13 am

The case # was 03524751. IT was a part of that case. What I noticed was the memoery was being consumed after a manual rescan of a 2.4P B SOBR was initiated. There was about 1.8PB used. With 22GB the VBR server ran out of physical memory and filled the page file. The system went non responsive. I moved the memory up incrementally until I hit 64GB. At that point the veeam backup manager process peaked at 40GB used. Most of the memory was released about 2 hours later when the rescan was complete. Normal backuo operations I see about 20GB peak used, but now I have to keep 64GB on the VBR server encase a rescan is needed.

Post Reply

Who is online

Users browsing this forum: Bing [Bot], sherzig and 11 guests