Pirx - Maybe this has already been mentioned, but as an FYI, I was on a call with Federico (the author from HPE of the original Apollo reference document) a few weeks ago, and he mentioned they didn't need that much RAM, but it was populated with this much RAM due to HPE's memory population rules and guidelines for Gen10 (and basic memory optimization in the platform).
Our environment is mixed, the majority are smaller VMs (100-300GB) but there are also 10+TB VMs. I get the feeling that 192GB from the Apollo example could be not enough. Still this is the optimal configuration in price/performance I guess (12 DIMMs, 16GB). Even if it should be used as Repo server and only for copy jobs.tsightler wrote: ↑Mar 18, 2021 7:56 pm Also, a lot of that memory recommendation on Windows was attempts to work around the crazy Windows buffer behavior, which was about 10x worse with ReFS due to various bugs. Now, with v11 bypassing the OS buffer for writes on Windows, memory usage should be more normal. Still I would never recommend to go with less than 4GB per core recommended in best practice, and, arguably 4 GB per planned task (generally core = task, but if you want to oversubscribe cores on your repo, having more memory is the best way to make sure you can do this). Admittedly, 4GB is probably overkill, but if your environment is large, VeeamAgent process can grow quite a bit larger. There can be quite a difference in memory usage when you are backing up 50 VMs with 100GB disk vs 50 VMs with 4TB of disks.
I can pretty much say this, I've had far more customers regret not putting memory in the box vs the other way around! If you want to have less problems, don't skimp on it. Admittedly, I don't think you need to go overboard either. I've had customers buy boxes with ~1TB of RAM, which seems maybe the other extreme! If I was buying a big box like that, I'd probably look at no less than 256GB if it's repo only, 384GB if it's proxy+repo.
Note that the customers I work with are on the larger side, almost all have >10,000 systems being protected, some a LOT more, and many have dozens of PB of data under protection, but the lessons are mostly valid for customers of all sizes.
Dave Chinner:
Filesystem capacity doesn't massively affect repair memory usage
these days.
The amount of metadata and the type of it certainly does, though. I
recently saw a 14TB filesystem require 240GB of RAM to repair
because, as a hardlink based backup farm, it had hundreds of
millions of directories, inodes and hardlinks in it. Resolving all
those directories and hardlinks took 3 weeks and 240GB of RAM....
I've seen other broken backup server filesystems of similar size
that have had close on 500GB of metadata in them, and repair needs
to index and cross-reference all that metadata. Hence memory demands
can be massive, even in today's terms....
Unfortunately, I haven't seen a broken filesystem containing
extensive production use of reflink at that scale, so I can't really
say what difference that will make to memory usage at this point in
time.
So there's no one answer - the amount of RAM xfs_repair might need
largely depends on what you are storing in the filesystem.
Lucas Stach:
xfs_repair can be quite a memory hog, however the memory requirements
are mostly related to the amount of metadata in the FS, not so much
with the overall size of the FS. So a small FS with a ton of small
files will require much more RAM on a repair run than a big FS with
only a few big files.
However, xfs_repair makes linear passes over its workingset, so it
works really well with swap. Our backupservers are handling filesystems
with ~400GB of metadata (size of the metadump) and are only equipped
with 64GB RAM. For the worst-case where a xfs_repair run might be
needed they simply have a 1TB SSD to be used as swap for the repair
run.
Any update?mkretzer wrote: ↑Mar 21, 2021 10:35 am I mean at least the xfs tools are telling you how much RAM is needed. Now we just have to do some tests with volumes with different sizes/block cloning data and so on and we can extrapolate a rough value per TB.
I already increased the number of GFS points in our test environment so that in about 1-2 weeks we have twice the blocked cloned data. I will update this thread!
Code: Select all
mkfs.xfs -b size=4096 -m reflink=1,crc=1 /dev/sda1So in my casesu = <RAID controllers stripe size in BYTES (or KiBytes when used with k)>
sw = <# of data disks (don't count parity disks)>
Code: Select all
# mkfs.xfs -b size=4096 -m reflink=1,crc=1 -d su=256k,sw=24 /dev/sdb -f
mkfs.xfs: Specified data stripe width 12288 is not the same as the volume stripe width 6144
meta-data=/dev/sdb isize=512 agcount=350, agsize=268435392 blks
= sectsz=4096 attr=2, projid32bit=1
= crc=1 finobt=1, sparse=1, rmapbt=0
= reflink=1
data = bsize=4096 blocks=93755080704, imaxpct=1
= sunit=64 swidth=1536 blks
naming =version 2 bsize=4096 ascii-ci=0, ftype=1
log =internal log bsize=4096 blocks=521728, version=2
= sectsz=4096 sunit=1 blks, lazy-count=1
realtime =none extsz=4096 blocks=0, rtextents=0Code: Select all
blockdev --getiomin --getioopt /dev/sdbCode: Select all
# blockdev --getiomin --getioopt /dev/sdb
262144
3145728
# xfs_info /dev/sdb
meta-data=/dev/sdb isize=512 agcount=350, agsize=268435392 blks
= sectsz=4096 attr=2, projid32bit=1
= crc=1 finobt=1, sparse=1, rmapbt=0
= reflink=1
data = bsize=4096 blocks=93755080704, imaxpct=1
= sunit=64 swidth=768 blks
naming =version 2 bsize=4096 ascii-ci=0, ftype=1
log =internal log bsize=4096 blocks=521728, version=2
= sectsz=4096 sunit=1 blks, lazy-count=1
realtime =none extsz=4096 blocks=0, rtextents=0
Code: Select all
# blockdev --getiomin --getioopt /dev/sdb
262144
3145728