1.5 PB Hardened Repository

[NightBird]

Hello all,

In 2025, is it possible to have a Hardened Repository with a SAN in backend presenting a single 1.5PB LUN, all in xfs ?

The idea is to limit the imbalance that SOBR can provide (especially with immubatability, and no rebalance possible).

Thanks for the feedback

[Gostev]

Hi, sure - this has always been possible technically. Thanks

[NightBird]

somebody with a feedback with this repo size ?

Thanks

[Gostev]

I believe @HannesK knows of a few customers over 1PB

[HannesK]

yes, @mkretzer for example runs that size.

[mkretzer]

Indeed. No issues whatsoever. 1,5 PB, blazing fast with >= 10 GByte/s write speed.
But we optimized it and tested alot: For example we configured LVM to use as many stripes as we have CPU cores in the storage system. So not ONE big LUN which i find is very inefficient as most storage systems have many CPU/cores/controllers.
Also, using 128 k stripes helps.

Our current command we use for a 40 core storage system:
lvcreate -n LVMNAME -i 40 -I 128k -l100%FREE VGNAME

[mkretzer]

BTW using multiple volumes has one additional benefit: You can migrate Repos via pvmove much more granular - without this you have to migrate the whole volume all at once.
Using many volumes under the LVM has many benefits and from what we found not much downsides.
The only issue we found is that with many volumes and all of them having multiple paths the SCSI scanning sometimes takes too long at boot and the LVM must be manually re-activated (lvchange -a y DEVICENAME). But we have only encountered this once and the fix was instant.

[NightBird]

Thanks for your feedback @mkretzer

Just another question, what is your storage array backend ? and in front for the repo what kind of server do you use (CPU/RAM) ?
Our need is about to backup 2500 VM (actually on SOBR with some bunch of 110TB servers...).

[mkretzer]

We use a Hitachi 5200 Fibre Channel Storage system. Its "slightly" oversized because we had issues with the previous Hitachi system and they made us a good deal (they are extremely good with customer service).
Our repo Server is a Dell Server with 2x AMD EPYC 9374F 32-Core CPUs and 768 GB RAM. All is set up in a way that we can also use this server as the central Veeam (Linux based) Server when V13 comes.

Our copy target servers do the same amount of backups (5000+ VM) with 10+ years old 8-core Intel CPUs and 128 GB RAM without issues, so should be good with much less server hardware.

[NightBird]

Thank you very much for your feedback, we will go I think that way instead of SOBR one (may be with some NetApp E-Series storage system in backend).

[NightBird]

Oh just one another question, 10GBytes/s (80Gbits/s) what kind of attachment do you have ? I think 100 GbE in front for the repository server, but in backend ? FC64 ?
What's your lun size ?

[mkretzer]

FC32 x 4 with least load loadbalancing (Linux default) so we can use nearly 128 GBit/s - Frontend 1x 100 GBit + redundant 25 GBit Fallbacks.
Thats another benefit of having multiple volumes - it gets distributed very well.

[christian.naenny]

@mkretzer:
Would you mind elaborating in a bit more details how your setup looks like?
'm just about to setup a hardened repository on a HP DL380 RHEL 9 system with its disks on a Hitachi VSP E1090 storage system. I was thinking of using about 12 x 100TB extents in a SOBR.

Thanks!
Christian

[HannesK]

Hello,

I was thinking of using about 12 x 100TB extents in a SOBR.

12 extents on one server I something I would avoid. It wastes space because every extent would have some free space and many extents on one server lead to inefficiencies in the Veeam scheduler. Normally it works, but in general it should be avoided to have more than a few (1-4, not a strict requirement, just experience) extents on one server.

The main decision to make is, how big you would want to grow your XFS. Then think about striped vs. spanned LVM and then create logical volumes accordingly.

Best regards
Hannes

[mkretzer]

@christian.naenny Why would you do SOBR at all? You can use SOBR at a later time, converting from the non-SOBR.
Just throw the 12 volumes in a striped (or if you don't trust it spanned) LVM when it comes from one storage.

What benefit would a SOBR have for you? The only thing i can thin of would be:
- limit the risk if some corruption occours -> but if something is corrupted it is likely it is not limited to one volume (for example if the storage has a firmware bug).
- make moving the volume easier -> this is where LVM shines. With pvmove (https://linux.die.net/man/8/pvmove) you can move the stipes to a new storage device and then extend the volume via LVM.

In the past there were some limitations (moving worked only if you move all the volumes belonging to one stripe set, you had to add exactly the amount of volumes you have in one stripe set) but in the latest kernels we found most of these limits are gone.

We started small: for example stripe only 2 or 4 volumes (so that both controllers of the 1090 have something to do) - "lvcreate -i 4 -I 128k". Put some test data on the XFS. Try pvmove while test backups are running. Extend the volumes via vgextend/lvextend.

Get to know how LVM works. Its really, really good and easy once you get the basics of it.

If you need more infos just let me know and i will go into more detail

[SergeM]

I'm intrigued by this set up but have a question that I’m genuinely interested in how you’ve evaluated this:
presumably the storage array already does striping as well, with that in mind, have you tried letting the storage array handle the performance vs data resiliency aspects and simply present a single (or small number of luns) to your LHR while maximizing the throughput of your FC adapters?

[mkretzer]

One more thing: According to https://community.hitachivantara.com/bl ... -e1090-arc the system has 32 cores per controller. So using 12 volumes would not be optimal. I think 16 or 32 volumes should be better.

[NightBird]

What is the relationship between the number of CPU cores of the storage system and the number of LVM volumes ? I don't understand this very well.

Why just not 2 big striped volumes, one per storage controller ?

[mkretzer]

Yes, the storage system stripes. But this is just the optimization on the disk level. Using only one volume often means only one controller and even only one core is working.
We had a lengthy optimization case with Hitachi support where they recommended that we should have IO distributed over multiple volumes to give all the controller cores something to do.

2 big striped volumes has the issue that Hitachi has a max LUN size of 256 TB.

Just test it out - on our VSP One B26 system which had a CPU bottleneck using striping over both controllers doubled the IO performance. Using more volumes further improved it.

It just has no downsides so i even would do it for 5 % more performance.
Another benefit of using multiple volumes is that you can use pvmove more flexible as you can in theory split the volume to multiple systems in the future.

Edit: Just ask Hitachi perhaps they tell you something different

[christian.naenny]

@mkretzer: Thank you very much for your answer! We're using a SOBR right now because the repository is now connected directly from the Veeam Server on Windows. NTFS was not able to handle volumes bigger than 256TB when we started (on Windows Server 2016). So that's why we had to use a SOBR. But thanks to your input we might re-think our approach. Limiting the risk if some corruption occurs, that was one of the main reasons to stay with a SOBR. But you might be absolutely right, we should embrace the advantages of the LVM!

[Florent Foulquier]

Hi All,
I would recommend the following White Paper which explain how to create an hardened linux repo on a single server. UP to 1PB and 1,5PB according to the server model:
https://www.hpe.com/psnow/doc/a50000150enw

also you will se some benched performances.

By the way you can have a FC HBA in the server but in that case it means that you will have the REPO and PROXY roles on the same unit which is not the VEEAM recommendation for security reason on an hardened linux repo.

[mkretzer]

@Florent Foulquier What you mean is direct backup from SAN.
We do everthing to a FC backend system (but still use Hotadd for the "frontend") which is no issue for the hardened repo if you make sure an attacker cannot attack the storage network or management.

@christian.naenny For us we decided we limit one volume to one backend storage system. With modern RAID, Hotspares and Flash its not that likely that we will loose a volume of the backend storage. But this all depends on the quality of the storage system.

[christian.naenny]

Now after quite some testing and research we had to decide to create a Scale-Out repository with 4 extents. Unexpectedly, we discovered that RHEL 9 imposes a limit of 1PB on the XFS filesystem!
https://access.redhat.com/articles/rhel-limits#xfs-10
We're going to create extents that can easily be extended up to the max filesystem size but start with about 250TB per extent. That's because we don't have to add more volumes from the storage system until the end-of-life of the repository server has been reached.

[HannesK]

yes, that's an artificial limit in RHEL. Not a technical limit of the file system itself. I talked to Red Hat and they will only increase it once there is online repair for XFS.

[mkretzer]

Why not use 1 PB extends then if 1 PB is supported?

[christian.naenny]

We're limiting the available space for safety reasons. The storage system we use is not just supporting Veeam but also another backup solution. The space mapped to all the customers servers is way more than what we have physical space in the system. But that requires us to limit the usable space and monitor the used space closely!