Best config for linux immutable repository and 2 volumes

[nvaes]

Hello,
following the limitation of 128Tb per volume of my DAS (Dell ME4012)
with two 100Tb volumes on my Linux immutable repository

The question is is it better to :

- make a scale-out-repository (2x100Tb)
- make an LVM with the two disks on the linux and present a single volume? (1x200Tb)

Thanks

[HannesK]

Hello,
and welcome to the forums.

I think there is no "right" or "wrong" for that question. People who like LVM will tell you "LVM". That's fine as the size is still relatively small with 200TB.

Personally I would go for SOBR with two extents. I like to avoid any unnecessary abstraction layers. Except you have very large VMs that have backups over 100TB (which don't fit in one extent) - then LVM is the way to go

Best regards,
Hannes

[biohazrd]

Hello,

We recently purchased a new enterprise storage appliance for our production VMs. After migrating our storage to the new appliance, we were left with several Dell/EQL PS6210 shelves. While older, these are perfectly functional units, support 10Gb iSCSI, and we maintain a 3rd party hardware warranty for drive/part replacement. We decided to repurpose these units to store local immutable backups in combination with a hardened Linux repository. Note that the SAN network has been completely airgapped on separate switches to prevent any possibility of attacking the storage directly. It is, in effect, configured as a DAS.

One of the limitations of this particular model is that it cannot present LUNs/volumes larger than 15TB. Because I need more than 15TB for storing backups, I am considering two possible paths forward:

1) Create multiple repositories and add all of these repositories as Performance Tier extents in a SOBR. Additionally, if I would be better served by using Data Locality or Performance file placement with this approach.
2) Use LVM to create a single virtual volume in order to aggregate the available space

It appears as though I am not the first person to consider these possible configurations, and I found a thread where a Veeam employee essentially said that both configurations are viable.

It occurred to me that, when using Performance backup file placement, if a full backup file and an incremental were placed on separate extents there would be no way for the two separate XFS filesystems to fast clone the blocks to quickly complete the merge. Significantly impacting the amount of time needed to complete a merge. I would think that a LVM based approach solves this, as there is only one large extent and there would never arise a situation where the full backup file and the incremental were on separate volumes. The following thread seems to indicate that this may be the case.

However, if the placement policy when using Data Locality keeps the chains together (which based on this KB article it appears to do so), this would be a non-issue as Veeam would attempt to keep the files on the same volume. Resulting in merges being able to reliably use the XFS features to quickly complete merges.

I'd like to do this right the first time, which is why any advice or experience in similar configurations would be welcome.

Oh, and one last question: Do the settings on a SOBR override those of a backup repository? For example, if my repository does not have "Use per-machine backup files" selected but the SOBR does, what happens? What if the repository has it enabled, but the SOBR does not?

Thanks

[HannesK]

Hello,
I would make the decision based on the size of the backups and the amount of extents you have. Sounds like it would be 2x15TB. Two extents are fine. If it was 20x15TB for example, then I would recommend LVM. LVM adds (a little) complexity. I tend to avoid as much complexity as possible. If you have large machines with many TB, I would also go for LVM to avoid uneven placement.

Yes, with XFS (also REFS) the software tries to put everything on one extent to allow block cloning. Only if an extent runs out of space, that rule will be violated.

Do the settings on a SOBR override those of a backup repository?

yes

Best regards,
Hannes

[mkretzer]

My take on this: always use LVM if you don't have a situation where only one for the volumes is likely to fail, which should not be the case in your situation. Even with V12 you cannot backup to a chain while the backups are being moved.
Also, you are much more flexible with LVM (online move to a new device without downtime as long as its bigger).

To be honest, i would even use LVM if the decision is to make two volumes and SOBR. There is just no reason not to use LVM if you know what you are doing.

Markus

[biohazrd]

Hello,
I would make the decision based on the size of the backups and the amount of extents you have. Sounds like it would be 2x15TB. Two extents are fine. If it was 20x15TB for example, then I would recommend LVM. LVM adds (a little) complexity. I tend to avoid as much complexity as possible. If you have large machines with many TB, I would also go for LVM to avoid uneven placement.

Yes, with XFS (also REFS) the software tries to put everything on one extent to allow block cloning. Only if an extent runs out of space, that rule will be violated.


yes

Best regards,
Hannes

Excellent, thank you. This is likely to be closer to 20 extents over 4 separate shelves, so it seems like LVM is the way to go based on your response and mkretzer's response.

Thank you for your insights.

[tsightler]

Use LVM, it's great and there's no reason not to use it. I definitely would not attempt to use SOBR to solve storage limitations. SOBR is primarily designed to consolidate mulitple servers into a single namespace, not to solve storage limitations. My best practice recommendation is to do everything you can to have no more than 1 SOBR extent per physical server. If you have to exceed that, exceed it by the minimum required to accomplish your goal.

Now, Hannes is correct that 2 extents would likely never matter, but it's just the slide down the slippery slope of bad design as soon as you start using SOBR to solve storage limitations, and it leads to people eventually having things like >100 SOBR extents on just 5 servers (yes I've seen this). Barely a few weeks go by where I don't see a deployment that has gone down this rabbit hole and now needs a full re-architect, which is really hard when you have 100s of TBs (or PBs) of data across 10's of extents.

Years ago, I did a "Worst Practices" presentation of the worst things I see in real-world deployments, but it was prior to the existence of SOBR. If I did the same presentation today, attempting to use SOBR to work around storage limitations would likely be one of the top 3 worst practices. So, in summary, I agree with mkretzer, use LVM, it gives maximum flexibility and it's quite simple for basic volume management and it's solves the storage limitation problem much better than SOBR. Leave SOBR to do what it does best, consolidate storage across independent compute nodes.

[mkretzer]

We did exactly what @tsightler mentioned as "worst Practice" - we used SOBR to work around ReFS limitations - now we have 5x ~250 TB extends all on one server with no easy way to consolidate them.
Everytime one extend is filled up very bad things start to happen since we loose fast cloning benefits because of rebalancing. Sind the full backups are then on two extends the whole system can get even more unbalanced when the backups are very large and SOBR chooses one extend for rebalancing for all the backups that need to be rebalanced (leading to another rebalance 4 weeks later)...

Gladly, we had no reason to do the same with XFS+LVM....

[bbaumgartner]

@tsightler,

Similar situation with storage limitations:

QNAP with 350TB, which can only server up 175TB per iSCSI LUN.
I want to combined (2x175TB LUNs) attached to Ubuntu for Veeam immutable repository.

From your presentation: "How to build secure Linux repositories" I used ZFS commands to successfully create zpool which combined LUN's and severed to zvol.
I was able to successfully layer XFS on top of zvol and create Veeam immutable repository.

But I have not implemented this before and wonder if about performance and other unforeseen issues.

I'm not using any of the other features ZFS offers just zpool and zvol.
Would it be better to use LVM instead?

Thanks,

[HannesK]

I remember Tom presented the ZFS + XFS at VeeamON 2020.







I cannot judge which approach is better. I think, more people use LVM, but I did not hear anything negative about ZFS either search.php?keywords=zfs&terms=all&autho ... mit=Search

[tgietz]

Hi @HannesK ,

just read your post about creating a XFS file system in a ZVOL.

Is it also supported for a Hardened Linux Repository with Block Cloning enabled?

Regards
Tobias

[galbitz]

Just wanted to say we ran into the same thing recently. We ended up doing lvm volume with xfs ontop of it and it has worked great in production for a few months, no issues.

[HannesK]

Hello,
from Veeam side, everything lower than XFS is irrelevant from a support perspective. So XFS in a ZVOL would be supported.

Best regards,
Hannes

[BackupBytesTim]

I can say we also have XFS in a ZVOL and haven't had any particular issues with that in quite some time.