Mass Storage Array Repository Design Best Practice

[jdodd6]

Let's say I have a Dell storage array with 1 Petabyte worth of storage added to a repository server as Direct-Attached Storage. A ReFS formatted volume would be impossible due to the RAM limitations for volumes over 200 TB.

What would be the best design for Veeam repositories with this device? 5 or so separate ReFS volumes added to a Scale-Out repository? Or just use it as 1 Petabyte NTFS volume repository and forgo the block cloning features that come with ReFS?

[david.domask]

Hi Joel,

Can you link to the RAM requirements you're discussing? ReFS used to have fairly substantial requirements, but as far as I'm aware that's no longer an issue, 1 PB should not be an issue for ReFS.

I would not use NTFS (especially since I think max volume size in NTFS is 256 TB). Main design concern should be one that includes mirror accelerated parity if you're going with ReFS, so you can get the healing aspect of ReFS in addition to fast clone.

Similarly, consider an XFS based repository -- we've seen extremely good performance and stability with XFS and even if you're not as confident with Linux, the Veeam Hardened Repository ISO simplifies the installation and management of an linux-based XFS repository.

[JRRW]

XFS.

Your answer is XFS.

No, I'm not a linux fanboy. I hate dealing with Linux.

But, it's the right answer.

We run a CEPH of around 1PB and use XFS. It's great.

[jdodd6]

David,

I read that info last year here, but I see it is no longer a requirement:

https://veeam-1.gitbook.io/veeam-backup ... types#refs


And it looks like the newest versions of Windows Servers support 8PB on NTFS. Depending on resources, I'll consider a Scale-Out Repository that contains the petabyte array with an applied configuration of ReFS mirror accelerated parity; or a scale-out repository with hardened linux XFS repository. The daily immutable backups would be a huge benefit with the Linux XFS repo.

[jdodd6]

Hi Joel,

Can you link to the RAM requirements you're discussing? ReFS used to have fairly substantial requirements, but as far as I'm aware that's no longer an issue, 1 PB should not be an issue for ReFS.

I would not use NTFS (especially since I think max volume size in NTFS is 256 TB). Main design concern should be one that includes mirror accelerated parity if you're going with ReFS, so you can get the healing aspect of ReFS in addition to fast clone.

Similarly, consider an XFS based repository -- we've seen extremely good performance and stability with XFS and even if you're not as confident with Linux, the Veeam Hardened Repository ISO simplifies the installation and management of an linux-based XFS repository.

Is mirror accelerated parity worth setting up if the petabyte does not have tiered storage? Currently, it's just a bunch of HDD's; no SSD's.

[david.domask]

Ah it's a fair point. Probably not in that case; ReFS does do some neat caching tricks for those situations (mentioned briefly in the article I linked), but there may be a performance cost here with HDDs that makes it not worth it.

[vmtech123]

1.5PB repo with REFS would be fine. You could either split the array and have a few repo servers if you want to gain performance/throughput, or just create a few REFS volumes on the storage as well. You can either span them or configure as totally separate repository's and use them as a SOBR. I tested this and it worked fine for me. Also consider your largest job/VM size.


Here is what I'd consider.

On the storage side make sure your Volume's or LUN's meet best practice for size and performance.
- 1 big LUN or 3 big LUN's often won't utilize all the CPU cores on the storage and it's better to have several.

Lets say you need 12 Volumes to utilize the cores and your SAN is 1.5 PB for this example.

You could create 12 - 100TB volumes on the storage, then in windows create 3 REFS partitions that utilize 4 LUNS.

on your SAN split into 3 REFS partitions.

[vmtech123]

1.5PB repo with REFS would be fine. You could either split the array and have a few repo servers if you want to gain performance/throughput, or just create a few REFS volumes on the storage as well. You can either span them or configure as totally separate repository's and use them as a SOBR. I tested this and it worked fine for me. Also consider your largest job/VM size.


Here is what I'd consider.

On the storage side make sure your Volume's or LUN's meet best practice for size and performance.
- 1 big LUN or 3 big LUN's often won't utilize all the CPU cores on the storage and it's better to have several.

Lets say you need 12 Volumes to utilize the cores, and your SAN is 1.5 PB for this example.

You could create 12 - 100TB volumes on the storage, then in windows create 3 REFS partitions that utilize 4 LUNS.

You now have 3 x 400TB REFS partitions, that you could add as a SOBR.

When you go to migrate, if you chose physical servers in the future, you could move 1 at a time to a new 400TB server vs 1 giant repository which makes it difficult. I read somewhere that 400TB may have been the max recommended size, or close to it, but that could be old.