ReFS vs XFS & StorageSpaces Mirror-acceleration (op-ed)

[JRRW]

I just wanted to put some data out for the community to read/consider/debate.

There was a post from a while back talking about ReFS vs XFS, and one poster talked about the Erasure Coding and the performance issues of Storage Spaces (the way Microsoft WANTS you to deploy ReFS to take advantage of all the special sauce of ReFS) but he was seeming to talk about S2D in the post... So I wanted to clarify things for anyone coming into Veeam and considering a Windows StorageSpaces
repository.

In theory, Mirror-accelerated parity looks awesome on paper.
Up to 80% parity Storage Efficiency (5 disk columns in a 10+ divisible by 5 array, i.e. 40 disks) for cheap-n-deep HDD WITH a high-performance 'mirror' SSD array/tier. All writes land on super-fast flash and move down to cheaper/deeper storage automagically!

Storage Spaces vs Storage Spaces Direct (S2D)

They are drastically different systems in performance even with similar underlying hard drives being used.

What I've seen is that S2D mirror-accelerated parity performance is significantly better than a single-host Storage Spaces mirror-accelerated parity configuration.

Why? Who knows. I don't have the specific numbers off hand on a S2D cluster but they are out there and real world proves this out (although I personally hate Hyper-V clusters, a properly configured Azure Stack HCI/Azure Local would be the route I'd go) that they are very performant. I have a friend who has an older Dell R730 based S2D cluster with nvme & spinning drives that gets amazing performance.

Stand alone Storage Spaces? Junk, even if you use SAS-SSD or NVMe drives that each can do 1+ GB/s read/write. NOTE: I HAVE seen improved performance in 2025 BUT I do NOT recommend using 2025 until Microsoft deals with a LOT of random bugs that have come out / are still out there for it (i.e. one of my hosts SET shoots itself every reboot and the bug is not scheduled to be patched by Msft until February despite them knowing about it since September) But even then, I have only seen performance improvements on the MIRROR writes - the parity writes/destage is still ~300MB/s - however that DOES make it a lot more attractive to use as long as you size your SSD tier large enough! You get a lot more bang for your buck - once Microsoft makes 2025 not suck.

____________________________________________________________

Honestly it’s frustrating that there are no FOSS tiered XFS-compatible SDS options (at least none I’m aware of that behave like “real” SSD-->HDD tiering).

Microsoft - despite being slower than it should be - at least gives us:
a true SSD tier that automatically tiers data down to spinning parity.

But write performance isn’t great:

• Best case burst: ~400–800 MB/s
• Once the SSD tier fills: almost never beyond ~200–250 MB/s
• Destage from “fast” tier: bottlenecks around ~200–250 MB/s

You can set the destage threshold as low as ~20%. We typically keep ours ~70% except during large full jobs / heavy ingest. This 'frees up' space on the SSD tier to allow incremental backup jobs to land directly on flash, which in turn 'should' help with Synthetic Full mapping and for completing those jobs faster.

So it's really the destage or when it hits ~90% on the "SSD/nvme" tier that performance really starts to tank hard - and the performance writing down to the 'slow' tier is WAY worse than it should be - none of our HDD systems tested/used were running less than 10x7.2k drives and most were running 10k drives - with that many spindles you should have MUCH faster write speeds than 200-250MB/s.

____________________________________________________________

Examples we’ve seen (different hardware, similar ceiling)

I'll include some basic 2GB file read/write 'real world' performance metrics but this doesn't tell the whole story as the 'writes' are only going to the SSD tier not the destage / slow tier. It gives you an idea though on how 'bad' Microsoft's software defined storage is, if even mirror SSD performance is this poor.

Example A
ReFS mirror-accelerated parity with:

• 4 × 3.2TB U.2 NVMe
  10 × 2.8TB 10k SAS
• 5-disk columns

I don't have this system running currently, but it performs more or less the same as:

Example B

• 4 × 1.92TB SATA SSD
  12get- × 8TB NL-SAS 7.2k
• 3-disk columns

[Read]
SEQ 1MiB (Q= 1, T= 1): 376.240 MB/s [ 358.8 IOPS] < 2784.41 us>
RND 4KiB (Q= 1, T= 1): 15.473 MB/s [ 3777.6 IOPS] < 264.02 us>

[Write]
SEQ 1MiB (Q= 1, T= 1): 229.030 MB/s [ 218.4 IOPS] < 4538.96 us>
RND 4KiB (Q= 1, T= 1): 5.329 MB/s [ 1301.0 IOPS] < 767.09 us>



Example C

• 8 × 1.92TB SAS-SSD
  40 × 1.8TB SAS-HDD 10k
• 5-disk columns

[Read]
SEQ 1MiB (Q= 1, T= 1): 264.430 MB/s [ 252.2 IOPS] < 3961.03 us>
RND 4KiB (Q= 1, T= 1): 10.120 MB/s [ 2470.7 IOPS] < 403.51 us>

[Write]
SEQ 1MiB (Q= 1, T= 1): 326.947 MB/s [ 311.8 IOPS] < 3203.52 us>
RND 4KiB (Q= 1, T= 1): 4.608 MB/s [ 1125.0 IOPS] < 887.08 us>

Taking it a step further…

Example D
Mirror-accelerated mirror with:

• 2 × 960GB U.2 NVMe
  4 × 480GB SAS SSD

…is also basically the same performance.

[Read]
SEQ 1MiB (Q= 1, T= 1): 610.518 MB/s [ 582.2 IOPS] < 1715.72 us>
RND 4KiB (Q= 1, T= 1): 21.487 MB/s [ 5245.8 IOPS] < 190.34 us>

[Write]
SEQ 1MiB (Q= 1, T= 1): 376.446 MB/s [ 359.0 IOPS] < 2781.33 us>
RND 4KiB (Q= 1, T= 1): 9.034 MB/s [ 2205.6 IOPS] < 452.52 us>

How?
How does Microsoft struggle so badly at SDS on a single host, yet in S2D clusters they can give vSAN a run for its money?

____________________________________________________________

One possibility: no meaningful CPU/EC “offload” being used

To the point on erasure coding / parity math itself:
It makes me highly suspect that Microsoft is not using the kinds of CPU acceleration paths other SDS stacks do.

Some performant stacks/projects that explicitly leverage CPU acceleration (AVX2/AVX-512 / ISA-L, etc.):

• Hadoop HDFS: Intel ISA-L acceleration
• Ceph: ISA-L EC plugin (Intel-specific) - benchmarks show it outperforming standard CPU mathing EC
• MinIO: AVX2 / AVX-512 optimized EC paths
• (I think) Excelero: AVX-512 (from what I’ve seen historically)

Microsoft has never published “hardware requirements” for parity/EC (e.g., AVX2/AVX-512/GFNI/ISA-L), which strongly suggests they aren’t relying on any specific accelerated paths beyond generic CPU math.

____________________________________________________________

One advantage we do see

The big advantage we see with Microsoft ReFS mirror-accelerated parity:
Synthetic fulls seem faster than hardware RAID on spinning disks.
I haven’t done extensive benchmarking on that yet, but that’s what we’ve observed.
Once 2025 is stable, the performance tier is ACTUALLY very fast for ingest - but in 2019-2022 we've not seen that same performance on the mirror tier.

____________________________________________________________

Notes (this is not a “bad build”)

Hardware is optimized:

• “Ideal” memory STREAM Triad bandwidth via proper DIMM layout
• Appropriate CPU types (xeon Gold 1st or 2nd gen)
• HBA adapters NOT Raid cards in 'pass-through'
• Microsoft-recommended ReFS optimization tweaks applied
• Power Protection enabled
• Large Column sizing
• Single parity NOT double parity

This also isn’t just for backup repositories as we see the same characteristics for Hyper-V workloads on it locally and general read/write IO.
Sorry for the rant..
____________________________________________________________

Why I’d prefer XFS most of the time

I’d use XFS 99% of the time if I could have either:

• A) All-flash under it (this is how I run my main repository: 128 x SATA-SSD Ceph-backed XFS), or
• B) A real SSD --> HDD auto-tiering approach under XFS (or mdraid/mraid/etc) - I wish this would come about

But otherwise, XFS is completely stable and usable. Plus Linux is - in many ways - a more secure and a safer deployment than a Windows OS.
Until Server 2025 is stable/not the buggy POS it is right now, the mirror-accelerated parity is a bit of a niche need but can be run on Server 2022.

[pybfr]

SSD is notably more expensive than HDD (and it is going to get much much worse in the coming months) and even though is offers better performance, for a large majority of customer multi GB (capital B) read/write is already very, very good...
But as usual your mileage may vary depending on your requirements.