We did some ReFS vs. XFS tests....

[mkretzer]

Hi all,

we did some ReFS vs. XFS testing.

Sadly we could not compare identical systems because our ReFS system is our offsite production repository and the XFS system was build from some old hardware laying around. Still, the results are so different that Hardware cannot be the only difference.

Hardware:
Our ReFS is using a tiered Hitachi G370 storage system with 8x 3,8 TB SSDs + 64x 6 TB HDDs. Raid6 14+2 in a HDT Pool. Everything is connected by 2x16 GBit FC. We mapped 16 luns to the Veeam Repo server (so the storage can use all its processors for IO). We used disk spanning to create one big ReFS. OS is Windows 2019, latest Updates.
Our XFS is using two Hitachi HUS110 (which is a 2 generations older, EOL system with not much cache) but 216 3 TB Raid6 10+2. The systems bottleneck is its CPUs so it cannot use the performance of the disks fully. 4x 8 GBit FC direct connected. In total 18 luns were mapped to the Repo server. Via LVM we created one big XFS. OS is Ubuntu LTS 20.4.
Both Repo Servers have 16 cores and 384 GB of RAM (Dell R720).

We then created two copy jobs which copied all of our nearly 4000 VMs to these two systems and scheduled GFS point creation.

We compared the GFS point creation times for 4 jobs that are different in size and number of VMs.

Job 1: 101 VMs, 43 TB of backed up data, 425 GB per VM:
XFS: 0:36:42
ReFS: 2:00:59

Job 2: 1 VM, 7,8 TB of backed up data, 7,8 TB per VM:
XFS: 0:08:40
ReFS: 0:47:20

Job 3: 8 VMs, 6,6 TB of backed up data, 825 GB per VM:
XFS: 0:45:28
ReFS: 1:07:34

Job 4: 415 VMs, 43 TB of backed up data, 104 GB per VM:
XFS: 0:06:27
ReFS: 0:53:20

XFS performs quite impressively.
Whats more important for us (as we were one of the early adopters who worked for months with Microsoft) is that XFS to this day showed ZERO issues!

Markus

[HannesK]

Hello Markus,
thanks for sharing this information! Very valuable.

I see some advantages in the number of spindles for XFS. But as you said, CPU of the HUS is busy... nice.

I'm curious how the times improve once you upgrade to V11. We expect something around 2x faster for REFS in V11.

Best regards,
Hannes

[mkretzer]

Hello Hannes,

can you tell me what changes in ReFS support in V11?
In theory XFS should be faster as well as V11 uses the new IO access mode that was implemented for reading to tape in V10 correct?

Markus

[Gostev]

Nothing significant changes with our ReFS integration in v11. It's a minor tweak really: we were able to identify a bottleneck in the fast cloning process caused by one of the Windows API calls, and simply increased its granularity significantly. By the way, this was when we were trying to understand the root case for your ReFS tape out performance issue with Microsoft since you insisted all signs pointed at the issue in 10a code, we tried really hard to find something there! So in the end, all your pushing was not for nothing

As for your test, it certainly makes XFS look very promising. I'm curious how XFS will behave for you in the long run on the volumes with hundreds TBs of data, please keep us posted. Not that I'm saying you should expect issues, so far I'm not aware of any.

[mkretzer]

Its nice to see something good coming from it - and also that we know that with the new IO mode in V11 the tape thing won't happen anymore.

BTW our XFS is already at 325 TB of used data. We will fill it as much as possible!

[pirx]

The numbers look impressive. Did you have any bad experience or unexpected behavior with xfs (I saw you ZERO, but these usually is never ZERO issues )? How long is this now being used?

[mkretzer]

Hi pirx,

about 6 weeks now. And believe me i am *very* warily with something like this (after all i opened the initial ReFS horror story thread). But: I really mean no issue whatsoever. And that with all data in one volume (even MS dev recommends not creating such big volumes and recommends splitting into multiple volumes).

One thing i especially love (not directly XFS related): The flexibility of LVM vs. Windows Volume Management. As i said we have two very old storage systems, each having to controllers (active/active on different volumes). To get them evenly loaded we used "lvcreate -i 4" to always stripe the data over 4 volumes. If i want to add additional disks to the XFS i can just add more disks but i have to add additional 4 volumes - which do not have to have the same size as far as i know.

Markus

[pirx]

How long is your backup chain, how long do you keep backups? I guess xfs/ReFS will have a lot of work once data has to be deleted.

[HannesK]

I guess xfs/ReFS will have a lot of work once data has to be deleted.

yes, there significant I/O load during metadata operations / deletes compared with "idle" or "sequential write of incremental backups". But that's much less than anything you see today on your NAS storage systems. I remember in the early days of REFS that a customer called me because the storage reported "unexpected load". Sure, that happens, because once merges / deletes start, that looks "unexpected" from the storage "big data analysis alarm management".

Even the worst XFS / REFS (on up-to-date Windows) works faster than your NAS setup (for those who are not familiar: I'm referring to other threads).

[pirx]

Please keep us/me updated with your xfs experience, as we are still months away from a possible decision or even a POC.

[mkretzer]

Sure, will do.
As it looks now we will switch to XFS for our primary copy target.

I just hope Veeam invests in further optimisations i would hate if all the optimisations go into ReFS and XFS would be much slower that ReFS in the future (... i know we just have to find bugs in XFS Gostev ...)

Markus

[mkretzer]

@HannesK: Sorry, but ReFS still sometimes does something strange. A few days ago it deleted a monthly GFS point for a few hundret VMs and after that all merges that normally take < one hour hang for ~ 40 hours until we rebootet.

For XFS we did the following: We filled it with 100 TB worth of backups, created new backup chains beside that and then deleted all the old backups. No issues whatsoever

@prix: We hold 7 points + 3 monthly for 50 % of our VMs + 2 weekly for the other 50 %. That way we should be able to test most szenarios around deleting bigger restore points.

[Carlos.Gomes]

Hi @mkretzer. And about restore performance? Have you performed any tests?

[mkretzer]

Its hard to compare as the repo sits at a remote site and is attached to an older hardware... But we are quite happy so far, especially since the performance is very predictable, which was not always the case with ReFS.

[pirx]

@Carlos.Gomes we are using XFS with our high density servers in production for 2 months now. No trouble yet, at least not from a filesystem perspective. I'm still surprised that 2 of those servers can handle all disk IO we have (~1300 VM's, backup, copy and offload tasks, proxies are still Windows, but this will change soon with some additional Linux systems).

[Carlos.Gomes]

Got it @mkretzer and @pirx. Thanks for your answer.

I observed poor restore performance from similar equipment to a Hitachi VSP F1500, this is why I'm asking about it. If by chance you perform any restore, just let me know your experience.

[mkretzer]

What is bad performance in your case? We also have Hitachi as a Backend but only a G370...