Question about XFS vs. ZFS space overhead

[mkretzer]

When we create a new XFS filesystem with reflink and crc some of the space is already preallocated for metadata. For a 1 PB repo this is a few TB (sadly i do not have the exact value).
How does ZFS compare to this? Is it more or less space efficient as XFS?

[HannesK]

Hello Markus,
for the comparison you ask for, ZFS is better. An empty 16TB volume has 115G used on XFS vs. only 128KB used on ZFS.

That question is, how relevant is it overall. To be fair, I did not test that, but I assume it will show similar results like 64KB REFS vs. 4KB REFS with the difference that ZFS uses 128KB block size / recordsize per default. I expect ZFS to require a bit more space than XFS because of the larger block size.

I remember you have SAN storage... do you plan to put ZFS on SAN? If yes, what would be the reason for that (because ZFS features make most sense when giving it disks directly)?

Best regards,
Hannes

[Gostev]

Perhaps ZFS may just not pre-allocate metadata, but rather create metadata banks as it goes. So ZFS might actually be much less efficient than XFS in terms of metadata size but you simply won't see it from the test above.

I do agree that metadata size should not be a consideration though - you should be comparing file system features, their maturity, reliability etc.

Also, from disk space usage perspective keep in mind that some file systems are known to struggle when volumes get close to full (over 90%). So you may be worrying about a few TB today while in reality you won't be able to use a few dozens of TB in the end... I'm not implying XFS or ZFS suffer from this issue - I don't know - just trying to convey that proper testing of disk space usage efficiency should include a number of additional considerations and tests.

[mkretzer]

@HannesK interesting, especially the 128 KB thing. We want to have one repo per backend storage device. With storage devices in the PB scale we always plan ahead on how to migrate this data if needed, without any repo/backup downtime. XFS + LVM is a very good solution for that but since ZFS is now also supported i was wondering which of the two solutions is the right one long-term.

With XFS we never had any issues even with quite filled filesystems (> 90 %) and i guess we will keep using this for the time beeing.

[HannesK]

I remember you do online repository migration with pvmove. So you are looking at "zpool replace" for ZFS I guess? Because if you were thinking to use "zfs send" and "zfs receive" for snapshot replication with short downtime (not online like pvmove), then there is a bummer: zfs send / receive dehydrates reflinks (not what I expected coming from other block-level replication methods). OpenZFS developers told me, that this behavior will stay and the solution for that would be using "fast dedup" in future.

Fast dedup is the the new inline deduplication engine the OpenZFS project is working on. Assuming that engine works massively better than the current ZFS deduplication, there is still a huge downside: the data still needs to be send over the wire as de-hydrated data.

If you had no issues above 90% fill grade of the file system, then XFS also seems to have an advantage there (although I heard of issues with 95%+). In general, the copy-on-write file systems don't like full disks.

For the SAN solution, I see XFS in advantage also in future. I see no reason for ZFS in that scenario and XFS has proven to be stable. ZFS strengths are relevant with directly connected disks. Also please keep in mind, that we allow reflink for any file system with the /etc/veeam/EnableZFSBlockCloning file, but it is not supported. We did not fully test ZFS reflink yet, but we have plans on that and depending on the outcome we would then support it.

[mkretzer]

Thank you very much Hannes!