ZFS Experience? good usage or don't touch

[righter]

Hi

We wanna setup a huge repo with RAID6 (260TB). But we were aware of a rebuilding process in case of a failing disk.
Yes one solution would be to split it up to multiple repos, but I don't like that way and loose space..

So I tried JBOD and setup a normal ZFS Pool 2x raidz2 in one pool. But our benchmarks resulted with half of the performance as of a RAID60 (Which I can' expand on the RAID Controller)
Therefore I also created a zvol with XFS to finally use the fast clone option. but a DD benchmark ended up with horrible CPU Usage.

Has anyone used ZFS, maybe also in combination with XFS, or would that be just a setup which is not suggested?

I tried everything with Ubuntu 20.04 and OpenZFS but the performance results are far away from the RAID Controller.

[mkretzer]

We do not know about ZFS but why don't you just use LVM? You can do the striping in LVM over multiple RAID6 for example.
CPU usage is minimal for such a configuration and rebuild time depends on how big you make the RAID6 components.

[tsightler]

ZFS is great, but you have to actually architect the ZFS storage to meet your performance and don't expect to use RaidZ2 pools and have performance similar to a hardware RAID controller in RAID60 with the same set of disks, etc. ZFS is not going to win any performance contest with RAIDZ2 especially if you didn't architect it with a performance target in mind and instead just get the biggest, slowest disks you can buy. Remember ZFS is doing all of the hard work with the system processor and it doesn't have the advantage of any battery backed write back caches or anything like that, you can improve the performance with various tweaks and doing things like putting the SLOG on super fast storage (you should) but, even with all of this, the same hardware will run much slower on ZFS vs any traditional RAID. But you also don't get the added data integrity features, snapshots, incremental copy, compression, etc.

ZFS is also incredibly conservative with how it handles and writes data, it always makes sure it's on disk before it is acknowledged unless you explicitly turn this off.

So, if you want to use ZFS, absolutely do it, but be prepared to learn about and architect a ZFS system that meets your performance needs, otherwise, just use RAID60 and be happy, 260TB is really not THAT huge these days (I work with customers with repos 3-4x that large in some cases). I mean, it's still big, but maybe I've become numb to it!

[righter]

@tsghtler yeah we aimed to use RAI60 but there's no way to increase the pool (LSI Controller but I think no RAID60 can't be increased)
So we want to have the ability to extend our RAID. Yes i was testing raidz2 of course
I was just horrified of the rebuilt task, but in my test it takes "only" 1 day to rebuild the RAID6 with 20x16TB disks, thought it was way more longer.
I just created a RAID6 and it has almost the same speed with a DD bench as RAID60. I thought RAID60 should be faster but it wasn't so maybe we just use RAID6.

I love ZFS and it's comfort but as long as Veeam doesn't support it plain it's not a good idea to use it for a repository.
Last time I was playing with ZFS was years ago, thought it's now better as a hardware RAID

@mkretzer: I didn't know that LVM has the capability to stripe two or more RAID6, but as long as I don't have more speed as a RAID60 I don't see the clue to use it.

So I see finally that Hardware RAID has still his privileges to exists

[tsightler]

LVM can be an option to allow you to expand in the future, even if using hardware RAID. For example, if you add disks in groups of 10, you can create a hardware RAID6 for each group and then use LVM to group them together into a single, striped pool, basically, allowing it to be the 0 of the RAID 60. So perhaps you can't expand your existing RAID group, but you can add new drives and a new RAID group and use LVM to expand it transparently. Just an example.

[mkretzer]

Thats what i meant LVM gives you extreme flexibility for minimal cost. Especially if you need to migrate your XFS to a new device without loosing block cloning.