Linux repository using ZFS

[andyg]

Anyone using Linux as a repository with ZFS as the filesystem?

How have you found it in production? Any good or bad points you have noticed?

Did you use RAID, or present each of the disks and let ZFS handle them?

[Gostev]

I found ZFS to be very popular choice among our service providers, and rarely heard anything but praise from its users.

[evilaedmin]

Does the Veeam file storage format work well with the native compression and/or deduplication features of ZFS?

[nitramd]

What file storage format are you referring to?

[csydas]

Hi Evilaedmin,

Data is data in *nix. It doesn't mess around, and as long as you're prepping the backups for a deduplication repository (and your repo has ample memory), you should see no issues. I use ZFS for one of our primary repositories at a site, and I'm quite happy with it. A cronjob to trigger snapshots every so often makes me feel way better about the backup files.

[Andreas Neufert]

As Veeam do compression on the Source Side Datamover (Proxy or Agent), the storage system need to handle only half the data which speed up things.

If you enable compression on the Storage and disable it on the Veeam side, your Storage need to handle 2x more data.
So best practices is to use Veeam Compression.

Double Compression will slow down things and usually produce bigger backup files. So not recommended.

[anthonyspiteri79]

From a dedupe point of view with ZFS, make sure you understand the hardware and performance overheads that are required when you turn it on. It's got a heavy RAM overhead which is why we decided not to turn it on. It can be pushed into L2ARC but that comes with its own costs.

This post explains it well. https://constantin.glez.de/2011/07/27/z ... ot-dedupe/

RAM Rules of Thumb

For every TB of pool data, you should expect 5 GB of dedup table data, assuming an average block size of 64K.

This means you should plan for at least 20GB of system RAM per TB of pool data, if you want to keep the dedup table in RAM, plus any extra memory for other metadata, plus an extra GB for the OS.

[Andreas Neufert]

Global deduplication usually has a performance impact on the backup and restore and is used inline with the Veeam best practices as secondary backup target for long term backups usually.

[evilaedmin]

Hi Evilaedmin,

Data is data in *nix. It doesn't mess around, and as long as you're prepping the backups for a deduplication repository (and your repo has ample memory), you should see no issues. I use ZFS for one of our primary repositories at a site, and I'm quite happy with it. A cronjob to trigger snapshots every so often makes me feel way better about the backup files.

Hi csydas,

One counterexample to your statement that immediately comes to mind is the Dell/EMC Networker on-disk storage format: When not using Data Domain appears to be by design "misaligned" with ZFS blocks and ZFS deduplication has nearly zero effect. I do appreciate all responses to my query.

[evilaedmin]

That is...a lot of RAM.

From a dedupe point of view with ZFS, make sure you understand the hardware and performance overheads that are required when you turn it on. It's got a heavy RAM overhead which is why we decided not to turn it on. It can be pushed into L2ARC but that comes with its own costs.

This post explains it well. https://constantin.glez.de/2011/07/27/z ... ot-dedupe/

[anthonyspiteri79]

It is indeed! Like Andreas mentioned, any global DeDupe implementation has a penalty relative to cost and performance. The trade off is something that needs to be considered.

[evilaedmin]

For those using ZFS as a repository, is it common to use as a volume manager in place of hardware raid also? Any special tips regarding ZIL or ARC, RAID-Zx vs Mirror?

[anthonyspiteri79]

Now you are getting into the nitty gritty of ZFS architecture and design. I would encourage doing a lot of research online on the subject, specially when it comes to RAID-Z design as this will be the most impactful decision you make.

My only bit of advice is to consider re-build times and the impact that has on the storage performance. The deeper you have your RZ pools the longer a rebuild will take. The larger the disks being used, the longer a rebuild will take.

[danswartz]

I use ZFS on a CentOS 7 storage server for my veeam backups. You want to disable compression on the filesystem that veeam will write to (it's harmless but a waste of cycles to do otherwise.) Also, set primarycache=metadata on that filesystem, since you don't want bulk files going to it from veeam to pollute your ARC. As others have said, stay far away from dedup - it's fraught with peril.

[danswartz]

Also, for the other poster: some flavor of raidz *should* be okay, since veeam is mostly doing sequential I/O (as I understand it). raidz flavors tend to do poorly with random I/O, due to the architecture. As far as ZIL goes, you definitely want either a very low latency SSD (optane?) or disable sync entirely on that filesystem, otherwise your write performance will be absolutely dreadful. Be aware though, that pool topology is for the entire pool, so you can't just make a raidz* filesystem, and depending on what (if anything) else uses that pool, raidz* might not be a good choice for those other clients of the storage.

[nitramd]

@evilaedmin,

Check out the FreeNAS worst practices guide: http://www.freenas.org/blog/freenas-worst-practices/ ; it should be useful.

You might also consider perusing the FreeNAS forum: https://forums.freenas.org/index.php

Hope this helps.