Host-based backup of VMware vSphere VMs.
Post Reply
andyg
Enthusiast
Posts: 58
Liked: 5 times
Joined: Apr 23, 2014 9:51 am
Full Name: Andy Goldschmidt
Contact:

Linux repository using ZFS

Post by andyg »

Anyone using Linux as a repository with ZFS as the filesystem?

How have you found it in production? Any good or bad points you have noticed?

Did you use RAID, or present each of the disks and let ZFS handle them?
-= VMCE v9 certified =-
Gostev
Chief Product Officer
Posts: 31455
Liked: 6646 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Linux repository using ZFS

Post by Gostev »

I found ZFS to be very popular choice among our service providers, and rarely heard anything but praise from its users.
evilaedmin
Expert
Posts: 176
Liked: 30 times
Joined: Jul 26, 2018 8:04 pm
Full Name: Eugene V
Contact:

Re: Linux repository using ZFS

Post by evilaedmin »

Does the Veeam file storage format work well with the native compression and/or deduplication features of ZFS?
nitramd
Veteran
Posts: 297
Liked: 85 times
Joined: Feb 16, 2017 8:05 pm
Contact:

Re: Linux repository using ZFS

Post by nitramd »

What file storage format are you referring to?
csydas
Expert
Posts: 193
Liked: 47 times
Joined: Jan 16, 2018 5:14 pm
Full Name: Harvey Carel
Contact:

Re: Linux repository using ZFS

Post by csydas »

Hi Evilaedmin,

Data is data in *nix. It doesn't mess around, and as long as you're prepping the backups for a deduplication repository (and your repo has ample memory), you should see no issues. I use ZFS for one of our primary repositories at a site, and I'm quite happy with it. A cronjob to trigger snapshots every so often makes me feel way better about the backup files.
Andreas Neufert
VP, Product Management
Posts: 6707
Liked: 1401 times
Joined: May 04, 2011 8:36 am
Full Name: Andreas Neufert
Location: Germany
Contact:

Re: Linux repository using ZFS

Post by Andreas Neufert »

As Veeam do compression on the Source Side Datamover (Proxy or Agent), the storage system need to handle only half the data which speed up things.

If you enable compression on the Storage and disable it on the Veeam side, your Storage need to handle 2x more data.
So best practices is to use Veeam Compression.

Double Compression will slow down things and usually produce bigger backup files. So not recommended.
anthonyspiteri79
Veeam Software
Posts: 741
Liked: 207 times
Joined: Jan 14, 2016 6:48 am
Full Name: Anthony Spiteri
Location: Perth, Australia
Contact:

Re: Linux repository using ZFS

Post by anthonyspiteri79 » 1 person likes this post

From a dedupe point of view with ZFS, make sure you understand the hardware and performance overheads that are required when you turn it on. It's got a heavy RAM overhead which is why we decided not to turn it on. It can be pushed into L2ARC but that comes with its own costs.

This post explains it well. https://constantin.glez.de/2011/07/27/z ... ot-dedupe/
RAM Rules of Thumb

For every TB of pool data, you should expect 5 GB of dedup table data, assuming an average block size of 64K.

This means you should plan for at least 20GB of system RAM per TB of pool data, if you want to keep the dedup table in RAM, plus any extra memory for other metadata, plus an extra GB for the OS.
Anthony Spiteri
Regional CTO APJ & Lead Cloud and Service Provider Technologist
Email: anthony.spiteri@veeam.com
Twitter: @anthonyspiteri
Andreas Neufert
VP, Product Management
Posts: 6707
Liked: 1401 times
Joined: May 04, 2011 8:36 am
Full Name: Andreas Neufert
Location: Germany
Contact:

Re: Linux repository using ZFS

Post by Andreas Neufert »

Global deduplication usually has a performance impact on the backup and restore and is used inline with the Veeam best practices as secondary backup target for long term backups usually.
evilaedmin
Expert
Posts: 176
Liked: 30 times
Joined: Jul 26, 2018 8:04 pm
Full Name: Eugene V
Contact:

Re: Linux repository using ZFS

Post by evilaedmin »

csydas wrote: Sep 26, 2018 5:39 pm Hi Evilaedmin,

Data is data in *nix. It doesn't mess around, and as long as you're prepping the backups for a deduplication repository (and your repo has ample memory), you should see no issues. I use ZFS for one of our primary repositories at a site, and I'm quite happy with it. A cronjob to trigger snapshots every so often makes me feel way better about the backup files.
Hi csydas,

One counterexample to your statement that immediately comes to mind is the Dell/EMC Networker on-disk storage format: When not using Data Domain appears to be by design "misaligned" with ZFS blocks and ZFS deduplication has nearly zero effect. I do appreciate all responses to my query.
evilaedmin
Expert
Posts: 176
Liked: 30 times
Joined: Jul 26, 2018 8:04 pm
Full Name: Eugene V
Contact:

Re: Linux repository using ZFS

Post by evilaedmin »

That is...a lot of RAM.
anthonyspiteri79 wrote: Sep 27, 2018 2:03 am From a dedupe point of view with ZFS, make sure you understand the hardware and performance overheads that are required when you turn it on. It's got a heavy RAM overhead which is why we decided not to turn it on. It can be pushed into L2ARC but that comes with its own costs.

This post explains it well. https://constantin.glez.de/2011/07/27/z ... ot-dedupe/
anthonyspiteri79
Veeam Software
Posts: 741
Liked: 207 times
Joined: Jan 14, 2016 6:48 am
Full Name: Anthony Spiteri
Location: Perth, Australia
Contact:

Re: Linux repository using ZFS

Post by anthonyspiteri79 »

It is indeed! Like Andreas mentioned, any global DeDupe implementation has a penalty relative to cost and performance. The trade off is something that needs to be considered.
Anthony Spiteri
Regional CTO APJ & Lead Cloud and Service Provider Technologist
Email: anthony.spiteri@veeam.com
Twitter: @anthonyspiteri
evilaedmin
Expert
Posts: 176
Liked: 30 times
Joined: Jul 26, 2018 8:04 pm
Full Name: Eugene V
Contact:

Re: Linux repository using ZFS

Post by evilaedmin »

For those using ZFS as a repository, is it common to use as a volume manager in place of hardware raid also? Any special tips regarding ZIL or ARC, RAID-Zx vs Mirror?
anthonyspiteri79
Veeam Software
Posts: 741
Liked: 207 times
Joined: Jan 14, 2016 6:48 am
Full Name: Anthony Spiteri
Location: Perth, Australia
Contact:

Re: Linux repository using ZFS

Post by anthonyspiteri79 »

Now you are getting into the nitty gritty of ZFS architecture and design. I would encourage doing a lot of research online on the subject, specially when it comes to RAID-Z design as this will be the most impactful decision you make.

My only bit of advice is to consider re-build times and the impact that has on the storage performance. The deeper you have your RZ pools the longer a rebuild will take. The larger the disks being used, the longer a rebuild will take.
Anthony Spiteri
Regional CTO APJ & Lead Cloud and Service Provider Technologist
Email: anthony.spiteri@veeam.com
Twitter: @anthonyspiteri
danswartz
Veteran
Posts: 264
Liked: 30 times
Joined: Apr 26, 2013 4:53 pm
Full Name: Dan Swartzendruber
Contact:

Re: Linux repository using ZFS

Post by danswartz »

I use ZFS on a CentOS 7 storage server for my veeam backups. You want to disable compression on the filesystem that veeam will write to (it's harmless but a waste of cycles to do otherwise.) Also, set primarycache=metadata on that filesystem, since you don't want bulk files going to it from veeam to pollute your ARC. As others have said, stay far away from dedup - it's fraught with peril.
danswartz
Veteran
Posts: 264
Liked: 30 times
Joined: Apr 26, 2013 4:53 pm
Full Name: Dan Swartzendruber
Contact:

Re: Linux repository using ZFS

Post by danswartz »

Also, for the other poster: some flavor of raidz *should* be okay, since veeam is mostly doing sequential I/O (as I understand it). raidz flavors tend to do poorly with random I/O, due to the architecture. As far as ZIL goes, you definitely want either a very low latency SSD (optane?) or disable sync entirely on that filesystem, otherwise your write performance will be absolutely dreadful. Be aware though, that pool topology is for the entire pool, so you can't just make a raidz* filesystem, and depending on what (if anything) else uses that pool, raidz* might not be a good choice for those other clients of the storage.
nitramd
Veteran
Posts: 297
Liked: 85 times
Joined: Feb 16, 2017 8:05 pm
Contact:

Re: Linux repository using ZFS

Post by nitramd »

@evilaedmin,

Check out the FreeNAS worst practices guide: http://www.freenas.org/blog/freenas-worst-practices/ ; it should be useful.

You might also consider perusing the FreeNAS forum: https://forums.freenas.org/index.php

Hope this helps.
Post Reply

Who is online

Users browsing this forum: Bing [Bot], Ivan239 and 91 guests