Feature Request: BTRFS or OpenZFS Backup Repository

[vt_lyndon]

Hi,

We're in a scenario with our XFS backup repositories where our ideal chain looks like this:

Primary Site Repository -> Backup Site Edge Repository -> Backup Site Bubble Repository

We do semi-frequent DR testing and like to keep backup assurance during these tests. To this end, we'd like to send to one repository then from there to the next. This isn't very well supported within Veeam itself, so we originally looked to rsync but using rsync uses a ton more space - we lose block cloning.

An ideal solution is to create the repositories using ZFS or BTRFS, and have a frequent block-level send over the network.

As I understand it, Veeam currently uses reflink for block cloning. BTRFS has existing capabilities for reflink and the efficient send/receive capabilities we're looking for. My personal preference would be to have Veeam use zfs snapshots or clone-copies on the backup repository over reflink, but any solution which solves our problem would be a welcome one.

Thanks for reading!

Lyndon

[tsightler]

Hi @vt_lyndon.

While it would be great if we could just support everything, it's a very difficult and expensive proposition to support all possible options out there for Linux, just supporting mulitple distros is a significant QA burden and adding even a single additional filesystem effectively doubles the existing QA effort.

When we were looking at support block clone on Linux, there were several filesystem options available, but only the XFS filesystem has the combination of stability and wide support. XFS is proven to scale to volumes up to 1PB and provides a very solid foundation on which to build a repository and, most importantly, is supported on all major distros.

BTRFS is great, but it simply has failed to fully reach the same status of reliability as XFS so far, although I do believe it continues to get closer and closer. However, one big strike against it is that it is no longer considered a supported filesystem on RHEL based distros, and that's a huge issue since such a high percentage of customers run RHEL based distros exclusively, it means the QA efforts there will only ever apply to a smaller subset of customers.

ZFS, well, that's another issue entirely. As far as I know it is only officially included and supported on Ubuntu, although certainly you can get it to work on others as well. It does not have the required block clone features implemented and, while it is somewhat popular with users, it currently has effectively zero percent chance of being a first class citizen in the Linux kernel, which will mean that it will always be this external thing supported by a subset of distros.

However, if you really like ZFS, then it may very well still be the best answer to your requirement as it's completely possible to use XFS on top of ZFS via the use of ZVOLS. Basically, this is what storage appliances like TrueNAS, which leverage ZFS underneath, actually do, you create a ZFS volume as normal, then create ZVOLs and then format those ZVOLS with XFS. You can then use snapshots and incremental send/receive to incrementally replicate volumes at the block level.

I'm also curious what issues you are having with Veeam and this setup as I believe it should work, at least, I know I've helped customers setup this scenario in the past, but it's admittedly been quite a while.

[vt_lyndon]

Good to know - thank you for the detailed response! I had considered XFS on a ZVOL initially but ultimately decided against it to ensure we're on a supported configuration, perhaps I should revisit this if it's actually being used in the wild.

Re: my issues, the "backup copy job from a backup copy job" logic doesn't seem to work too well. I can configure it to start working and it kicks off, but it doesn't seem to stay healthy. If, for example, we rename or restore a server it won't find it's way into the final repository.

[dandav]

@tsightler,

I understand your hesitancy to rely on BTRFS or ZFS due to their limited support, but there are some things that I think need to be explained to customers before they implement XFS reflink though. XFS has no data rot detection mechanism built in, so when forever incremental encrypted backups are stored on an XFS volume there is risk that a single flipped bit can result in an entire backup chain becoming unusable. Because the initial data is never re-written and some blocks in the chain may NEVER be changed, the chance of this increases with time. Without reflink enabled the synthetic fulls are re-written to disk fairly regularly so data is always fairly fresh and you increase the chance of a drive error being picked up before it causes issues, also a flipped bit will only affect a small subset of the backups in the chain. Suggesting XFS on top of ZVOLS with regular scrubbing is a good way to mitigate the issue, however then you may as well just go all out with ZFS reflink support.

Obviously everyone should have other copies of their backup data, however data rot happens, usually it's not a big problem, a word document ends up with bad formatting or a JPEG looks a bit weird in one corner but when we're talking about storing deduplicated, forever-forward, compressed, encrypted backups it is a real problem.

[mkretzer]

Isn't the CRC option that is needed for XFS repos + Veeam health checks exactly what you mean by "bit rot detection"?

[dandav]

Sorry, I worded it incorrectly that should have read "data rot correction" as yes, it can be detected, just the filesystem has no way of correcting it. Hence why I think there needs to be some kind of warning in the user guide warning users that if they store data on a repository with reflinks enabled that they MUST use backup file health checks.

[mkretzer]

The same goes for ReFS on simple volumes. Scrubbing does nothing in such situations!

[soncscy]

Block cloning technology, for me at least, natively implies redundancy elsewhere. While it's amazingly fast and allows you to size overall smaller servers, I believe it's plain as day that it's very much so an "all eggs in one basket" situation. Anyone who walks away from reading about reflinks/block cloning and just understands "free space" needs to read again until they understand the cost involved.

To be clear, I'm heavily an advocate to my clients for XFS/ReFS, but we also push redundancy. Even without block cloning the same risks apply; I'm willing to commit to any maths on it, but it's basically the same risk in my opinion -- will a given backup be hit by data corruption?

For me, since the risk is relatively the same, the difference is that without block cloning, you spend far more space and have much slower random IO operations. if the risk is relatively the same, but you get benefits with block cloning, I'd suggest why not? Proper administrators will be performing the correct checks and ensuring they have proper redundancy as well. Those who are not able to afford it at least can make an informed decision and understand that they will be unprotected in certain situations.

[tsightler]

XFS has no data rot detection mechanism built in, so when forever incremental encrypted backups are stored on an XFS volume there is risk that a single flipped bit can result in an entire backup chain becoming unusable.

While I agree that it is possible for this to happen, I do think the risks are very low if you also combine XFS with with RAID scrubbing and health checks. The reason is that, even if Veeam itself never reads the block again, when using disk scrubbing, the block will be read from disk. All modern drives that I know of detect and correct single bit errors themselves, including rewriting the impacted block, so as long as you have a scrubber process reading the data consistently, single bit errors will be corrected at the drive level before there even a detectable problem. Now, perhaps there are multiple bit flips in a single block, that would produce an unrecoverable error, but then RAID kicks in and reads the data from another drive.

RAID scrubbers will also detect minor inconsistencies and , in some cases, rewrite stripes (exact capabilities vary per vendor) but note that most won't actually correct the data, they are simply correcting the parity. Linux RAID had both "check" which just reports stripe inconsistencies, and "repair", which just rewrites the stripe with new parity (or using the first readable data in the case of RAID1). Note that, if using Linux based RAID6, you should almost certainly not use "repair", but rather perform a "check" scrub and, if you receive any errors you can use a tool like raid6check to actually determine which component has the bad data.

Now, don't get me wrong, I'm not saying that the data protection features in ZFS/BTRFS are not nice, indeed having automatic recovery from such issues is really nice, but, to be fair, both ZFS/BTRFS recommend scrubs as well because that's where 99% of errors will be fixed, not different than with basic RAID. Basically, my suggestion is always "scrub early, scrub often, scrub using all available methods". Oh, and always have a second copy (or more)!

Suggesting XFS on top of ZVOLS with regular scrubbing is a good way to mitigate the issue, however then you may as well just go all out with ZFS reflink support.

But, unless something has changed very recently, ZFS doesn't support reflink, because ZFS maintainers seem to continue to think that reflink is the same as dedupe. So XFS on ZVOL is the only way to get reflink with ZFS data checks, as far as I know. Feel free to provide references otherwise. And don't get me wrong, I know you can do filesystem style snapshots of files, which provides a somewhat similar level of functionality to reflink, basically, snapshot the file, then modify the file in place, but this is a completely different approach that introduces a whole share of challenges.

Now, BTRFS is different, it does support reflink, so it's really about supportability. BTRFS continues to improve in stability and is really looking quite good in modern kernels, but supportability is still an issue today. We may support BTRFS one day, but it needs to be a supported and recommended choice by our Linux partners.

[jeremybsmith]

@vt_lyndon - I am using the XFS on ZFS method to provide snapshot and replication capabilities and could share my experiences if you are curious. It has been working well on an 80 TiB ZVOL/XFS filesystem but there are some caveats and unknowns as to the performance difference versus direct XFS and how well it will be able scale.
I'm doing something similar to Tom's Ultimate Linux Repository <https://drive.google.com/file/d/1MwzmH5 ... gvAOF/view> but using a LXC container (with the Veeam persistent data mover service) rather than Docker.

I would also love to be able to remove the ZVOL and XFS dependencies and simply use native ZFS (or even BTRFS since it already supports reflink).
There is interest in implementing this -- see https://github.com/openzfs/zfs/issues/405 -- but as Tom mentioned, the devs aren't quite there yet.

@tsightler - As for Veeam's support (or lack thereof) of reflink on BTRFS, do you know if that is hard-coded to only support XFS or does it specifically check whether the filesystem supports that functionality? Even if Veeam doesn't officially support it (yet), I would like to be able to at least test it. I would hope that it queries the filesystem so that when other filesystems do have support, it will be available immediately.

[Gostev]

It is hard-coded to only support XFS.

[michaelobrien]

Looks like the ZFS block reference table (https://www.youtube.com/watch?v=hYBgoaQC-vo) is making progress, reading into latest OpenZFS Leadership Team notes: https://docs.google.com/document/d/1w2j ... ZAnWHhV-BM

This provides ZFS an XFS-like reflinks capability. As I understand it, the capability should use the same linux syscalls as XFS.

Given all of this, couple of questions:

1. Does anyone know if Veeam team has this on their radar?
2. Given it uses the same syscalls as XFS, is there a way to test this internally (on non-production datasets, of course) once the capability launches?

[HannesK]

Hello,
1. yes, please see above
2. no without changing the code, please see above

Best regards,
Hannes

[michaelobrien]

Thanks @HannesK. Looking forward to being able to test this some day

[ashleyw]

late to the party here. but I've been doing some interesting experiments. We've been using ZFS for many years as a Veeam repository but need the reflinks capability for quick weekly synthetics (which as you know is not available under ZFS).

The only way I can see to layer on an XFS file system on top of ZFS is to use a loop device on top of the ZFS file system and then format that loop device as an xfs file system.
This actually worked but under full Veeam loads the throughput is slow and the choke point appears to be the loop device itself.
So right now we are testing with Centos8 Stream using Cockpit as a webui to make the admin of the box simple and we are using 20x8TB spindles in a raid 10 configuration.
My initial observations;
- The performance knock of using an XFS file system mounted via a loopback device on a ZFS file makes it unsuitable for high loads such as Veeam. (this was what I found when I ran tests using TrueNAS Scale and creating an XFS mount from the CLI of TrueNAS, you can't do it natively and you have to inject the Veeam components into the appliance which makes it problematic).
- performance of XFS natively in a VM (ESXI7u3g) with PCI passthrough of the controller is great.
- OpenZFS has inbuilt l2cache via the l2arc and ram caching which appears to be superior to Linux.
- OpenZFS serialises the writes making it ideal for high write workloads.
- OpenZFS is unlikely to ever get native reflink support IMHO as it appears to be bogged down by concerns around Oracle owner technology.
- OpenZFS has protection against bitrot.
- The XFS backups jobs are running at about 500MB/s write throughput with about 100MB/s read throughput (On active fulls), but there are parts of the backup process where the read/write patterns are reversed - presumably as part of the de-dupe architecture.

Maybe I'm missing something here.
But from my perspective, the biggest bang for the buck would be if Veeam were some how able to implement the weekly synthetic fulls using an alternative to reflinks for those file systems unable to support it.
Maybe I'm oversimplifying/misunderstanding things, but all a reflink is is a pointer to an existing file. So why can't the weekly synthetic files have some sort of abstracted container which just refers to the original files on the filesystem to get similar functionality albeit in a much more convoluted way.

[ashleyw]

From re-reading the ZFS documentation and running a few tests I was able to create an XFS file system directly on a zpool, which is great - so I'll run some test in our next window period and compare the performance against XFS natively; (below is what I did on a test VM with 5 disks with a raid10 equivalent (stripe across mirror pairs using 4 disks with 1 hot spare).;
Definitely a case of ZFS for the win if I can get close to the XFS native performance.

Code: Select all

# zpool create Test1 mirror sdb sdc mirror sdd sde spare sdf
# zpool list
NAME    SIZE  ALLOC   FREE  CKPOINT  EXPANDSZ   FRAG    CAP  DEDUP    HEALTH  ALTROOT
Test1   199G   135K   199G        -         -     0%     0%  1.00x    ONLINE  -
# zfs create -V 186G Test1/Backups
# mkfs.xfs -b size 4096 -m reflink=1,crc=1 /dev/zvol/Test1/Backups
# mkdir /mnt/Backups
# mount -t xfs /dev/zvol/Test1/Backups /mnt/Backups
# xfs_info /mnt/Backups
# echo "/dev/zvol/Test1/Backups /mnt/Backups xfs defaults" >> /etc/fstab
# mount -a

[HannesK]

to implement the weekly synthetic fulls using an alternative to reflinks for those file systems unable to support it.

almost done... object storage is the way to go

[ashleyw]

Well, I managed to reconfigure one of our development backup units to layer XFS on top of a ZFS storage pool but this time using ZFS block devices instead of a loopback device, and it's exceeded all expectations!
We are pulling a consistent 800MB/s throughput (highest we've ever seen), and we are seeing the CPU on our virtualised storage instance maxing out but with iostat only showing about 33% busy state on the spindles - so we are going to throw some more virtual cores over the next few weeks and switch our proxies to Linux so we can really sweat the tin and get maximum value while giving us complete flexibility over the configuration using open license free Linux based storage solution.

In-case others are interested I've pasted some info below;

Primary Storage for farm: EMC Unity all flash array.
Primary compute: 8 x Dell 1.2TB RAM blades (running ESXi 7u3g) - fibre connected directly to Unity. Workload is around 500 VMs mix of Windows/Linux.
BackupUnit: 1 x Dell PowerEdge R740 xd2 with 23x8TB spindles - fibre connected directly to Unity (running ESXi 7u3g). (128GB RAM)
IP Network layer for primary compute and BackupUnit goes through Cisco 25Gb core I believe but we rather use hotadd via storage layer for backups.
BackupUnit is running VMware ESXi 7u3g and also acts as our vcentre host.
The backup unit runs the following VMs;
- vCentre server: 4vCPU,24GB RAM
- Veeam Backup server: Windows 2019 server (6vCPU, 24GB RAM)
- 3 x Veeeam Proxies: Windows 2019 Server (each 4vCPU, 8GB RAM)
- Storage server: CentOS8 Stream (4vCPU, 32GB RAM with OS disk on 8GB vmdk on EMC LUN, and 2nd vmdk on EMC LUN of 128GB), Dell HBA330 and using PCI pass-through from esxi host to CentOS VM with 23x8TB spinners. ZFS needs to be installed using dkms for rebuilding package after kernel updates.
- We manage the storage server using https://cockpit-project.org/
- We use a plugin for ZFS on cockpit called; https://github.com/45Drives/cockpit-zfs-manager/
- We configure the zpool to use 22 spindles (in a 2x11vdev mirrored pair configuration), with an additional drive as a hot spare.
- We configure the 128GB vmdk as the L2ARC on the zpool.
- I created the zpool from command line as the UI would have taken me too long getting all the pairs right in the UI!

Code: Select all

# zpool create BackupPool mirror sdc sdd mirror sde sdf mirror sdg sdh mirror sdi sdj mirror sdk sdl mirror sdm sdn mirror sdo sdp mirror sdq sdr mirror sds sdt mirror sdu sdv mirror sdw sdx spare sdy cache sdb

and then check it was created as expected;

Code: Select all

# zpool status
  pool: BackupPool
 state: ONLINE
config:

        NAME         STATE     READ WRITE CKSUM
        BackupPool   ONLINE       0     0     0
          mirror-0   ONLINE       0     0     0
            sdc      ONLINE       0     0     0
            sdd      ONLINE       0     0     0
          mirror-1   ONLINE       0     0     0
            sde      ONLINE       0     0     0
            sdf      ONLINE       0     0     0
          mirror-2   ONLINE       0     0     0
            sdg      ONLINE       0     0     0
            sdh      ONLINE       0     0     0
          mirror-3   ONLINE       0     0     0
            sdi      ONLINE       0     0     0
            sdj      ONLINE       0     0     0
          mirror-4   ONLINE       0     0     0
            sdk      ONLINE       0     0     0
            sdl      ONLINE       0     0     0
          mirror-5   ONLINE       0     0     0
            sdm      ONLINE       0     0     0
            sdn      ONLINE       0     0     0
          mirror-6   ONLINE       0     0     0
            sdo      ONLINE       0     0     0
            sdp      ONLINE       0     0     0
          mirror-7   ONLINE       0     0     0
            sdq      ONLINE       0     0     0
            sdr      ONLINE       0     0     0
          mirror-8   ONLINE       0     0     0
            sds      ONLINE       0     0     0
            sdt      ONLINE       0     0     0
          mirror-9   ONLINE       0     0     0
            sdu      ONLINE       0     0     0
            sdv      ONLINE       0     0     0
          mirror-10  ONLINE       0     0     0
            sdw      ONLINE       0     0     0
            sdx      ONLINE       0     0     0
        cache
          sdb        ONLINE       0     0     0
        spares
          sdy        AVAIL

errors: No known data errors

I then created the largest volume I could on that pool;

Code: Select all

zfs create -V 77.2T BackupPool/Backups

and then the largest xfs filesystem on the entire volume;

Code: Select all

mkfs.xfs -b size 4096 -m reflink=1,crc=1 /dev/zvol/BackupPool/Backups

then mount the file system;

Code: Select all

mkdir /mnt/Backups

Code: Select all

mount -t xfs /dev/zvol/BackupPool/Backups /mnt/Backups

and mount at boot time;

Code: Select all

echo "/dev/zvol/BackupPool/Backups /mnt/Backups xfs defaults" >> /etc/fstab

so we now have an xfs file system mounted under a directory /mnt/Backups that can be used as a target fro Veeam.
And because XFS is layered on top of ZFS we get all the benefits of ZFS reliability and throughput along with the xfs reflinks feature.

Code: Select all

# xfs_info /mnt/Backups
meta-data=/dev/zd0               isize=512    agcount=78, agsize=268435455 blks
         =                       sectsz=512   attr=2, projid32bit=1
         =                       crc=1        finobt=1, sparse=1, rmapbt=0
         =                       reflink=1    bigtime=0 inobtcount=0
data     =                       bsize=4096   blocks=20723217204, imaxpct=1
         =                       sunit=0      swidth=0 blks
naming   =version 2              bsize=4096   ascii-ci=0, ftype=1
log      =internal log           bsize=4096   blocks=521728, version=2
         =                       sectsz=512   sunit=0 blks, lazy-count=1
realtime =none                   extsz=4096   blocks=0, rtextents=0

IOstats are showing quite high utilisaiton but we believe we can pump more through it with a few adjustments;

Code: Select all

# iostat -x
Linux 4.18.0-408.el8.x86_64        13/09/22        _x86_64_        (4 CPU)

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           9.73    0.00   56.09    8.99    0.00   25.18

Device            r/s     w/s     rkB/s     wkB/s   rrqm/s   wrqm/s  %rrqm  %wrqm r_await w_await aqu-sz rareq-sz wareq-sz  svctm  %util
sda              1.66    1.27     35.48     24.61     0.06     2.29   3.71  64.35    0.45    1.44   0.00    21.44    19.42   0.53   0.15
sdb             13.59  506.94    179.23  62591.04     0.00     0.00   0.00   0.00    0.84    1.31   0.68    13.18   123.47   0.67  34.91
scd0             0.00    0.00      0.00      0.00     0.00     0.00   0.00   0.00    0.00    0.00   0.00     0.23     0.00 94610.82   4.87
sdd              1.56  252.78     13.05  38726.48     0.00     1.58   0.00   0.62    5.25    5.29   1.34     8.36   153.20   1.36  34.48
sdc              1.59  259.70     13.26  38726.76     0.00     1.67   0.01   0.64    5.61    4.86   1.27     8.32   149.12   1.25  32.75
sde              1.55  258.37     12.85  38611.30     0.00     1.70   0.00   0.65    5.46    4.88   1.27     8.31   149.44   1.26  32.76
sdf              1.54  251.30     12.81  38611.26     0.00     1.72   0.00   0.68    5.51    5.34   1.35     8.31   153.65   1.36  34.51
sdg              1.56  259.59     13.00  38703.24     0.00     1.95   0.00   0.74    5.34    4.85   1.27     8.32   149.09   1.25  32.71
sdh              1.59  253.28     13.26  38702.59     0.00     1.62   0.00   0.64    5.35    5.21   1.33     8.36   152.81   1.35  34.44
sdj              1.56  253.71     13.11  38713.14     0.00     1.58   0.00   0.62    5.40    5.20   1.33     8.40   152.59   1.35  34.50
sdi              1.54  259.76     12.94  38713.18     0.00     1.92   0.01   0.73    5.42    4.85   1.27     8.40   149.03   1.25  32.72
sdk              1.57  259.36     13.10  38726.71     0.00     1.84   0.00   0.71    5.35    4.90   1.28     8.32   149.32   1.25  32.72
sdl              1.51  253.21     12.65  38726.67     0.00     1.57   0.00   0.62    5.34    5.24   1.33     8.35   152.94   1.35  34.48
sdm              1.55  258.36     12.85  38676.53     0.00     1.91   0.00   0.73    5.43    4.92   1.28     8.27   149.70   1.26  32.78
sdp              1.53  251.71     12.80  38665.17     0.00     1.56   0.00   0.62    5.42    5.26   1.33     8.36   153.61   1.36  34.48
sdq              1.53  259.54     12.76  38706.02     0.00     1.74   0.00   0.67    5.52    4.83   1.26     8.37   149.13   1.25  32.75
sdr              1.56  252.18     13.11  38706.03     0.00     1.63   0.01   0.64    5.30    5.29   1.34     8.39   153.48   1.36  34.51
sds              1.58  258.82     13.30  38736.59     0.00     1.84   0.00   0.70    5.56    4.89   1.28     8.39   149.67   1.26  32.77
sdu              1.56  258.53     13.00  38703.66     0.00     1.68   0.00   0.65    5.26    4.92   1.28     8.35   149.71   1.26  32.70
sdt              1.57  253.19     13.20  38736.30     0.00     1.58   0.00   0.62    5.43    5.20   1.33     8.40   153.00   1.35  34.47
sdw              1.59  258.83     13.27  38700.35     0.00     1.81   0.00   0.69    5.19    4.84   1.26     8.34   149.52   1.26  32.75
sdv              1.58  251.97     13.23  38703.68     0.00     1.53   0.00   0.60    5.30    5.31   1.35     8.38   153.60   1.36  34.48
sdx              1.56  252.47     13.11  38700.39     0.00     1.58   0.00   0.62    5.19    5.24   1.33     8.40   153.28   1.36  34.45
sdy              0.01    0.00      0.24      0.00     0.00     0.00   0.00   0.00    6.89    4.50   0.00    19.22     8.00   5.77   0.01
sdn              1.53  252.73     12.71  38676.61     0.00     1.53   0.00   0.60    5.33    5.23   1.33     8.31   153.04   1.36  34.50
sdo              1.60  258.35     13.23  38665.13     0.00     1.78   0.00   0.69    5.26    4.89   1.27     8.30   149.66   1.26  32.77
zd0              0.05  870.02     12.91 288546.05     0.00     0.00   0.00   0.00   18.61    0.64   0.56   273.89   331.65   0.65  56.88

* I've been doing PCI pass-through under ESXi for many years in both work and at Home for virtualised gaming systems with GPU pass-through for my kids (and very recently to get Google Coral AI USB devices working under ESXi). Yet again the technology appears perfectly suited to this type of use case to create a cost effective out of band backup and management server.

[KoflerTx]

OpenZFS 2.2 now support block cloning like XFS.

Would be awesome to get support from Veeam. Together with the other ZFS features like snapshotting and replication would lift it on a new resistance level.

[tyler.jurgens]

OpenZFS supporting block cloning has been brought up before here: veeam-backup-replication-f2/openzfs-2-2 ... 90517.html

Very interesting read (especially if you dive into the linked GitHub rabbit hole). Not something I'd readily use, but neat to see.

[HannesK]

Hello,
yes, please continue for OpenZFS in veeam-backup-replication-f2/openzfs-2-2 ... 90517.html and I answered there.

I leave this topic open for BTRFS, but I don't expect too many BTRFS requests anymore if OpenZFS reflink works

Best regards,
Hannes