OpenZFS 2.2 support for reflinks now available

[ashleyw]

hi,

For years we were running OpenZFS and really enjoy its benefits. With Veeam we've previously had to overlay XFS onto a zvol in-order to getthe COW reflink support.
We just see this week that OpenZFS 2.2 has been released with native support for reflinks; https://github.com/openzfs/zfs/releases

This feature has been more than 10 years in the making of OpenZFS and changes things significantly in terms of Veeam.

If we switched from XFS on md raid10 dataset back to OpenZFS 2.2, has Veeam been tested on a native OpenZFS 2.2 file system with specific regards to block cloning and reflink support?
Ironically we switched from OpenZFS to md raid set only two days ago to get to the bottom of a long standing performance issue - which in the end turned out to be related to needing to disable NCQ on the drives rather than a problem with OpenZFS in general.

* I do have access to a lab where I can attempt to test this myself but I'd need to simulate this using virtual drives rather than physical drives and it would take me quite a few hours to create a complete simulation of this without access to tin.

[HannesK]

Hello,
no, it's untested / unsupported today. I will check with my colleagues how complicated it would be to support it. Then the second question would be, how stable it is. REFS had many issues in the beginning, we still remember that

Best regards,
Hannes

[ashleyw]

thanks,
Just to help speed things along a little, I was able to set up a quick test using our VMware cluster.

I provisioned a minimal install of Rocky Linux 8.8 - 2CPU, 16GB ram, 8GB primary disk, 300GB secondary disk, single NIC - all thin provisioned.

* install OpenZFS 2.2 release;

Code: Select all

# dnf install https://zfsonlinux.org/epel/zfs-release-2-3$(rpm --eval "%{dist}").noarch.rpm
# dnf config-manager --enable zfs-testing
# dnf install -y epel-release
# dnf install -y kernel-devel
# dnf install -y zfs
# reboot

* check ZFS version

Code: Select all

# zfs --version
zfs-2.2.0-1

* create a single zpool using the 2nd disk (dev/sdb)

Code: Select all

# zpool create VeeamBackup /dev/sdb
# reboot

* check zpool is mounted after a reboot - which it should be;

Code: Select all

# df -h|grep VeeamBackup
VeeamBackup     289G  128K  289G   1% /VeeamBackup

* Go into Veeam, create a new linux repository, and map that server using a backup folder of; /VeeamBackup/backups
When you navigate to that part of ht eVeeam UI, you get a message saying;
"File System behind the path is not XFS: /VeeamBackup/backups"

So I guess a quick test shows that Veeam is likely checking the file system type rather than a general test to see if the reflinks feature is available on the file system in question.
I'd rather see a message saying something like; "Warning: reflink support is detected, but file system is not XFS"

It would be great if you could have a chat with the team.

thanks
Ashley

[HannesK]

Hello,
thanks for the details. We are working on the 12.1 releases and adding a new file system with reflink support needs proper testing. We will not add ZFS reflink support "over night". I assume you are also interested to have your data stored safely

Yes, the software checks both... XFS & reflink.

Best regards,
Hannes

[biohazrd]

With the release of ZFS 2.2, block cloning is now supported. This means that, along with ReFS and XFS, ZFS would be able to create synthetic fulls using fast clone if implemented.

Being able to use triple parity and/or distributed RAID would be a fantastic addition for petabyte scale repositories where traditional RAID is not feasible.

We're already using ZVOLs formated with XFS on ZFS and it works well enough, however there are significant performance limitations due to the necessity of read-modify-write that comes with a mismatched pool recordsize combined with the ZVOL block size. Bypassing the need for a ZVOL by having Veeam directly support the ZFS file system would solve this issue.

Hopefully this is considered. Thanks.

[Gostev]

I can tell you that many folks at Veeam (including me) are truly excited about this in theory, but some (including me) are really afraid of the repeat of ReFS block cloning woes which took years to stabilize and costed us some customers. Jumping on a brand new block cloning capability just seem like stepping on the same rake again.

Question: would you personally use such integration in production environment if we had this feature in the experimental support status for its initial release?

[chris.childerhose]

I think you could probably get quite a few folks that would test this, maybe not so much in production, but I would give it a full run-through in my lab to give feedback to Veeam.

[biohazrd]

I'd agree that production might be a hard sell, but at the same time we're using SOBRs and pushing copies to S3 compatible storage. It might not be infeasible to test this out for the local copy.

I guess part of the question is when the feature might arrive. If this is going to be a large undertaking that might arrive with, say, v13. Then I'd be more confident that the bugs on the ZFS side have been worked out.

[tdewin]

To just show you what @Gostev means that we are concerned since it is a new feature, just look at the latest release :

Note: We recommend that anyone running zfs-2.2.0 to please update to this release. Gentoo users have identified a block cloning bug (#15526) that can result in data corruption in zfs-2.2.0. To workaround this, zfs-2.2.1 disables block cloning by default (see #15529 and 479dca5). Users running the 2.1.x branch or older are unaffected, as block cloning is a 2.2.x-only feature.

https://github.com/openzfs/zfs/releases/tag/zfs-2.2.1

[chris.childerhose]

Yeah, I can see concerns with things like this happening. Maybe it is not ideal until the kinks are worked out, and the file system is more mature. Maybe a 2024 thing.

[biohazrd]

The OpenZFS team released 2.2.1 pretty quickly in response to the bug, thinking it was related to block cloning. It turns out to have been completely unrelated and is actually due to a race condition in hole seeking. Here's a link to the issue on Github which goes into a lot more detail.

Not to say caution isn't warranted with a new feature on a file system, I get where you're coming from. But this particular example, isn't related.

[tyler.jurgens]

That GitHub link is a fascinating read! Thanks for sharing.

As for Gostev's question? No, I wouldn't test this in production. I would be willing to throw it at a lab though.

[DonZoomik]

This could be an experimental/unsupported registry key that would for example relax requirements for filesystems. I presume that current version is not file system specific but there's just some filter to only allow ReFS/XFS. It'd be nice to test block clone on unsupported file systems like btrfs or zfs. Hell, it'd be nice it SMB block clone worked on non-ReFS (eg NAS supporting btrfs).
Not that it's a great idea but for lab purposes, not too bad either.

[HannesK]

Hello,
I merged the new thread with an older existing thread on the same topic. I did a quick test a month ago or so and the software does block cloning on ZFS already today with some hacks (which I don't want to post in public, because it's far off being supported)

So we will have a closer look at it after 12.1 is GA. Also thanks for your feedback about experimental support

Best regards,
Hannes

[biohazrd]

Awesome, thanks! Exciting to hear that we might see this in a future release.

[bct44]

can't wait to read some lab tests with OpenZFS on Linux Harden' repo

[HannesK]

Hello,
as the topic popped up again here, I will give a short update:

We are currently testing OpenZFS reflink for stability. Adjustments need to be made to avoid high CPU load. I strongly recommend avoiding any hacks to get ZFS working today.

Best regards,
Hannes

[chris.childerhose]

Great to hear you are testing OpenZFS now and look forward to the updates.

[ashleyw]

Are there any updates as to how close Veeam is to a release that alloweds for ZFS with COW support?
Reason why is we are running tests with MinIO at the moment and hitting a ceiling in terms of throughput on a spinning disk based single node and I'd like to run similar tests on a stripe across multiple raidz1 data sets but only if support is available in Veeam (even with an unsupported flag).
Thanks

[HannesK]

Hello,
yes, the plan is to enable ZFS reflink integration preview with the next minor update as an unsupported option. The reason for "unsupported" is, that reflink is still disabled per default and we expect it will take some time before reflink gets "backup storage maturity". Having that said, I'm very positive about working with the OpenZFS team.

Best regards,
Hannes

[JTT]

Is there ETA for the next minor update?

[Gostev]

Next month.

[JTT]

And ZFS reflink integration will make it to that release?

[Gostev]

Yes, a preview like Hannes explained.

[Gostev]

Hi all,

Once you upgrade to 12.1.2 (to be released shortly), you can enable this functionality for Linux-based backup repositories with ZFS volumes by creating /etc/veeam/EnableZFSBlockCloning file on a repository server.

This functionality is currently not supported for production use (not even under Experimental Support terms) and therefore should be used in test labs only. The purpose of this technology preview is to allow Veeam enthusiasts to test long term stability and performance of this new ZFS capability.

We will reuse this discussion for sharing feedback on the technology preview (I will link this topic in the Release Notes).

Thank you for your interest and time!

[mr-manuel]

Will this also work, if a QNAP (QuTS hero) ZFS storage pool is mounted via NFS on a Linux server?

[Gostev]

I don't think so (because of NFS layer between Veeam and ZFS).

[DonZoomik]

But no generic enablement without filesystem filter (btrfs, non-ReFS SMB)?

[Gostev]

That is correct.

Edit HannesK: please see longer explanation here

[ashleyw]

For anyone contemplating getting ready for testing any of this, block cloning on OpenZFS is not enabled by default and needs to be enabled.
Current release version of OpenZFS is 2.2.4 - I would strongly recommend that any potential testing is not done on earlier releases to minimise the risk of issues.

To enable block cloning, first check to see if this has been enabled or not;

Code: Select all

# cat /sys/module/zfs/parameters/zfs_bclone_enabled
0

Modify the kernel module settings to enable bclone and for this value to persist across reboots;

Code: Select all

# vi /etc/modprobe.d/zfs.conf
options zfs zfs_bclone_enabled=1

then reboot, then check the setting again;

Code: Select all

# cat /sys/module/zfs/parameters/zfs_bclone_enabled
1

Confirmation that block cloning is not enabled by default at this stage
https://www.reddit.com/r/linux/s/egWpsUNGfF

Discussion confirming 2.2.4.1 has been pushed to the el8 zfs-testing repositories inline with other Linux varients
https://github.com/openzfs/zfs/issues/16170

Discussion on whether block cloning should be made default of not in upcoming 2.2.5 release;
https://github.com/openzfs/zfs/issues/16189