v10 XFS Fast Clone "partition size" in pre-release documentation

[evilaedmin]

In the docs for fast clone is this:

To use Fast Clone, Veeam Backup & Replication requires that Linux backup repositories meet the following conditions:

• Linux distribution has the reflink kernel module.
• Supported distributions: Ubuntu 19.0.4 or later. For other distributions, Fast Clone support is experimental.
• File system is XFS.
• Cyclic redundancy check (CRC) is enabled.
• The minimum supported data block size is 1 KB. The maximum supported block size is 4KB.
• Disk partitions is less than 200 TB.

What does the "disk partitions is less than 200TB" mean? Many storage set ups do not deal with disk partitions at all.. (LVM on whole devices for example). Is this meant to describe a file system limitation?

[mkretzer]

Thats strange.... I tested the Beta with a 360 TB Repo and REFS. I wonder if that is because of performance limitations with bigger volumes?

[nitramd]

Red Hat's maximum certified XFS filesystem size is 500TiB - starting with RHEL 7.

A quick search for Ubuntu XFS max certified partition size did not yield any results. I wonder if the 200TB figure is Ubuntu's limit then?

[Gostev]

Guys, I will check on that - but in general, please don't pay attention to documentation accuracy until GA, and especially please don't quote it anywhere outside these forums. Please appreciate the fact that the resource linked above has not been officially launched yet, and is currently being reviewed by our QC folks, who are finally off the RTM build testing. Thanks!

[lasseoe]

Sure hope "19.04" is a typo as it's already end of life, shouldn't it be 18.04 ? The non-LTS releases are only supported for 9 months, hardly enough to warrant using it for a production server. https://ubuntu.com/about/release-cycle

[Gostev]

XFS block is fairly new capability to Ubuntu, so it is safe to say by just looking at 18.04 release date that it is unlikely to have it.

[tsightler]

XFS block clone (technically reflink in Linux terms) is in Ubuntu 18.04, but it's a fairly early iteration and you have to explicitly enable it when formatting because the older xfsprogs doesn't enable it by default. Also, stock kernels are before what would be "ideal" versions for using this feature, so it's difficult to know how stable it would be. In fairly small scale testing in my lab it seemed fine, but there were significant enhancements to XFS block clone performance over the development lifecycle. The XFS developers didn't declare XFS reflink support "production ready" until extremely recently:

https://blogs.oracle.com/linux/xfs-data ... ng-reflink

It's now enabled by default in xfsprogs 5.1 and higher (and seems to be enabled by default in RHEL8/CentOS 8 which still has xfsprogs 5.0). My suggestion is that kernel 4.18 or higher is probably required to get the most reasonable performance. The good thing is that Ubuntu 20.04 is only a couple of months away so we'll soon have a new LTS release with production ready (from kernel dev perspective) XFS block clone built right in.

[mkretzer]

tsightler,

Ok - what linux would you recommend right now for V10+xfs bc? Would Debian 10 + 5.2 Kernel Backport be a good candidate?

[tsightler]

The reality is, it's difficult to know as, just like with ReFS, I suspect there will be some problems discovered in the real-world at scales that I don't reach in my lab, although my test lab still breaks Windows 2019 to this day, so there's that, and it would hard hang 2016 back in the day as well. So far, I haven't had any such issues XFS block clone, but 100's of TBs could easily change this. I do see some memory allocation failures which cause short (a few seconds) stalls under very high load, these seem to have largely been mitigated once I increased min_free_kbytes, which seemed to be the culprit based on logs.

To this point I've tested CentOS 8 with default 4.18 kernel and Debian 10 with default 4.19 kernel, and both have performed just fine. I'm now using CentOS 8 with 4.18 as my primary repos with about 8TB of backups, so not a huge amount, but performance for fast clone is faster than before I migrated those backups from Windows 2016 repos they were on before.

In theory, assuming there's been no other regressions, kernel 5.4 would be the best to use, it's a long term release and has all of the XFS optimizations and fixes (5.2 is still missing some performance and memory optimizations), but I haven't yet tested Debian 10 with the 5.4 backport kernel, and theory is commonly beaten to a pulp by practice, but it's my current thought as to what would be a great combo, which is why I'm looking forward to Ubuntu 20.04, which should come with 5.4 or newer out of box.

[DerOest]

Guys, I will check on that - but in general, please don't pay attention to documentation accuracy until GA, and especially please don't quote it anywhere outside these forums. Please appreciate the fact that the resource linked above has not been officially launched yet, and is currently being reviewed by our QC folks, who are finally off the RTM build testing. Thanks!

Just a heads-up for everybody stumbling over this thread:
Exactly as Sir Gostev said, the documentation has actually been updated, and as the OP specifically and only asked about this item, i guess the thread could as well be deleted to not confuse people stumbling over this in the future.

Here's the official/GA documentation:
https://helpcenter.veeam.com/docs/backu ... ml?ver=100

• No mention of size limitations anymore

• Supported distributions: Ubuntu 18.0.4 or later. For other distributions, Fast Clone support is experimental.

[ferrus]

Is it likely that Ubuntu 20.04 will be supported with v10 following its release, in a few weeks - or will it have to wait for a later Veeam version?

Just asking because this feature has arrived at the right time for the infrastructure I'm currently building - but 18.04.4 is refusing to install in our diskless SAN boot environment.
RHEL/CentOS 6, 7 & 8 do, and the current build of Ubuntu 20.04 does. Just seems to be an issue with 18.04 (the only one listed as fully supported).

Our infrastructure won't be live for a number of weeks, and it's only the DR site - so I'm just trying to work out the best install/support path.

[DerOest]

Hi Ferrus,
the Documentation specifically says "Ubuntu 18.0.4 or later".
But i totally understand your concern, i guess there will be official confirmation/validation when 20.04 gets released (I suppose that Veeam - just like you - already tested the current 20.04 branch).

[ferrus]

I'm not overly concerned about support - (because it's only to store offline Backup Copies/Replicated VMs at a DR site), but after three days of trying - and failing to get 18.04 multipath/grub working correctly, I think I'd feel more confident about experimental support of a new LTS release for a few months - than an entire OS installation that I've had to hack at for several days to get working

[evilaedmin]

Just a heads-up for everybody stumbling over this thread:
Exactly as Sir Gostev said, the documentation has actually been updated, and as the OP specifically and only asked about this item, i guess the thread could as well be deleted to not confuse people stumbling over this in the future.

Here's the official/GA documentation:
https://helpcenter.veeam.com/docs/backu ... ml?ver=100

• No mention of size limitations anymore

• Supported distributions: Ubuntu 18.0.4 or later. For other distributions, Fast Clone support is experimental.

OP checking in. I would delete if I could.

However, a 'canonical' thread on XFS fast clone similar to the ReFS discussions which repeat form time to time, I plan to follow very very closely.

[Gostev]

I will adjust the title to avoid any possible confusion, and simply lock this topic to let it sink.

As the summary, there are no known XFS partition size limits at the moment, while best practices will be established based on results from the field.

I agree re: the dedicated XFS thread, and I'm keep looking for a good post to split into the new discussion, unless someone starts it soon anyway