V10 & XFS - all there is to know (?)

[DerOest]

Hello everyone,

i guess we need an XFS megathread. I hope it turns into a long love letter instead of a horror story

Official documentation is thin right now, mostly just a block here:
https://helpcenter.veeam.com/docs/backu ... =100#linux

Maybe we can get official statements or community-based feedback this way!

So i'll start it off with some questions regarding deduplication:

Deduplication
Is XFS + deduplication supported - especially in combination with FastClone?

There are no limitations mentioned @ https://helpcenter.veeam.com/docs/backu ... =100#linux
But that doc was for ReFS and just got updated with the XFS block.

I've just opened Case #04034676 to ask this question, will update this post when i hear from them.

Why am I interested in deduplication?
15+ month retention for ~40TB. Running Incremental forever doesn't feel safe with the VMware CBT bugs and everything else that could cause corruption.
I'd love to run active fulls monthly and dedup afterwards.

How to dedup?
This was my starting point: https://strugglers.net/~andy/blog/2017/ ... ation-demo

The tool I used is this - are there other tools or better ways?
https://github.com/markfasheh/duperemove

When given the -d option, duperemove will submit those extents for deduplication using the Linux kernel extent-same ioctl.

It's in the official repos (tested on Ubuntu 18.04.4 and 19.10). Here's how I did some quick testing (deduplication worked GREAT, but (time constrained) i had no chance for running incrementals or restores with those deduplicated backups afterwards ).

Code: Select all

~# apt install duperemove
~# duperemove -hdr /veeam-test-repo/Testbackup

Dedup-Performance?
It ran at a speed of >2GB/s (even touching 3GB/s) on a Cisco S3260, 42x 10TB HDD @ Raid60, Ubuntu 19.10

So yeah, should be easy to let it run once a month after the active fulls.
And because you can run duperemove with which files or folders to act on, it could be run specifically for jobs (folders) after you ran an active full.

[DerOest]

Other questions i'd be interested in:

RAID setup recommendations? Stripe Size etc?
There's a ton of information regarding ReFS with 64k filesystem blocks, for example here on Page#7: https://www.veeam.com/veeam-availabilit ... de_wpp.pdf
Take those settings or could other settings be better, because XFS has to be formatted with max. 4kb blocks?
Safe tuning options like special mount options or additional settings when formatting the XFS volume?

[tsightler]

The concern I would have with using a tool like "duperemove" is the sheer amount of metadata and fragmentation it will create on the filesystem. If you've followed ReFS you already know that fragmentation is an issue. XFS also had a history of significant issues with hangs during low memory situations on heavily fragmented volumes, well known to people that use XFS for streaming video. XFS attempts to avoid this with pre-allocation, and you can tune this during initial writes (to help the streaming issue), however, there's no way to avoid fragmentation with block clone, it will increase fragmentation, there's no two ways around it, and fragmentation will put more pressure on the kernel for memory allocation because of higher metadata, which will also increase with block clone.

Using a tool like dupemove will increase fragmentation, and I'm not sure of the overall benefit, at least not with default settings. Deperemove uses 128k blocks by default, which is admittedly smaller than our 512K average size, but it uses fixed blocks, Veeam's compressed block size is not fixed (we start with 1MB block, then compress that, then store it at whatever size it compresses to, literally it could be anything from 4K to 1MB). The thing is, when we "block clone" we know the size of the block we are cloning, so we clone exactly that size. With duperemove, it's just picking random 128K (or whatever you set it too) chunks. If we assume Veeam average block size vs duperemove, you could see 4x more IOP load during restore due to this fragmentation.

I'll admit, I don't know for sure, but it seems unlikely to make a huge difference overall. Whether it's supported or unsupported, I'd say this, it's not tested, and, in general, we don't support things we don't test and validate. Might it work, sure, but I'd suggest taking a cautious approach with your backups.

I personally think a more promising approach is VDO (in RHEL/CentOS 8, not sure about other distros since I don't think it's upstream yet). It lives at the block level and provides dedupe more like a traditional deduplication appliance. That being said, it has significant partition size limitations (I think 50TB right now) and needs gobs of memory because it needs to keep the dedupe tables in memory to have decent performance (like any dedupe appliance).

Personally, I would at least let XFS run normally, without any of these things, for some time, before starting to add even more variables. If you do have problems you may not know what component is causing trouble and we really don't know how XFS block clone will work long term right now.

[tsightler]

There's a ton of information regarding ReFS with 64k filesystem blocks, for example here on Page#7: https://www.veeam.com/veeam-availabilit ... de_wpp.pdf
Take those settings or could other settings be better, because XFS has to be formatted with max. 4kb blocks?
Safe tuning options like special mount options or additional settings when formatting the XFS volume?

These suggestions are based more on Veeam's average read/write block size and don't really change due to the XFS block size. Typically 256K or 512K seem to work best, on average for Veeam's workload (you can get higher raw throughput with some bigger sizes, but you pay the cost on random operations)

As far as tuning options, XFS does a really good job at formatting volumes optimally, so I can't think of anything too exciting there, probably the only one is agcount, which I think might be beneficial with the really large filesystems, but I don't have a good feel for what real difference it could make

As far as tuning options in general, I generically recommend "noatime" for all Linux filesystems. Veeam does a LOT of read access to files, so disabling access time updates can free up IOPS for other workloads and I can't think of any reason why updating access time really matters very much for the Veeam use case. I've been recommending this for almost a decade now so I think it's pretty safe.

One general system level recommendation is to increase the value of vm.min_free_kbytes. I'm already seeing some issues with memory allocation failures causing the OOM killer to kill veeamagent processing. These appear to be trigger by not being able to clear memory fast enough to perform a critical kernel level allocation. Keeping more memory free increases the aggressiveness of cache flushing mechanism and makes sure there's enough memory when there are sudden bursts of memory pressure. I'm not yet sure of the ideal value, but I'm thinking something between 1-3% of memory is probably enough. Here's an example of setting the value to 1GB:

echo 1048576 > /proc/sys/vm/min_free_kbytes

Obviously you can set this in your sysctl.conf or whatever vendor specific way. I'm currently using a value of 256MB (default was ~64MB) on a system with 16GB of RAM running 8 parallel tasks (yes, over-allocated vs best practice but designed to produce the issue) and that seems to have been enough to stop the issue so far.

[doum]

Does Veeam on Linux repository with XFS has the same features as on ReFS ? (like spaceless full backup for example) ?

[Gostev]

Yes.

[doum]

Thanks.

[nitramd]

@tsightler,

Would you recommend using noatime over relatime?

[tsightler]

Would you recommend using noatime over relatime?

I'd say yes. These days relatime should be the default for any filesystem anyway, but I just don't see any real advantage to it for a Veeam repo. Veeam doesn't use these attribute values for anything and, because relatime still updates access time on the next read following any write, and Veeam does lots of write/read cycles to the same file, I think it's good to not waste updating access time.

If you have some external reason why access time is important for you, then relatime (the default in all modern Linux distros) is fine, but I think for most people it's just extra IOPS for nothing useful. Do I think the difference is huge? Probably not, but why waste them if you're not using them for anything?

[nitramd]

Great.

Thank you.

[DerOest]

Hello,

(Oh no, unfortunately I just realize that it's not possible to update the first post! I'd remove most of the deduplication info there...)

Update on deduplication - not supported, will cause corruption!
EDIT HannesK: deduplication / duperemove was not tested by Veeam. we expect performance issues.
That was just explained to me via Case #04034676

[opg70]

Corruption with XFS? Can you be more specific why that may be the case? If the filesystem is sound, I don't see why it would do so - after all the files are still there and complete. Once duperemove finished its run the filesystem will know where the file blocks are and be able to read the whole file anyway. The only problem I see would be if duperemove runs during a fast clone operation, as neither would know the other's status and they might want to change the block pointers at the same time.

P.S. I was looking forward to an alternative to ReFS with better functionality and stability

[DerOest]

Sorry, that could have been clearer (and again it's not possible to edit previous posts [once there are answers?])

EDIT HannesK: deduplication / duperemove was not tested by Veeam. we expect performance issues.

XFS + FastClone is fine
or
XFS + Deduplication is fine

Just not both at the same time!

[zyrex]

I set up a new offsite repository, running Ubuntu 19.10 and XFS, hoping to use the spaceless tech and fast clone.
I already use it on my primary repository running ReFS, and it's splendid, plus another Windows offsite repository.

However, I noticed in the logs after it has finished processing the Backup Copy Job, that it doesn't say [Fast Clone], which it does for the other jobs to the other repo's...

I doublechecked that the filesystem was made with crc=1 and reflink=1, bsize=4096... Anything else I might have forgotten?

[DerOest]

Did you check the XFS FastClone checkbox in the repository settings?
See that screenshot: https://helpcenter.veeam.com/docs/backu ... ml?ver=100

[zyrex]

Well, wow. I had not checked it, and I can not remember that box from when I setup the repository.
Thank you very much, it is checked now

I have been checking out this doc; https://helpcenter.veeam.com/docs/backu ... ml?ver=100 which doesn't mentioned this.

[DonZoomik]

I enabled XFS support on one XFS based repo (admin claimed that it was formated with reflink support, Debian 10) and it started throwing errors like:
Failed to calculate block alignment size for LinuxLocal repository XXX (XXX) (System.Exception)
Invalid block alignment size 0 (System.Exception)
XFS is on top of LVM.

As service account doesn't have full sudo/su privileges, it's probably due to some limited/missing command, as there are only these commands currently in sudoers post170788.html#p170788

Will investigate it further.

[nitramd]

@DonZoomik

You can verify if reflink is enabled by using the command xfs_info /<mountpoint> or more succinctly xfs_info /<mountpoint> | grep reflink

[tsightler]

Repository service account should not require any sudo configuration. It's actually one of the reasons I like Linux repos and I normally recommend securing the account with an SSH key. I definitely do not give my Linux repo service accounts ANY elevated privileges. Of course the account must have access to the directory that you want to browse to and use for a repository.

Now, admittedly, if you have the account you are attempting to use configured to use sudo/elevation, it might be attempting to elevate even if it's not actually required, and failing do to so since your sudo environment is restricted, but I'd suggest creating an unprivileged account for your repos where possible.

[zyrex]

Just going back to my previous post here about Fast clone not working, and realizing I hadn't activated XFS-support in the repository-config... I have done that, but there's still no indication that fast clone is working.
Does it exist any log of any sort, telling me what specifically is stopping it from working?

Edit: Double-checked that Align data-blocks was enabled - it was now, but I don't think it was before. When i press Next, a windows shortly pops up saying it's checking if XFS Fast Clone is supported, so it seems it decides it is since I get no error message. Will check again tomorrow then

[DonZoomik]

@DonZoomik
You can verify if reflink is enabled by using the command xfs_info /<mountpoint> or more succinctly xfs_info /<mountpoint> | grep reflink

Code: Select all

# xfs_info /srv/veeam
meta-data=/dev/mapper/sys-veeam--storage isize=512    agcount=33, agsize=251658112 blks
         =                       sectsz=4096  attr=2, projid32bit=1
         =                       crc=1        finobt=1, sparse=1, rmapbt=0
         =                       reflink=1
data     =                       bsize=4096   blocks=8053063680, imaxpct=5
         =                       sunit=128    swidth=640 blks
naming   =version 2              bsize=4096   ascii-ci=0, ftype=1
log      =internal log           bsize=4096   blocks=521728, version=2
         =                       sectsz=4096  sunit=1 blks, lazy-count=1
realtime =none                   extsz=4096   blocks=0, rtextents=0

However GUI still claims

Code: Select all

File system behind the provided path is not XFS: /srv/veeam/backups
Proceed anyway?

[soncscy]

@DonZoomik,

Debian won't from, from my long experience trying to get it to go.

In the User Guide:

https://helpcenter.veeam.com/docs/backu ... ml?ver=100

"Linux distribution has the reflink kernel module.
Supported distributions: Ubuntu 18.0.4 or later. For other distributions, Fast Clone support is experimental."

I honestly never got it working after hours of fiddling with it, and I suppose that just some action needs to be done behind the scenes to make it "supported". I know that Ubuntu is Debian-based, but legit -- as soon as I spun up an Ubuntu VM, it was almost instantaneous.

It may not be ideal, but just use an Ubuntu box for this.

[DonZoomik]

Not my choice unfortunately.

[tsightler]

@DonZoomik The error you are seeing is likely due to the fact that VBR can't find xfs_info. On Debian xfs_info is in /usr/sbin but regular users typically do not have this in their path. If VBR attempts to run xfs_info and fails to find the command it will generate the "Filesystem behind the provided path is not XFS" error and will fail to calculate offsets. If you login with the account you are using and running xfs_info gives an "file not found" error then that is the issue.

For me, Debian 10 worked fine once I corrected this issue by adding the following to the top of the veeam user .bashrc file:

Code: Select all

PATH="$PATH:/sbin:/usr/sbin"

Hope that helps.

[DonZoomik]

Thanks, that worked!

[HannesK]

about the duperemove deduplication topic before: I talked to QA and it was not tested. But we don't expect data corruption. We only expect low performance.

I edited the posts above to avoid confusion as people might not read until here

[opg70]

When they find the time, could they test it HannesK?

[DonZoomik]

I found it kind of strange as well as dedupe and block clone use exactly the same functionality and they shouldn't in theory care about each other, performance loss due to heavy fragmentation is expected. And XFS really doesn't like high fragmentation. Kernel improved memory need for fragmented systems but even with newer kernels, it can occasionally cause problems (not my own experience, I just work with a team that uses a lot of XFS).

[Gostev]

You are right to doubt here, as actually this was an incorrect statement by your support engineer. He admitted he did not have any grounds to say what he said, and promised not to make personal assumptions again

I see Hannes has already edited it out the corresponding quote from your post to prevent further confusion by future readers. But just to re-iterate what he already said above: in theory, Veeam fast cloning and duperemove should be fully compatible, because they are using exact same I/O controls. However, it's currently unknown how well these technologies work along together in practice. Normally, this would require comprehensive compatibility, scalability testing, and long-term stress testing.

Performance impact from added fragmentation is definitely something to watch out for... however, it may not be a big issue for secondary backup repositories, for example.

[zyrex]

Still not getting Fast Clone to work on my repository. Any tips on what to look for? Everything seems to be in order, but still nothing. Would've loved a log file telling me "Not using fast clone because x or y"