Backup of NAS, file shares, file servers and object storage.
Post Reply
ksimon
Service Provider
Posts: 11
Liked: 8 times
Joined: Feb 18, 2020 4:24 pm
Full Name: Ken Simon
Contact:

Is Fragmentation the Issue? XFS repo

Post by ksimon »

This issue may be related to 07545793, 07520414, 07443538, or 07392626.

We are having continual speed issues with a repository on an XFS filesystem.
This repo is set up on a linux machine. The filesystem is an iSCSI mount from a Synology box running DSM 7.2.2-72806 Update 3. I know, Synology bad. Synology is not the issue here.
The repo is used as an off-site backup location for 7 clients.

The issue is write speed to this repo.

I tested speed from a device on the same network using CrytsalDiskMark.
Sequential write speed was as high as 17 MB/s.
Random writes were as slow as 0.36 MB/s.
These speeds are consistent with what I see in Veeam. Brutally slow.

We recently had to run an active full on a server. It took 277 hours to transfer 1 TB over a 1 Gb/s link.

I did a second test. I configured a new iSCSI target and created a new LUN on the same RAID as the repository. I mounted this NTFS on a Windows box on the same network.
Sequential writes were as high as 86 MB/s.
Random writes were much better at 4 MB/s.

In general, the speeds were 7-10 times as fast with NTFS on Windows as compared to XFS on Linux.
When this repo was first set up in January, the speeds were much faster.

I checked the fragmentation on the filesystem and I found fragmentation over 99% and >5000 extents per file.

xfs_db> frag
actual 6447827, ideal 1150, fragmentation factor 99.98%
Note, this number is largely meaningless.
Files on this filesystem average 5606.81 extents per file

Currently the filesystem is 32 TB with 76% used.

I attempted to defrag the files. After 4 continuous days, the defragmentation had completed about 10% of the files. I suspect that this will never complete.

So my questions are:
  1. What is the Veeam recommendation when dealing with fragmentation on XFS?
  2. What are people doing in the real world to deal with it?
karsten123
Service Provider
Posts: 575
Liked: 141 times
Joined: Apr 03, 2019 6:53 am
Full Name: Karsten Meja
Contact:

Re: Is Fragmentation the Issue? XFS repo

Post by karsten123 » 1 person likes this post

2. do not use Synology ;)
ksimon
Service Provider
Posts: 11
Liked: 8 times
Joined: Feb 18, 2020 4:24 pm
Full Name: Ken Simon
Contact:

Re: Is Fragmentation the Issue? XFS repo

Post by ksimon »

Interesting you should say that.
There are a number of Synology products that are considered by Veeam to be "Veeam Ready" and Veeam support has already assessed our setup and approved.

Let's talk about filesystems rather than hardware vendors.
karsten123
Service Provider
Posts: 575
Liked: 141 times
Joined: Apr 03, 2019 6:53 am
Full Name: Karsten Meja
Contact:

Re: Is Fragmentation the Issue? XFS repo

Post by karsten123 »

use a x86 server with a real hardware raid controller and compare it with that.
you always get what you put into it.

what are your Synology setup details? is the Linux server a hardware server? what are the specs on that?
what is your Linux distro and version? how did you create the xfs filesystem?

why not using a Windows repository if it is 10 times faster?
Gostev
Chief Product Officer
Posts: 32230
Liked: 7592 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Is Fragmentation the Issue? XFS repo

Post by Gostev » 1 person likes this post

ksimon wrote: Apr 21, 2025 5:37 pmLet's talk about filesystems rather than hardware vendors.
Only talking about about filesystems but not hardware that runs them is akin to only talking about fuel quality without considering the actual car engine design and performance. Octane number may matter a little, but nowhere near as much as the actual engine.

I agree with Karsten. If you want good performance and reliability, get something like DL380 with proper enterprise-grade RAID controller with cache. And never worry about this stuff again.

If you want to struggle with both, then get yourself some low-end NAS storage and you will be spending much time in the topics like this. And I'm not being arrogant here, I'm just summarizing 17 years worth of posts on this community.
ksimon
Service Provider
Posts: 11
Liked: 8 times
Joined: Feb 18, 2020 4:24 pm
Full Name: Ken Simon
Contact:

Re: Is Fragmentation the Issue? XFS repo

Post by ksimon »

So, Gostev. Does this mean to ignore "Veeam Ready"?

Sorry, but you are being arrogant. You are following the Veeam Support way of saying, "Throw out what you have. Try some other stuff. It may work better." Not everyone has $4000-$5000 to throw at a potential solution.

You are possibly correct, but unhelpful.

Are you saying that different hardware would have different fragmentation results?
Are you saying that the fragmentation is irrelevant? Because if you are there are many sources that would say you are incorrect.
Gostev
Chief Product Officer
Posts: 32230
Liked: 7592 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Is Fragmentation the Issue? XFS repo

Post by Gostev »

Keep in mind Veeam Ready is merely a compatibility testing, as opposed to performance and reliability certification. Veeam Ready means hardware is compatible with Veeam, but you will get what you pay for as it comes to performance and reliability. Which is perfectly fine as hardware vendors differentiate based on these two aspects specifically.

It's not a potential solution, what I say in my previous post has been proven times and again over nearly two decades now.

Enterprise-grade storage hardware gets you Enterprise-grade performance and reliability, simple as that. Proper hardware RAID controller implements many performance and reliability enhancing features that software RAID simply does not have, which are there to reduce impact of fragmentation (cache), catch and fix bit rot (patrol reads) etc.

Not having these features quite expectedly leads to poor performance, failed recoveries due to untreated bit rot and so on and so forth. None of this is important for storing family pictures and videos, which quite frankly is what low-end NAS are best suited for (I use one myself for this purpose). But enterprise backup is a completely different story.
ksimon
Service Provider
Posts: 11
Liked: 8 times
Joined: Feb 18, 2020 4:24 pm
Full Name: Ken Simon
Contact:

Re: Is Fragmentation the Issue? XFS repo

Post by ksimon »

So what you are saying is that the fragmentation is NOT a problem. The only problem is that my hardware is crap. Better hardware will just breeze over the fragmentation. 5-6000 extents per file is dandy fine.

Bitrot?? Seriously? With files that are less than 2 days old?? I would be able to hear the particles flying off the disks if they were degrading that quickly.

I feel like I would have received a completely different answer if I had not used the word "Synology".
karsten123
Service Provider
Posts: 575
Liked: 141 times
Joined: Apr 03, 2019 6:53 am
Full Name: Karsten Meja
Contact:

Re: Is Fragmentation the Issue? XFS repo

Post by karsten123 »

maybe. maybe not. although Veeam is hardware agnostic, the hardware is the base ingredient for performance and reliability.
you said its a repo for customers. are they not worth a proper solution?
Gostev
Chief Product Officer
Posts: 32230
Liked: 7592 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Is Fragmentation the Issue? XFS repo

Post by Gostev » 1 person likes this post

ksimon wrote: Apr 21, 2025 9:26 pmI feel like I would have received a completely different answer if I had not used the word "Synology".
I chose Synology as my home NAS myself and have nothing against this particular brand.

Wrt. to bit rot, this has nothing to deal with how old files are... google for "hard disk URE" as a starting point. Although of course there are many more ways to lose data that do not involve cosmic rays kicking out electrons from HDD plates, for example "RAID write hole" or even just a sticky bit in a faulty RAM module.

Since I've made all the technical points I wanted and this is seemingly turning into a flame discussion, I'm excluding myself from it going forward due to a very busy week ahead.
karsten123
Service Provider
Posts: 575
Liked: 141 times
Joined: Apr 03, 2019 6:53 am
Full Name: Karsten Meja
Contact:

Re: Is Fragmentation the Issue? XFS repo

Post by karsten123 »

Thank you Anton. Always a pleasure
ksimon
Service Provider
Posts: 11
Liked: 8 times
Joined: Feb 18, 2020 4:24 pm
Full Name: Ken Simon
Contact:

Re: Is Fragmentation the Issue? XFS repo

Post by ksimon »

So nobody wants to talk about fragmentation at all?
Post Reply

Who is online

Users browsing this forum: No registered users and 4 guests