Comprehensive data protection for all workloads
Post Reply
mkretzer
Veeam Legend
Posts: 1140
Liked: 387 times
Joined: Dec 17, 2015 7:17 am
Contact:

[V10] Veeam XFS Server high disk IO on system disk

Post by mkretzer »

Hello,

today we tried to find the bottleneck of a XFS merge on our ~500 TB XFS test system. The interesting thing according to iostat was that not the XFS had high load but the system disk of the linux system.

Since this is still rotating disk it seems to be our bottleneck as it wrote at about 50-100 MB/s at the time of merges (~300 IOPS).

We compared this with our ReFS systems and saw only minimal disk io (< 5 MB/s) at the time of bigger merges.

Is this normal? Or only a V10 problem?

We would open a case but the ticket system seems to be offline.....

Markus
mkretzer
Veeam Legend
Posts: 1140
Liked: 387 times
Joined: Dec 17, 2015 7:17 am
Contact:

Re: [V10] Veeam XFS Server high disk IO on system disk

Post by mkretzer »

Here a iostat output example:

Code: Select all

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           3,46    0,00    2,21    1,53    0,00   92,80

Device            r/s     rkB/s   rrqm/s  %rrqm r_await rareq-sz     w/s     wkB/s   wrqm/s  %wrqm w_await wareq-sz     d/s     dkB/s   drqm/s  %drqm d_await dareq-sz  aqu-sz  %util
dm-0             0,00      0,00     0,00   0,00    0,00     0,00  162,20 113774,40     0,00   0,00   75,52   701,45    0,00      0,00     0,00   0,00    0,00     0,00   12,25  84,24
dm-1             2,00     49,60    10,40  83,87    3,50    24,80  342,60   1370,40     0,00   0,00    0,95     4,00    0,00      0,00     0,00   0,00    0,00     0,00    0,21   7,92
dm-10            1,80     44,00    10,40  85,25    6,89    24,44  337,40   1349,60     0,00   0,00    1,94     4,00    0,00      0,00     0,00   0,00    0,00     0,00    0,19   9,12
dm-11            1,60     43,20    10,40  86,67    7,75    27,00  343,20   1372,80     0,00   0,00    2,02     4,00    0,00      0,00     0,00   0,00    0,00     0,00    0,22   9,36
dm-12            2,20     45,60    10,40  82,54    6,27    20,73  337,60   1350,40     0,00   0,00    1,10     4,00    0,00      0,00     0,00   0,00    0,00     0,00    0,08   9,60
dm-13            2,00     38,40    10,40  83,87    4,10    19,20  344,20   1376,80     0,00   0,00    1,08     4,00    0,00      0,00     0,00   0,00    0,00     0,00    0,06   8,40
dm-14            1,80     36,80    10,20  85,00    3,56    20,44  339,80   1359,20     0,00   0,00    1,93     4,00    0,00      0,00     0,00   0,00    0,00     0,00    0,18   9,12
dm-15            0,40     26,40     6,20  93,94   14,00    66,00   10,80     43,20     0,00   0,00    0,19     4,00    0,00      0,00     0,00   0,00    0,00     0,00    0,01   2,16
dm-16            0,20     24,80     6,00  96,77   11,00   124,00   10,80     43,20     0,00   0,00    0,20     4,00    0,00      0,00     0,00   0,00    0,00     0,00    0,00   1,52
dm-17            0,20     25,60     6,20  96,88   18,00   128,00   11,00     44,00     0,00   0,00    0,31     4,00    0,00      0,00     0,00   0,00    0,00     0,00    0,00   1,84
dm-18            0,20     25,60     6,20  96,88   18,00   128,00   11,00     44,00     0,00   0,00    0,25     4,00    0,00      0,00     0,00   0,00    0,00     0,00    0,00   1,68
dm-19          126,20    504,80     0,00   0,00    6,42     4,00 2112,60   8450,40     0,00   0,00    1,40     4,00    0,00      0,00     0,00   0,00    0,00     0,00    3,77  11,92
dm-2             0,20     25,60     6,20  96,88   12,00   128,00   11,00     44,00     0,00   0,00    0,18     4,00    0,00      0,00     0,00   0,00    0,00     0,00    0,00   1,84
dm-3             0,40     26,40     6,20  93,94   11,00    66,00   11,00     44,00     0,00   0,00    0,16     4,00    0,00      0,00     0,00   0,00    0,00     0,00    0,00   2,16
dm-4             0,00      0,00     1,20 100,00    0,00     0,00    0,00      0,00     0,00   0,00    0,00     0,00    0,00      0,00     0,00   0,00    0,00     0,00    0,00   0,16
dm-5             0,40      6,40     1,20  75,00   11,00    16,00    0,20      0,80     0,00   0,00    0,00     4,00    0,00      0,00     0,00   0,00    0,00     0,00    0,00   0,40
dm-6             0,20      0,80     1,20  85,71   20,00     4,00    0,60      2,40     0,00   0,00    0,33     4,00    0,00      0,00     0,00   0,00    0,00     0,00    0,00   0,64
dm-7             0,60      7,20     1,20  66,67    7,00    12,00    0,60      2,40     0,00   0,00    0,00     4,00    0,00      0,00     0,00   0,00    0,00     0,00    0,00   0,56
dm-8             0,60      7,20     1,20  66,67    8,00    12,00    0,40      1,60     0,00   0,00    0,50     4,00    0,00      0,00     0,00   0,00    0,00     0,00    0,00   0,56
dm-9             0,60      7,20     1,20  66,67   10,00    12,00    0,40      1,60     0,00   0,00    0,00     4,00    0,00      0,00     0,00   0,00    0,00     0,00    0,00   0,64
tsightler
VP, Product Management
Posts: 6009
Liked: 2843 times
Joined: Jun 05, 2009 12:57 pm
Full Name: Tom Sightler
Contact:

Re: [V10] Veeam XFS Server high disk IO on system disk

Post by tsightler » 1 person likes this post

I would not consider it "normal", although it's difficult to tell for sure where the I/O is coming from. Did you attempt to track the I/O to a process/file? The fact that it appears to be 100% writes makes me think it might be logging (perhaps a little too verbose), but just a guess.
mkretzer
Veeam Legend
Posts: 1140
Liked: 387 times
Joined: Dec 17, 2015 7:17 am
Contact:

Re: [V10] Veeam XFS Server high disk IO on system disk

Post by mkretzer »

Alot of log files are open, yes. But why is there so much more logging activity on linux than on windows?
pirx
Veeam Legend
Posts: 568
Liked: 72 times
Joined: Dec 20, 2015 6:24 pm
Contact:

Re: [V10] Veeam XFS Server high disk IO on system disk

Post by pirx »

Are you running low on memory is the system swapping to a swapfile? But this wouldn't only be writes....
mkretzer
Veeam Legend
Posts: 1140
Liked: 387 times
Joined: Dec 17, 2015 7:17 am
Contact:

Re: [V10] Veeam XFS Server high disk IO on system disk

Post by mkretzer »

Nope, 384 GB Ram. < 10 Gb used.
tsightler
VP, Product Management
Posts: 6009
Liked: 2843 times
Joined: Jun 05, 2009 12:57 pm
Full Name: Tom Sightler
Contact:

Re: [V10] Veeam XFS Server high disk IO on system disk

Post by tsightler »

mkretzer wrote: Mar 16, 2021 9:14 pm Alot of log files are open, yes. But why is there so much more logging activity on linux than on windows?
Well, logging was just a guess, I don't know why it would be logging more, but it is completely different code after all. Could just be some portion of that code that is too verbose by default as we've seem that be an issue many times over the years since we have a tendency to log a lot. Might even be tunable. But the first thing you have to do is figure out where it's coming from, then it's easy to determine if it's too much or not.
pirx
Veeam Legend
Posts: 568
Liked: 72 times
Joined: Dec 20, 2015 6:24 pm
Contact:

Re: [V10] Veeam XFS Server high disk IO on system disk

Post by pirx »

In past I found iotop useful to find such processes.
mkretzer
Veeam Legend
Posts: 1140
Liked: 387 times
Joined: Dec 17, 2015 7:17 am
Contact:

Re: [V10] Veeam XFS Server high disk IO on system disk

Post by mkretzer »

I got a case opened now (04704613).

My suspect currently are the agents writing to /tmp all the time.

@pirx problem with tools like iotop is that we do not see the folder where the processes write to. And the Veeam Agents write alot anyway so its difficoult to see if it currently writes to the XFS or to something else.
Did
Service Provider
Posts: 3
Liked: 1 time
Joined: Apr 24, 2018 2:24 pm
Contact:

Re: [V10] Veeam XFS Server high disk IO on system disk

Post by Did » 1 person likes this post

With that much RAM available I wouldn't hesitate to mount /tmp with tmpfs. With tmpfs the directory is stored in memory (moved to swap in case of memory congestion) and cleared on every reboot.
That should really speed things up and lower the pressure on your rotating disks, if a lot of the I/O is being done in /tmp.
pirx
Veeam Legend
Posts: 568
Liked: 72 times
Joined: Dec 20, 2015 6:24 pm
Contact:

Re: [V10] Veeam XFS Server high disk IO on system disk

Post by pirx »

@mkretzer any update on this?
mkretzer
Veeam Legend
Posts: 1140
Liked: 387 times
Joined: Dec 17, 2015 7:17 am
Contact:

Re: [V10] Veeam XFS Server high disk IO on system disk

Post by mkretzer »

To be honest we did not find the reason yet and we have now purchased SSD as a quick fix for the OS.
Strangely its really difficult to find out the reason for the disks IO in linux compared to ressource monitor in windows...
tsightler
VP, Product Management
Posts: 6009
Liked: 2843 times
Joined: Jun 05, 2009 12:57 pm
Full Name: Tom Sightler
Contact:

Re: [V10] Veeam XFS Server high disk IO on system disk

Post by tsightler »

I wouldn't think it would be that difficult to narrow down. The iotop or dtrace utility should make quick work of at least knowing what processes is doing the writes and then strace only write operations on that PID should quickly tell you what files, or heck, even an lsof on the PID would probably give enough clues. Have those standard methods been unable to find the process performing the I/O?
mkretzer
Veeam Legend
Posts: 1140
Liked: 387 times
Joined: Dec 17, 2015 7:17 am
Contact:

Re: [V10] Veeam XFS Server high disk IO on system disk

Post by mkretzer »

We tried with Veeam Support - lsof showed so much that is was very difficult to find the correct process.

As far as i know there is no such utility which shows IOPS per file like it is possible in windows... Per process is no problem....
tsightler
VP, Product Management
Posts: 6009
Liked: 2843 times
Joined: Jun 05, 2009 12:57 pm
Full Name: Tom Sightler
Contact:

Re: [V10] Veeam XFS Server high disk IO on system disk

Post by tsightler »

lsof showed so much that is was very difficult to find the correct process.
Sure, that's why you use iostat/dstat to find the top I/O process and then run lsof only on that process, or even just strace.
As far as i know there is no such utility which shows IOPS per file like it is possible in windows... Per process is no problem...
Probably no single tool, but combining multiple tools should get you there pretty quickly. Normal process for me is to do something like iostat/dstat to find the process, then strace (filtered to just write system calls) on the process ID, or, if I don't want to interfere with the process, lsof of the specific process ID which will usually quickly give me a list of path least common paths to look in. Then I either look manually, or use something like inotifywatch in recursive mode to collect data on all file changes in those paths.

I haven't been able to reproduce this behavior in my lab, but I'm guessing it may simply be that I don't have enough data as I certainly don't have 100's of TBs, but I thought it might reproduce on at least a smaller scale. I'm certainly interested in finding out what's causing it.
mkretzer
Veeam Legend
Posts: 1140
Liked: 387 times
Joined: Dec 17, 2015 7:17 am
Contact:

Re: [V10] Veeam XFS Server high disk IO on system disk

Post by mkretzer »

Ok - right now we are filling our new 620 TB XFS production repo. It will take some time but as soon as the first merges are running there i will try again.
tsightler
VP, Product Management
Posts: 6009
Liked: 2843 times
Joined: Jun 05, 2009 12:57 pm
Full Name: Tom Sightler
Contact:

Re: [V10] Veeam XFS Server high disk IO on system disk

Post by tsightler » 1 person likes this post

Thanks @mkretzer! I'm going to research a little bit and see if it's possible to put together a script, or at least a set of commands, that might help in narrowing it down. What distro are you using?
mkretzer
Veeam Legend
Posts: 1140
Liked: 387 times
Joined: Dec 17, 2015 7:17 am
Contact:

Re: [V10] Veeam XFS Server high disk IO on system disk

Post by mkretzer »

Ubuntu 20.04 as recommend
Post Reply

Who is online

Users browsing this forum: Bing [Bot], Ivan239, sergiosergio and 262 guests