[V10] Veeam XFS Server high disk IO on system disk

[mkretzer]

Hello,

today we tried to find the bottleneck of a XFS merge on our ~500 TB XFS test system. The interesting thing according to iostat was that not the XFS had high load but the system disk of the linux system.

Since this is still rotating disk it seems to be our bottleneck as it wrote at about 50-100 MB/s at the time of merges (~300 IOPS).

We compared this with our ReFS systems and saw only minimal disk io (< 5 MB/s) at the time of bigger merges.

Is this normal? Or only a V10 problem?

We would open a case but the ticket system seems to be offline.....

Markus

[mkretzer]

Here a iostat output example:

Code: Select all

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           3,46    0,00    2,21    1,53    0,00   92,80

Device            r/s     rkB/s   rrqm/s  %rrqm r_await rareq-sz     w/s     wkB/s   wrqm/s  %wrqm w_await wareq-sz     d/s     dkB/s   drqm/s  %drqm d_await dareq-sz  aqu-sz  %util
dm-0             0,00      0,00     0,00   0,00    0,00     0,00  162,20 113774,40     0,00   0,00   75,52   701,45    0,00      0,00     0,00   0,00    0,00     0,00   12,25  84,24
dm-1             2,00     49,60    10,40  83,87    3,50    24,80  342,60   1370,40     0,00   0,00    0,95     4,00    0,00      0,00     0,00   0,00    0,00     0,00    0,21   7,92
dm-10            1,80     44,00    10,40  85,25    6,89    24,44  337,40   1349,60     0,00   0,00    1,94     4,00    0,00      0,00     0,00   0,00    0,00     0,00    0,19   9,12
dm-11            1,60     43,20    10,40  86,67    7,75    27,00  343,20   1372,80     0,00   0,00    2,02     4,00    0,00      0,00     0,00   0,00    0,00     0,00    0,22   9,36
dm-12            2,20     45,60    10,40  82,54    6,27    20,73  337,60   1350,40     0,00   0,00    1,10     4,00    0,00      0,00     0,00   0,00    0,00     0,00    0,08   9,60
dm-13            2,00     38,40    10,40  83,87    4,10    19,20  344,20   1376,80     0,00   0,00    1,08     4,00    0,00      0,00     0,00   0,00    0,00     0,00    0,06   8,40
dm-14            1,80     36,80    10,20  85,00    3,56    20,44  339,80   1359,20     0,00   0,00    1,93     4,00    0,00      0,00     0,00   0,00    0,00     0,00    0,18   9,12
dm-15            0,40     26,40     6,20  93,94   14,00    66,00   10,80     43,20     0,00   0,00    0,19     4,00    0,00      0,00     0,00   0,00    0,00     0,00    0,01   2,16
dm-16            0,20     24,80     6,00  96,77   11,00   124,00   10,80     43,20     0,00   0,00    0,20     4,00    0,00      0,00     0,00   0,00    0,00     0,00    0,00   1,52
dm-17            0,20     25,60     6,20  96,88   18,00   128,00   11,00     44,00     0,00   0,00    0,31     4,00    0,00      0,00     0,00   0,00    0,00     0,00    0,00   1,84
dm-18            0,20     25,60     6,20  96,88   18,00   128,00   11,00     44,00     0,00   0,00    0,25     4,00    0,00      0,00     0,00   0,00    0,00     0,00    0,00   1,68
dm-19          126,20    504,80     0,00   0,00    6,42     4,00 2112,60   8450,40     0,00   0,00    1,40     4,00    0,00      0,00     0,00   0,00    0,00     0,00    3,77  11,92
dm-2             0,20     25,60     6,20  96,88   12,00   128,00   11,00     44,00     0,00   0,00    0,18     4,00    0,00      0,00     0,00   0,00    0,00     0,00    0,00   1,84
dm-3             0,40     26,40     6,20  93,94   11,00    66,00   11,00     44,00     0,00   0,00    0,16     4,00    0,00      0,00     0,00   0,00    0,00     0,00    0,00   2,16
dm-4             0,00      0,00     1,20 100,00    0,00     0,00    0,00      0,00     0,00   0,00    0,00     0,00    0,00      0,00     0,00   0,00    0,00     0,00    0,00   0,16
dm-5             0,40      6,40     1,20  75,00   11,00    16,00    0,20      0,80     0,00   0,00    0,00     4,00    0,00      0,00     0,00   0,00    0,00     0,00    0,00   0,40
dm-6             0,20      0,80     1,20  85,71   20,00     4,00    0,60      2,40     0,00   0,00    0,33     4,00    0,00      0,00     0,00   0,00    0,00     0,00    0,00   0,64
dm-7             0,60      7,20     1,20  66,67    7,00    12,00    0,60      2,40     0,00   0,00    0,00     4,00    0,00      0,00     0,00   0,00    0,00     0,00    0,00   0,56
dm-8             0,60      7,20     1,20  66,67    8,00    12,00    0,40      1,60     0,00   0,00    0,50     4,00    0,00      0,00     0,00   0,00    0,00     0,00    0,00   0,56
dm-9             0,60      7,20     1,20  66,67   10,00    12,00    0,40      1,60     0,00   0,00    0,00     4,00    0,00      0,00     0,00   0,00    0,00     0,00    0,00   0,64

[tsightler]

I would not consider it "normal", although it's difficult to tell for sure where the I/O is coming from. Did you attempt to track the I/O to a process/file? The fact that it appears to be 100% writes makes me think it might be logging (perhaps a little too verbose), but just a guess.

[mkretzer]

Alot of log files are open, yes. But why is there so much more logging activity on linux than on windows?

[pirx]

Are you running low on memory is the system swapping to a swapfile? But this wouldn't only be writes....

[mkretzer]

Nope, 384 GB Ram. < 10 Gb used.

[tsightler]

Alot of log files are open, yes. But why is there so much more logging activity on linux than on windows?

Well, logging was just a guess, I don't know why it would be logging more, but it is completely different code after all. Could just be some portion of that code that is too verbose by default as we've seem that be an issue many times over the years since we have a tendency to log a lot. Might even be tunable. But the first thing you have to do is figure out where it's coming from, then it's easy to determine if it's too much or not.

[pirx]

In past I found iotop useful to find such processes.

[mkretzer]

I got a case opened now (04704613).

My suspect currently are the agents writing to /tmp all the time.

@pirx problem with tools like iotop is that we do not see the folder where the processes write to. And the Veeam Agents write alot anyway so its difficoult to see if it currently writes to the XFS or to something else.

[Did]

With that much RAM available I wouldn't hesitate to mount /tmp with tmpfs. With tmpfs the directory is stored in memory (moved to swap in case of memory congestion) and cleared on every reboot.
That should really speed things up and lower the pressure on your rotating disks, if a lot of the I/O is being done in /tmp.

[pirx]

@mkretzer any update on this?

[mkretzer]

To be honest we did not find the reason yet and we have now purchased SSD as a quick fix for the OS.
Strangely its really difficult to find out the reason for the disks IO in linux compared to ressource monitor in windows...

[tsightler]

I wouldn't think it would be that difficult to narrow down. The iotop or dtrace utility should make quick work of at least knowing what processes is doing the writes and then strace only write operations on that PID should quickly tell you what files, or heck, even an lsof on the PID would probably give enough clues. Have those standard methods been unable to find the process performing the I/O?

[mkretzer]

We tried with Veeam Support - lsof showed so much that is was very difficult to find the correct process.

As far as i know there is no such utility which shows IOPS per file like it is possible in windows... Per process is no problem....

[tsightler]

lsof showed so much that is was very difficult to find the correct process.

Sure, that's why you use iostat/dstat to find the top I/O process and then run lsof only on that process, or even just strace.

As far as i know there is no such utility which shows IOPS per file like it is possible in windows... Per process is no problem...

Probably no single tool, but combining multiple tools should get you there pretty quickly. Normal process for me is to do something like iostat/dstat to find the process, then strace (filtered to just write system calls) on the process ID, or, if I don't want to interfere with the process, lsof of the specific process ID which will usually quickly give me a list of path least common paths to look in. Then I either look manually, or use something like inotifywatch in recursive mode to collect data on all file changes in those paths.

I haven't been able to reproduce this behavior in my lab, but I'm guessing it may simply be that I don't have enough data as I certainly don't have 100's of TBs, but I thought it might reproduce on at least a smaller scale. I'm certainly interested in finding out what's causing it.

[mkretzer]

Ok - right now we are filling our new 620 TB XFS production repo. It will take some time but as soon as the first merges are running there i will try again.

[tsightler]

Thanks @mkretzer! I'm going to research a little bit and see if it's possible to put together a script, or at least a set of commands, that might help in narrowing it down. What distro are you using?

[mkretzer]

Ubuntu 20.04 as recommend