How do you use diskspd on Linux Repo for read testing?

[BackItUp2020]

Oh how I miss Windows sometimes.

I am trying to test read performance on our repo and am struggling to get a test working. The Veeam document isn't 100% complete, at least as I see it, so I'm asking for help.

diskspd.exe -Sh -d600 #X

This does not work unless you specify a path/file. If I specify a path, I get:

ret from aio not equal to block size, it's -22
Error during warmup phase!
Job 0 failed, exiting


Not sure what that means. I've tried creating a file using the -c parameter which seems to work (although I have no idea if the content of the file is appropriate for the testing), however I still get a block size error.

Has anyone used this successfully?

I've created the repo according to widely-used documentation.

Cisco S3260, RAID6, 512K stripe size.
XFS, 4096K block size.

Struggling here to do a test read capabilities of this repo in order to rule it out as a true bottleneck in our environrment.

[karsten123]

why not fio or dd?

[BackItUp2020]

I'm not much of a Linux guy, especially as of late and I'm not sure I know the proper test to run or how to interpret the results, let alone support.

I could try though.

[BackItUp2020]

I'd love a recommended test for FIO.

I did this, but I don't think this represents the correct I/O pattern.

veeam@vlt-vbrrep01:~$ sudo fio --name=fiotest --filename=/dev/sdb1 --size=16Gb --rw=read --bs=1M --direct=1 --numjobs=8 --ioengine=libaio --iodepth=8 --group_reporting --runtime=60 --startdelay=60
fiotest: (g=0): rw=read, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=libaio, iodepth=8
...
fio-3.16
Starting 8 processes
Jobs: 4 (f=3): [_(1),R(3),_(1),f(1),_(2)][100.0%][r=3751MiB/s][r=3751 IOPS][eta 00m:00s]
fiotest: (groupid=0, jobs=8): err= 0: pid=2483033: Tue Sep 5 17:30:20 2023
read: IOPS=3957, BW=3957MiB/s (4150MB/s)(128GiB/33120msec)
slat (usec): min=82, max=1174, avg=169.85, stdev=39.03
clat (usec): min=835, max=751915, avg=15897.17, stdev=28266.59
lat (usec): min=1024, max=752060, avg=16067.79, stdev=28267.28
clat percentiles (usec):
| 1.00th=[ 1090], 5.00th=[ 1156], 10.00th=[ 1237], 20.00th=[ 1467],
| 30.00th=[ 1909], 40.00th=[ 2704], 50.00th=[ 5145], 60.00th=[ 8717],
| 70.00th=[ 12911], 80.00th=[ 22152], 90.00th=[ 43254], 95.00th=[ 68682],
| 99.00th=[137364], 99.50th=[168821], 99.90th=[258999], 99.95th=[308282],
| 99.99th=[396362]
bw ( MiB/s): min= 2364, max= 5690, per=100.00%, avg=3980.48, stdev=81.61, samples=523
iops : min= 2364, max= 5690, avg=3980.27, stdev=81.60, samples=523
lat (usec) : 1000=0.08%
lat (msec) : 2=31.70%, 4=14.42%, 10=17.37%, 20=14.74%, 50=13.43%
lat (msec) : 100=5.95%, 250=2.20%, 500=0.11%, 750=0.01%, 1000=0.01%
cpu : usr=0.70%, sys=9.60%, ctx=118052, majf=0, minf=16495
IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
issued rwts: total=131072,0,0,0 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=8

Run status group 0 (all jobs):
READ: bw=3957MiB/s (4150MB/s), 3957MiB/s-3957MiB/s (4150MB/s-4150MB/s), io=128GiB (137GB), run=33120-33120msec

Disk stats (read/write):
sdb: ios=525544/0, merge=21/0, ticks=4734876/0, in_queue=3932880, util=99.75%
veeam@vlt-vbrrep01:~$

[david.domask]

why not fio or dd?

Would second fio, it's what your Support Engineer would use for such cases, and we have a few pre-built job files for the various IO patterns Veeam uses; if there's interest in this, I can ask the KB managers to add them to the KB.

I wouldn't use dd, at least, not without being very careful about how you interpret the results and what parameters you use. I've seen dd tests proposed in cases when people (both clients and regrettably my engineers =,=) with if=/dev/random of=/some/repo/path or similar improper tests, and it just causes confusion and frustration.

[david.domask]

Also @BackItUp2020, what kind of performance issues are you seeing? Is it really on the normal read/write operations or are there concerns on the fast clone (reflink) operations with XFS? Or is this more just to test the new repo out and establish some expectations?

[BackItUp2020]

Would second fio, it's what your Support Engineer would use for such cases, and we have a few pre-built job files for the various IO patterns Veeam uses; if there's interest in this, I can ask the KB managers to add them to the KB.

I wouldn't use dd, at least, not without being very careful about how you interpret the results and what parameters you use. I've seen dd tests proposed in cases when people (both clients and regrettably my engineers =,=) with if=/dev/random of=/some/repo/path or similar improper tests, and it just causes confusion and frustration.

I think that would be great! The KB doesn't currently specify those details and therefor the command seems to be in complete.

[david.domask]

Fair enough! I'll ask the KB Manager to look into it, but probably too early to know if/when it will be done.

As for the commands from the article, maybe you can show the full shell commands you're running? I've never bothered with diskspd on Linux since there was fio, but we can adjust the documentation easily enough if it's really drastically different.

[BackItUp2020]

Also @BackItUp2020, what kind of performance issues are you seeing? Is it really on the normal read/write operations or are there concerns on the fast clone (reflink) operations with XFS? Or is this more just to test the new repo out and establish some expectations?

I'm really unsure if we ARE experiencing source issues, so its more testing the repo and going from there. This same repo/disks was a Windows repo prior and after the re-purpose, all jobs to the cloud show the "source" as the bottleneck, so I wanted to see if that is truly the case. We are upgrading to a dedicated cloud backup circuit from a shared internet circuit in hopes that our overall copy times decrease, which is another reason I want to make sure there are no other larger bottlenecks in our environment.

*The bottleneck is the source, but only by a few percent, but I still want to get some performance data. Example: Source: 85, Proxy: 79, Network: 73, Target: 8

The particular job in question that I'm testing with is a 20TB copy job, Active Full to our cloud backup provider. Job throughput is a measly 20-32 MB/s and takes weeks to get a full off-site. In the case we have a "corruption" or other glitch, we have to reseed, etc. and its just not feasible. On top of that, we just want faster throughput

BTW: Veeam Case: 06271014

[BackItUp2020]

I think that would be great! The KB doesn't currently specify those details and therefor the command seems to be in complete.

Thank you!

For instance, support had me run this:

TEST_FILE=/mnt/veeamrepo/test.dat
mkdir -p $TEST_FILE
.\diskspd -Sh -d600 $TEST_FILE

But the file is just empty and you get an error back when running diskspd:
target /mnt/veeamrepo/test.dat's specified size is too small for block size of 65536 bytes

[david.domask]

I will check in on this case tomorrow. I must confess I'm not happy seeing those shell commands since it's clear if you know bash or any similar shell what's wrong with this test I will give the engineer some hints, likely they just got some bad advice from somewhere.

I'll also ask them to send you the appropriate fio job file for random read tests, which should be similar to backup copy behavior,

Just my opinion without seeing the logs though, test it the other way; test what a restore from your cloud storage; I'm not quite getting if you have a Cloud Connect Provider or if you mean an Object Storage repository, but either way, see what it's like the other way. If you see very similar speeds AND the cloud storage is fairly reliably measured, you should have a pretty good idea on how bad the network is really bottlenecking.

The bottlenecks can be a tricky concept since it's important to understand that a high bottleneck doesn't necessarily mean something's wrong, it's just where our datamover agents needed to "wait" the longest, and it doesn't directly correlate to any speed value. So in my mind just based on your bottlenecks, I suppose probably it's pretty true that the datamover agent is busy on your repository, but probably it's still going fast enough to hit the network limit, and that's where the 20-30 MB/s comes from, not the repository. But of course, this is just speculation.

[BackItUp2020]

Thank you so much for the assistance!

I agree - I try not to get to flustered over the bottleneck statistic. I've been trying to get our copy job speeds to iland up significantly over the past year, so I'm looking at everything. On top of it, I feel a bit lost with the Linux repo without the ease (for me) to measure all the stats as easily without googling for tools, etc.

[bct44]

Hello,
https://github.com/microsoft/diskspd-for-linux

diskspd -b512K -r4K -Sh -d600 /repo_path/XY.vbk

It will do sequential read as the data mover will act during BCJ

I think it's probably what you are looking for!

[BackItUp2020]

Thanks for the attempt - I tried, but only got this in return:


/mnt/veeamrepo/bendnav2_2023-08-29_2023-09-29$ diskspd -b512K -r4k -Sh -d600 xxxxnav2D2023-08-28T200347_4409.vbk
Invalid byte size!
Argument to option -r was invalid!

/mnt/veeamrepo/bendnav2_2023-08-29_2023-09-29$ diskspd -b512K -Sh -d600 xxxxnav2D2023-08-28T200347_4409.vbk
ret from aio not equal to block size, it's -22
Error during warmup phase!
Job 0 failed, exiting

[karsten123]

did you try 4K with a K in capital letters?

[BackItUp2020]

I have, same warning about block size.

[bct44]

Well the command example i sent before, works on my system "rhel 8.8"

What is your storage optimization option?

By default, Veeam’s block size is set to local target, which is 1MB before compression. Since compression ratio is very often around 2x, with this block size, Veeam will write around 512KB or less to the repository per Veeam block.

[BackItUp2020]

Interesting. I'm running the Ubunutu 20.04 build.

I'm concerned about a copy job (to the cloud), so there isn't an optimization option. However, on the backup jobs it is set to "Local Target".

[BackItUp2020]

For more color, I think this may be helpful:

Disk /dev/sdb: 181.94 TiB, 200024470323200 bytes, 48834099200 sectors
Disk model: UCS-S3260-DRAID
Units: sectors of 1 * 4096 = 4096 bytes
Sector size (logical/physical): 4096 bytes / 4096 bytes
I/O size (minimum/optimal): 4096 bytes / 4096 bytes
Disklabel type: gpt
Disk identifier: E9BBA124-2C45-43F0-8477-D018B2EF70DB

Device Start End Sectors Size Type
/dev/sdb1 256 48834098943 48834098688 181.9T Linux filesystem

[tyler.jurgens]

These "xxxxnav2D2023-08-28T200347_4409.vbk" aren't actual VBKs, are they?

The diskspd tests should be targeting a nonexisting file, or a file you don't care is overwritten. If you're targeting an actual VBK you may have just corrupted it.

This command:
diskspd -b512K -r4K -Sh -d600 /same disk as repo/tempfile

I wouldn't target the /backups directory itself (if your backup repository is on /backups) because I wouldn't want to risk any backup file corruption. I'd also not name the file .vbk because I wouldn't want to confuse this temp file as an actual backup file.
As long as the target is the same disk as your actual repository (aka, where did you mount /dev/sdb to), then the test is a valid performance test.

[BackItUp2020]

That helps a lot! I must be doing something clearly wrong, because that command gives me back a file not found error (which is why I thought it needed to be pointed to an existing file).

diskspd -b512K -r4K -Sh -d600 /mnt/veeamrepo/test.dat
Target "/mnt/veeamrepo/test.dat" does not exist!


If I need to

[BackItUp2020]

For fun, I created a 10GB file named test.dat(diskspd -c10G /path/test.dat) and ran the same command, but I get the same error:

ret from aio not equal to block size, it's -22
Error during warmup phase!
Job 0 failed, exiting

[tyler.jurgens]

Does the user you are running diskspd from have permission to create a file on /mnt/veeamrepo/? Have you tried running it as root?

[BackItUp2020]

I've tried sudo with no changes. I was able to run the diskspd command to create a file without issue though, so I imagine I'm good there.

[BackItUp2020]

Verified permissions are okay. If I try to run with lesser perms, I get a permission error.

[bct44]

The previous command i sent was to read the vbk files which be used for BCJ as your request. It will just read the file and won't do anything weird on it. I used in the past with the support team because before V12, i had not attended performances for backup to tape. I had to proove my repos can handle the read load.

For your error about block, at this time i have no idea maybe the weekend will give me some

If some Veeam PM can try it on a lab with an ubuntu and do a feedback, it could be great!

[cdw23]

Greetings,

I recently built a Linux Veeam Hardened Repository but used WinDO$ (windows) to test the hardware first. Here are some of my notes to assist. There is good info about testing parameters here: https://www.veeam.com/kb2014.

The errors your getting ... it could be that you're not using high enough privileges. In WinDO$ I have to run from a command prompt I opened as administrator.

Direct Disk Max Read - X = disk management ID #, your target drive / raid array / etc. Under WinDO$ I know you can get this ID # from diskpart.

Code: Select all

diskspd.exe -Sh -d600 #1

You can specify the size of the test file to create all in one command with the test parameters, see "-c100G" in examples below, it will create a 100GB test file with the name you specify at the start of the test.

Active Full / Forward Incremental

Code: Select all

diskspd.exe -c100G -b512K -w100 -Sh -d600 d:\testfile.dat

Synthetic Full / Merge operations

Code: Select all

diskspd.exe -c100G -b512K -w50 -r4K -Sh -d600 d:\testfile.dat

Reverse Incremental (*Not Applicable to an Immutable XFS repository)

Code: Select all

diskspd.exe -c100G -b512K -w67 -r4K -Sh -d600 d:\testfile.dat

Restore / Health Check / SureBackup (worst case)

Code: Select all

diskspd.exe -c100G -b512K -r4K -Sh -d600 d:\testfile.dat

Restore / Health Check / SureBackup (best case)

Code: Select all

diskspd.exe -c100G -b512K -Sh -d600 d:\testfile.dat

A decade ago I would use a ramdisk, dd and very large zip files to get raw read and write speeds of my drives and arrays. BEWARE if your test files are not a LOT larger than the cache on your RAID controller for example you'll be testing cache speed, not drive speed. For example I got ridiculously high transfer speeds when using a 1GB test file on a RAID controller that had 4GB of cache... I ended up using 100GB test file sizes to get meaningful results.

[Eco]

We wrote a tool to make life simpler for this - https://github.com/minio/dperf. Currently there is an issue with multi socket systems but for single socket it is quite nice. Outside of that, as others mentioned fio is the standard tool for real world performance testing, and dd can be used for quick and easy sequential tests. In either case its best to use O_DIRECT flags to bypass cache so you see real world numbers. For dd you want to add `conv=fdatasync` and `oflag=direct`. For fio I think they are ioengine=sync and direct=1.

[BackItUp2020]

I'm still at a loss with diskspd. fio seems to work, but I am still curious as to why I get the error with diskspd.

That being said, I have installed iotop which is a pretty nice utility to monitor current disk IO. Not a speed test, however, but just some insight into activity which I can match up against the Veeam UI/log stats.

[tpayton]

Fair enough! I'll ask the KB Manager to look into it, but probably too early to know if/when it will be done.

As for the commands from the article, maybe you can show the full shell commands you're running? I've never bothered with diskspd on Linux since there was fio, but we can adjust the documentation easily enough if it's really drastically different.

I'm pleased to announce that https://www.veeam.com/kb2014 has been updated to now include 'fio' command examples. Thanks for the suggestion!