Low performance writing to LTO 8 drives

[SSowa]

Veeam Support Case ID: 02370198

We currently are using a Quantum Scalar i40 with 2 LTO 5 drives direct connected to our tape server using a fiber channel HBA (not connected to a SAN Switch). We are looking at upgrading to a Quantum Scalar i3 with LTO 8 drives and this is connected to our Brocade SAN switch.

We are getting terrible write performance on the LTO 8 drives with an average write of only 76MB/s reported on the job, where the LTO 5 drive gets 117MB/s. To make this case even stranger, when I look at the throughput graph for the job, the LTO 5 has pretty much a sustained write with only an occasional dip. The LTO 8 drive will burst up to 330MB/s for about 1 minute, then fall to 0MB/s for around 10 minutes, then 107MB/s for a couple minutes before repeating this cycle until the job is complete

The firmware on the Quantum i3 is up to date as of today's 278G firmware release. The tape firmware is the N9M1 firmware that is the only firmware available at this time. The I have the IBM tape drivers loaded with the Quantum i3 Media Changer driver loaded on the Windows 2019 OS

To eliminate any additional slowdown, my test job of 650GB across 5 VMs is running with no encryption. I store the backup to disk first, then transfer to unencrypted tapes (again, just for the test to remove any extra processing). I have tried storing the back to disk job to both local storage, and to SAN storage. The results are the same.

The Brocade FC switch reports no errors on the ports that the tape drive, or the server are connected to.

I have a ticket open with Quantum as well, and so far the technician is stumped

[HannesK]

Hello,
I guess a driver / hardware / firmware issue (I heard so many ugly things...). But that's nothing I expect to be solved on a forum. I suggest to continue with support.

There must be (Quantum) tools that can do speed tests without Veeam.

Best regards,
Hannes

[soncscy]

Just my take, IBM/HPE tools should work directly with the drives regardless of it being a Quantum Library -- test a full-tape write test with no compressible data (e.g., ensure whatever data is generated is 1:1 written to tape, as backup files are already compressed typically so you should not expect compression benefits)

Get this first and see how the performance is. Remember to stop Veeam Services on the tape server else the tools might not be able to lock the drives.

The library firmware I doubt is relevant here, this is about writing to the drives; while the Library will be important for feeding tapes into the drives, it's not involved in the data transfer path.

Don't look for errors here, it's almost certainly not an error thing; I would not be surprised if the quantum logging things everything is "A-Okay", as errors are like CRC errors or connectivity errors, not performance errors.

I have my team pushing a LOT of our clients to tape, and you need to remember that bad performance does not mean errors!

Errors are just that -- a problem that could not be recovered; if the data is writing, it means no error that cannot be overcome. Don't let the Support (any) tell you there's no performance issue just because there are no errors, this is just an immature understanding.

If I were you, I'd ask Quantum how to simulate a full tape-write from your tape server without compression independent of a backup application. If they cannot do this, you need to escalate with Quantum as you have a support person who is more interested in closing the case than understanding the issue.

A test from the Tape Web-UI is not valid! It misses the tape driver and connection between the tape server and tape hardware, so do not accept this!

[SSowa]

Thanks for the advise on the IBM tools. That was getting pretty close to the same results as Veeam was. I'll have to test again with the full write test, but it does help me with giving info to the techs

[vmtech123]

For reference my LTO8 lib gets about 300MB/s on each drive. It's IBM not Quantum, but Veeam works great with it. I'm usually at 1.6GB/s on a backup using 5 drives.

I had a few issues that were FC related. Bad SFP in the server caused jobs to fail intermittent, slow speeds etc (was logging errors on switch though)

I had another one, that was poor signal causing a similar issue to yours. Looking at the FC switch gui, I could see the optical power was much lower than the rest. It was not giving errors. Changed the SFP and it started working.

You could swap a few SFP's, and/or cables to rule it out.

I would lean towards a config/driver issue. Veeam is able to hit 330 which is about the max for LTO8.

[TaylorB]

If you have tape compression turned on, that can cause this. Veeam files are already compressed so hardware compression works really hard to do nothing.

We get about 275MB/s to each drive, uncompressed. LTO 8 are so fast that often the disk will become a bottleneck. It doesn't take very many drives running near 300MB/s to max out your standard 12x 3.5" 2U disk array.

[joachimeberhard]

I use a HPE LTO-8 Drive as well and get around 280MB/Sec steadily from a entry-level NAS with 10Gbit and virtualized Veeam + Tape proxy.
You should look under bottlecks in the job details, maybe there is a good indication of your problem.