Tape Throughput?

[rboynton]

We just began using an IBM Tape Autoloader with LTO8 tapes. The throughput is holding around 270/MBs, but what I found interesting is that the bottleneck is showing target only at 3%, with the others all at zero. This tells me there is a lot more throughput available that is not being used. Am I reading this wrong?

[skrause]

One thing to remember about the B&R console: there will always be a "primary bottleneck", even if things are running optimally.

Are you using software encryption or hardware (or no) encryption? Software encryption will usually drop your throughput from the max by quite a bit, though you are running at about 75% of max theoretical speed which is not terrible.

[rboynton]

We are not using any encryption. Though I understand there will always be a "bottleneck"; the fact that the target device is only 3% used (apparently), and the other devices are at zero tells me there ought to be way more performance than I'm getting. Also, according to the IBM manual, the maximum performance is 750 MB/s with 2.5:1 compression (native 300 MB/s). Am concerned since we are not getting even native throughput levels. We are connected using 6 GB/s SAS.

[skrause]

That % number is not exactly a utilization number. The highest % number is always going to be reported as the bottleneck.

And if you are sending compressed backup files to the tape, you won't get the compressed speed you will see the native speed (at best, trying to compress already compressed files can sometimes lead to slowdowns).

Are you running the newest Non-Exclusive IBM drivers?

[rboynton]

We downloaded the most current drivers for Server 2012r2 from the IBM website. Not really current, think they are dated 2015.

[rboynton]

Something I am a bit puzzled about though; The LTO8 tapes are 12tb native/30tb compressed, but Veeam only shows them at 10.9tb. I understand formatting will take some, but 1.1tb? Also, if only 10.9 is presented to Veeam, how will Veeam know that up to 30tb can be written to the tape compressed? It seems to me that Veeam will only write up to what it sees. Am I missing something here?

[ers_kentwick]

We use an IBM TS3310 with LTO5 tapes. LTO5 tapes are supposed to hold 1.5 TB normal and 3 TB compressed. Veeam reports them as having 1.4 TB capacity.
Once thing that I learned while we starting to implement tape for Veeam is that when Veeam interrogates the library it will report on the normal capacity of the tape. As such Veeam, since it always sends compressed data to the Tape Server, will only write up to the "normal" tape capacity. Veeam (at the time we started with tape) does NOT attempt to use hardware "end-of-tape". So any hardware compression occurring at the tape drive level is compressing an already compressed data stream. (or at least that is the way it is behaving for us).

[rboynton]

Wow, thanks for that input. With that in mind; we should forget any compressed ability with these tapes and look to native, er 10.9 out of 12tb available. Those tapes are reaaaaally expensive (until the lawsuit is over), so will have to shell out lots of bucks to make this work.

Now, back to the throughput/backup rate issue. Are you getting anything close to the rated throughput for your backups?

[tomnewman]

The native LTO-8 transfer rate with LTO-8 tapes is 360MB/s (300MB/s when used with LTO-7 tapes).

If you are sending compressed data e.g. secondary backup to tape from a compressed primary backup, then you are not going to see anything better than that. Also, if you are sending compressed data then don't enable compression on the tape drive, as has already been said, this won't improve anything and can just make it worse.

We get about 275MB/S using LTO-7 tapes so pretty close to the theoretical maximum of 300MB/s. You don't actually say what throughput you are getting but if you are sending compressed data then you aren't ever going to get over 360MB/S

[fgw]

to add my experience with HPE LTO8 drives here:

we using half height drives not full height, so throughput is about 10% to 15% lower anyway.
ignore compression! as already said by others, veeam data is already compressed, thus the tape drive itself wont be able to compress any further! i would stick to the recommendation to even disable compression at tapelevel.

what i see here is bandwidth up to about 300MB/s which is pretty good! although this 300MB/s are not kept constantly over a 5 hrs backup session. it depends how steady data can be delivered to the tapr drive. so there are periods which may last seconds or even minutes when bandwidth drops.
but over all i can see about 250MB/s average over several hours!

another point i observed during my testing is temperature. if the tape gets hot, bandwidth will drop! during testing i used a tapedrive in an unsupported enclosure where cooling could not be maintained as required. bandwidth was at specification for about 10 or 20 minutes but after that it dropped down to 100MB/s. if u see this behaviour, i would suspect cooling issues! its always worth to check this too ...

but to sum it up:
forget about compression, recommendation is to disable compression at tape level
expect something near native speed, may be around 300MB/s or even up to 350MB/s for full height drives but nothing more
expect about 11TB capacity per LTO8 cartridge
keep an eye on cooling/temperature if bandwidth drops after 10 minutes of writing to tape

[mkretzer]

What kind of repo do you have which can stream 270/MBs? We have much worse rates...

[tomnewman]

Synology NAS with 12 x 7200rpm SATA hard drives (RAID 6) configured as a Linux repo with a 10Gb NIC.

I've never done a proper test but from experience I would say it delivers 450-500MB/s.

The thing that I found made a BIG difference with B&R was using jumbo frames. Also moving from CIFS/SMB to Linux distro also made a noticeable difference in backup times but you have to have a NAS that has a decent CPU and memory or it can be better to stick to CIFS/SMB.

[tomnewman]

we using half height drives not full height, so throughput is about 10% to 15% lower anyway.

You must be as old as I am! There used to be a speed difference between full and half-height tape drives but I don't think that has been the case for a looong time. Does anybody even make full-height drives any more?!

[DaveWatkins]

We are not using any encryption. Though I understand there will always be a "bottleneck"; the fact that the target device is only 3% used (apparently), and the other devices are at zero tells me there ought to be way more performance than I'm getting. Also, according to the IBM manual, the maximum performance is 750 MB/s with 2.5:1 compression (native 300 MB/s). Am concerned since we are not getting even native throughput levels. We are connected using 6 GB/s SAS.

Just to clarify this, those percentage numbers aren't a measure of throughput or utilization, they are a measure of the percentage of time Veeam spent waiting for that device.

So in your case the only thing Veeam ever has to wait for is your tape drive, and then only 3% of the time, which is an excellent situation to be in.

[mkretzer]

Thats kind of impressive. Do you use REFS? We do get ~130 MB/s from a fibre channel array with 128 Disks. Tape drive direct attached via Fibre Channel.

With multiple streams the array can push > 2 GB/s but not single stream...

[fgw]

You must be as old as I am! There used to be a speed difference between full and half-height tape drives but I don't think that has been the case for a looong time. Does anybody even make full-height drives any more?!

probably even older *gg*

yes, this info may be outdated as i stumbled across this in an hp document some time ago. u seem to be right as i cant find full height drives in current hp libraries anymore anyway.

[fgw]

Thats kind of impressive. Do you use REFS? We do get ~130 MB/s from a fibre channel array with 128 Disks. Tape drive direct attached via Fibre Channel.

With multiple streams the array can push > 2 GB/s but not single stream...

if u was asking me:
using a single phys server to act as proxy and repository.
there are 12 x 1.8TB 12G SAS 10K 2.5IN disks configured in RAID 5
connection to backup server wich also acts as tape server is trough 10Gbs link
this config easy can stream more than 500MB/s which just half way saturates the network link
bandwidth writing to tape (HP LTO averages as said at about 260-270MB/s over 7 or 8 hours and peaks often at tapes specified max of 300MB/s

[MGT1981]

So a few things here, First, your SAS tape drive is definitely NOT 6 GB/s (or 6 Giga BYTES per second) it is 6 Gb/s or 6 Gigabits per second) This makes a huge difference and has caught me on numerous occasions in the past. Veeam on the other hand, reports throughput using BYTES per second.

for reference, 1 Gb/s only equals 125 MB/s or about 1/8 of a GB/s. I completely understand WHY Veeam reports this way, it does not mean it has not confused many a trained eye over the years.

So when you say you are 275 MB a second you are really getting over 2 Gb/s transfer which is pretty good.

you are probably not going to be in a situation where you are going to saturate a 6 GB eSAS card unless you are sourcing from fairly fast storage and going over 10 Gb (I almost did big B there) networking etc etc, and if you got to that point then you would probably be past the point of using a 6 GB SAS HBA and would want to look at a 12 GB HBA or a FC drive.

Also, I believe (Gostev please correct me if I am wrong) that Veeam considers anything after the proxy as "target" so in other words if your veeam setup has to go from the proxy over a network to a server that hosts the tape drive and that server only has 2 bonded 1Gb/s connections, you will only get up to 250MB/s throughput and it will show as "target"

Long and short, from experience, you can go mad trying to chase these numbers down and improve them, but it actually sounds like your throughput is pretty good all things considered

[aceit]

IBM LTO7 here, via FibreChannel, pushing 280MB/s to 290MB/s as reported by Veeam.

On LTO7 I would define something greater than 260 as "good", don't nitpick here as long as the time window is reasonable... some hardware/software vendors report number in base 10 other in base 2 (ie. MB or MiB)
If your environment allows, you can use parallel tapes to decrease the time window if necessary.

Don't expect miracles migrating from LTO7 to LTO8 bandwidth wise... but the announced LTO-9 should provide quite a bump, or at least is expected to if respects the announcements of 708 MB/s ...

[Brian_Larsen]

You must be as old as I am! There used to be a speed difference between full and half-height tape drives but I don't think that has been the case for a looong time. Does anybody even make full-height drives any more?!

https://www.dell.com/en-us/work/shop/po ... pe-library.
If you look at the specs for LTO-8, there is still a difference between full and half-height drives in regards to speed performance.

Transfer Rate and Backup Rate Per drive:
LTO-6: 160MB/s, 576GB/hr
LTO-7: 300MB/s, 1080GB/hr
LTO-8: 300MB/s, 1080GB/hr (half-height)
LTO-8: 360MB/s, 1296GB/hr (full-height)