Slow backup with new LTO-8

[bdoe]

This is about case 03514202.

My primary Veeam server is a 12 core, 32GB system, with Server 2012 R2, and runs Veeam itself as well as working as a processing proxy. A second system with 16 cores, 64GB RAM, and Server 2012 runs as a proxy. Both Veeam servers have lots of attached storage, and three Linux repositories are in use. All are connected by 10GbE.

The main server has a Quantum Superloader 3 LTO-7 autoloader with 16 slots, connected using a Dell SAS 12Gbps HBA. This drive consistently runs at 260MB/s. The second system just had a new LTO-8 Superloader 3 installed three weeks ago. Tape jobs run every weekend. So far, each time it has only run at 60MB/s, using LTO-7 media formatted M8.

The thing I've noticed is that the LTO-7 is using hardware encryption every time, while the LTO-8 is using software encryption. All the Veeam media pools are set to use encryption, so far as I can tell Veeam is set up correctly. The Superloader 3 doesn't have anything in its configuration to enable or disable it. I've gone through the browser interface on both and verified they're set exactly the same. I opened a case with Quantum and was told the following:

I have been informed if Veeam does all the key exchanges, it can be setup with HW Encryption. The keys go over the SAS interface. So if you want to setup both the LTO7 and LTO8 for HW Encryption, you may. I understand, SW Encryption is slower as it has to encrypt the data before sending it to the tape drive.

Any ideas? I must be missing something simple.

[asc4]

I currently try to start encryption on tapes and made the experience that I had to enable enrcyption on the tape library itself to run on hardware encryption.
On my IBM TS3310 the logical libraries have an attribute "Encryption Method" that I had to change from "None" to "Application Managed Encryption". After changing this setting Veeam tape jobs were hardware encrypted.
But last week I went to Update4a of B&R and hardware encryption is gone. Will be Investigated under case 03525887.

[bdoe]

I should have mentioned, when I installed the LTO-8, I was on Veeam 9.5 Update 4. The case advised upgrading to 4a, which I did, but with no change. Fortunately it didn't break encryption for the LTO-7. Wish there was an option somewhere on the Superloader 3 (both have IBM drives, for what it's worth), but apparently it should just work.

Also, I'm currently running manufacturer diagnostics on the drive, and it is writing at nearly 300MB/s to an M8 tape. The drive itself and the server seem to be fine, so for the moment I think it's a Veeam issue.

[bdoe]

Disabled encryption on the job and restarted. It's running at 280MB/s now, on M8 media. I cancelled that because it's not encrypted, but it seems to rule out a lot of things.

[Dima P.]

Will be Investigated under case 03525887.

Axel, encryption logic was not changed between Updates, so there should be some infrastructure specific reason. Please keep working with our support team and let us know if you need any further assistance.

Disabled encryption on the job and restarted. It's running at 280MB/s now, on M8 media. I cancelled that because it's not encrypted, but it seems to rule out a lot of things.

Bryan, if hardware encryption is enabled, Veeam B&R automatically disables its software encryption mechanism for tape device. Can you please check with your hardware vendor if the hardware encryption is support by your device (you might need to get additional license key for that if I remember correctly). Cheers!

[bdoe]

My understanding is the Superloader 3 has it enabled, with no option to change that. I never had to do anything extra for the LTO-7 one to work. I'll check with them again though, maybe they have something that can verify it.

[bdoe]

Has anyone used LTO-7 M8 media with Veeam, and had hardware encryption work? I'm basically down to the SL3 is malfunctioning, or Veeam/SL3 aren't liking the M8 media. I will test encrypted LTO-7 media with it if I can find some.

[Dima P.]

Bryan,

Can confirm that Veeam B&R full supports M8 formatted media and, according the whitepapers hardware encryption should work for such tapes as well. Please reach your vendor to check if the hardware encryption is being supported by you device (might be an option in the tape library UI).

[bdoe]

Quantum swears up and down the SL3 has encryption enabled with no way to turn it off. It is definitely not in the UI anywhere. I tried running a file to tape job on an LTO-7 (not M8) tape in the LTO-8 library, and it still uses software, which eliminates M8 as an issue.

So:
-All Veeam media pools have encryption enabled
-Veeam is configured to use hardware encryption when available
-Veeam uses hardware encryption on the LTO-7
-Veeam does NOT use hardware encryption on the LTO-8
-This has been the case on both Veeam systems
-Non-encrypted jobs run at expected speed on LTO-8
-All drivers and firmware are up to date on both systems
-I've even tried changing cables and HBA's

Therefore, seems like the LTO-8 drive is the issue, not the software.

[moondew]

I have a Super loader with an IBM LTO8 drive and LTO8 Media and we use hardware encryption.
My bottleneck is the source but doing a file size over time calculation and my jobs run at an average of 120MB/Sec

[bdoe]

And you didn't have to do anything special to get the hardware encryption working, right? I never have with the Superloader in the past. I think this is the fourth generation of them I've used with Veeam, not to mention several replacements (I had a bad run of defective LTO-7's last year), and hardware encryption always just worked.

[mkretzer]

We also have no issue with Quantum i3 and LTO8 and HW encryption. It just works,
In only Veeam would not read single-stream from source and would provide good speed (sorry)...

[moondew]

I didn't have to do anything special to get it working. Pretty much default out of the box config, IP changed for management interface and unrelated changes like cleaning tape options.
Server the autoloader is connected to is a HP unit with a P431 controller
Tape driver version is 6.2.6.6
Block size on the drive in veeam 262144 (perhaps unrelated, its not the largest option listed)

[bdoe]

That's basically what I've always done. For now I'm sticking with the drive is defective and keeping on Quantum about it.

[Dima P.]

James,

Block size value should not be related to the hardware encryption.

In only Veeam would not read single-stream from source and would provide good speed (sorry)...

Noted as a feature request

[asc4]

Axel, encryption logic was not changed between Updates, so there should be some infrastructure specific reason. Please keep working with our support team and let us know if you need any further assistance.


Bryan, if hardware encryption is enabled, Veeam B&R automatically disables its software encryption mechanism for tape device. Can you please check with your hardware vendor if the hardware encryption is support by your device (you might need to get additional license key for that if I remember correctly). Cheers!

Dmitry,

Level 2 confirmed that there have been changes made to encryption logic between U3a and U4 to overcome a bug where encryption was tried on tapes that had no license to do so.
Unfortunately this code change throws B&R to software encryption under circumstances that have not been fully understood yet.

Bryan,
please take this information to your support contact. Your case, where for one library hardware encryption capability is still honored, and on the other one not, might be an interesting case for the support guys to come closer to a solution. My case is 03525887

Regards Axel

[kenmil66]

I have used LTO-8 in a Superloader 3 with no problems in the past. I have had to use LTO-7 because LTO-8 tapes are in short supply due to some legal issues between the LTO manufacturers. Most suppliers do not have LTO-8 and if they do they are charging HIGH prices.

[bdoe]

I have used LTO-8 in a Superloader 3 with no problems in the past. I have had to use LTO-7 because LTO-8 tapes are in short supply due to some legal issues between the LTO manufacturers. Most suppliers do not have LTO-8 and if they do they are charging HIGH prices.

That's exactly why I'm using LTO-7 (formatted M8), the LTO-8 media is just ridiculous. I'd like LTO-8 for even less tapes, but M8 is a good compromise. Glad to hear of yet another with no problems.

Bryan,
please take this information to your support contact. Your case, where for one library hardware encryption capability is still honored, and on the other one not, might be an interesting case for the support guys to come closer to a solution. My case is 03525887

Thanks, I'll add that. My case was escalated, so I'm waiting to hear from the next engineer, hopefully today.

[bdoe]

I wound up trying three different servers, four HBA's, and even installed Backup Exec. In the end Quantum sent a replacement drive, and on the first attempt with Veeam it ran with hardware encryption using LTO-7. On the second job, hardware with M8. I have no idea what was wrong with that first drive, but it was the cause.