Tape Positioning Error

[nunciate]

I have Case#: 00545435 open for this but thought I would post it to the forums to see if anyone has dealt with this.

I built a new Windows 2012 R2 backup server. I have a Quantum i80 tape library with 2 HP Ultrium-6 FC attached drives. The server is physical. I have this exact setup on my old system only using Windows 2008 R2 and all has been working fine for months. When I look at the drivers I see the exact same version (HP v1.0.7.1) on both servers. I tried updating to the latest version (HP v1.0.7.2) from HP and also 1 version below (HP v1.0.7.0) with no luck. I seem to have success with some jobs and failures with others. I get StartSessionTape failed & Tape Positioning errors. Veeam is at the latest patch level, Windows 2012 R2 if fully updated. I am on the latest library/drive firmware and driver level.

I should also mention that I can't do a short erase on tapes either. I get similar errors. I actually have 2 of these libraries in different locations and both hooked up to 2012 R2 servers. Both experience similar issues.

Anyone have any ideas? I would rather not reinstall Windows 2008 if possible.

[veremin]

Are there any other backup software or its remnants installed in the environment, by any chance? Also, are there any additional (non-default) settings, such as compression, encryption, etc. configured on the given device? Sometimes VB&R might inappropriately to both other backup software and specific device settings. Though, the latter is typically the case with VTL libraries, not physical ones.

Thanks.

[nunciate]

Here is how my full environment is setup.

Tape Library: Quantum Scalar i80 (4 HP Ultrium-6 LTOO6 FC Drives)
Library has 2 Partitions (Looks like 2 libraries to the server)
Partition 1 is for Veeam and has 2 tape drives and 20 tapes.
Partition 2 is for SyncSort DPX and also has 2 drives and 20 tapes.
Scalar Encryption is enabled. The Quantum Library handles all encryption at the hardware level there is no configuration in the software needed.

The backup server is a physical HP DL380G8.
The backup server & tape drives are all connected through a FC switch. I actually have separate physical paths configured so the Veeam tape drives only talk to the server over one line and the SyncSort drives talk over another. So I can disconnect the SyncSort side easily for testing.

The server is running Windows 2012 R2 and has both Veeam & SyncSort DPX installed.
All 4 drives are detected with the latest driver in Windows. Windows sees 2 separate libraries as well.
SyncSort only sees the 2 drives and 1 library that are assigned to it. The second library is not configured in SyncSort.
Veeam however detects everything but I manually disabled the 2 SyncSort tape drives in Veeam so it doesn't accidentally use those.

I can tell you SyncSort works perfectly. Backups to disk and to tape work every night.
Veeam is a different story. I had those positioning errors all weekend. For some reason I was able to get everything taped out after monkeying with it for a while. Monday and Tuesday this week went fine. I was able to get all my weekend full tapes outs and my daily incrementals done. Then last night every incremental tape out failed again with these same issues. I should also mention that the quick tape erase operation fails with these errors as well.

Some troubleshooting I have done:
I disabled all SyncSort services but that didn't help.
I removed the SyncSort library by unplugging the FC cable and stopped all services. No luck.
I moved the tape drives around in the library so I used the SyncSort drives as the Veeam drives and visa versa. No Luck.
I removed the SyncSort Partition in the library and assigned all 4 drives to Veeam. No Luck.

I started focusing on Encryption yesterday and discovered some things.
When I turn encryption off completely I am able to erase a tape.
As soon as I turn it back on the erase fails.
In the Quantum setup you can choose to reuse keys so only 1 key is generated per tape or you can generate a new key for every backup.
When key reuse is turned on tape erase fails.
When key reuse is turned off tape erase succeeds.

This is all on my new setup running Windows 2012 R2. I have my old server still setup and I used that for many months. It was running Windows 2008 R2 with the exact setup described above and I never had these types of issues backing up.
In case you are wondering, I did disconnected the old server completely from the drives before testing all of this so it is not conflicting.

[veremin]

The exact same configuration (Library, partioning, said backup solutions) worked fine in case of Windows Server 2008, right? In case of Windows 2008, was the partition#1 with drivers #1 and #2 connected to VB&R and partition#2 with drive #3 and #4 to SyncSort? Thanks.

[nunciate]

Yup it is the exact same setup and the same library on the WS08 server. I mean literally the same library. I connected it to the old WS08 server with the same partition configuration same driver version and was able to erase every tape. The old server is running the latest version of Veeam & SyncSort just like the new one. I then hook it up to the new server and can't erase any tapes with Veeam. Very odd behavior.

Basically, all I did was build a new server with more local storage but otherwise exactly like my old server. I didn't migrate the old database or anything so it is a completely new setup. Then I connected my library over and tried to do tape backups.

[veremin]

You're right, the described behavior looks pretty odd, indeed. Based on the provided ticket number, the case is currently under investigation. Both support and QA team are interested in this case. Hopefully, they will be able to shed some light, once the thorough investigation is conducted. Thanks.

[nunciate]

I really think it is a Windows 2012 R2 driver issue at this point. At least I can't find any other issue within Veeam or the Physical hardware. I have almost identical setup in my DR facility. Same tape library, single physical server running 2012 R2 and I have the same kind of issue. Only in DR I have never been able to make it work. At least here in production I had it working for a few days.

I tested one more time and turned off encryption on the Veeam library partition. I did this here and in DR. I was able to erase tapes with no issues. I am able to backup to tape with no issues. I turn encryption back on and it fails. I know encryption works at some level because like I said SyncSort works fine and Veeam actually worked for the last couple of days. I am just baffled as to why it stopped and what is causing the encryption issue and why it works find in WS08.

I have a ticket with Quantum and have been working through this on that side. I also opened a ticket with HP but I am guessing I will get limited support that way since the drives were sold with the library.

[nunciate]

I have a work around for now. We discovered that if we disabled Key Reuse in the encryption settings on the library we were able to work with the tapes.

I turned off Key Reuse and formatted every tape in the library.
They all showed encrypted now so I turned Key Reuse back on.
I then attempted a backup to tape job and it was a success to multiple tapes.

I did the same process in DR and now that is working again as well.

So it comes down to this. Either there is some base incompatibility with Windows 2012. Either the driver or the O/S don’t send the commands correctly that allow the tape to be formatted or encrypted properly. The only other thin I think it could be is the fact that I was taping out from one server and then physically moved to the new server. Maybe something in the system isn’t responding well to the fact that the tapes that were once written to by the WS08 server are now being written to by the 2012 server.

The underlying issue still exists though. I cannot format or write to tapes that have not been formatted first with the Key Reuse setting turned off. If I try to format or write tapes that display the Not Encrypted or Unknown status within the tape library they will fail as long as Key Reuse is turned on. This has never been an issue in the past.

[veremin]

I would blame the encryption, or more specifically, re-use option, but the fact that the the very same configuration has worked properly in Windows 2008, doesn't allow me to do so. Hopefully, the support team will be able to find the root cause of this behavior.

Thanks.

[nunciate]

I received a reply from Quantum today. It looks like the issue was reproducible and they are testing a fix which requires a firmware update to the drives. Weill post again once I have the fix. They say it could take a while to get through their testing and certification process for firmware releases though.

For now if any of you notice this issue with your tape library you will simply need to disable key reuse on the partition, format you tapes and then re-enable key reuse. This only has to be done once per tape. So any tapes that are new or that have been written with a previous version of Windows will probably need an initial format.

[nunciate]

Just wanted to write an update to this:

I received a firmware update for my HP LTO6 Half Height FC drives from Quantum today. The new firmware is version 23KZ.

I applied this to my 4 drives and this fixed the issue. I am now able to format at backup to new tapes without disabling anything in the library.

I don't think it is available on their site just yet but it should be soon.

[veremin]

Thank you, Alan, for coming back and updating the topic with the found resolution; much appreciated!