HP LTO6 tape issues

[20100]

Hi

We have a new server running WIndows 2012R12 with HP 1x8 G2 Autoloader and HP LTO Ultrium-6 drive
We have not been able to perform a full backup on tape since we got it now and it is becoming a big issue.
The backup fails after anything between 50Gb to 150Gb of data copied to tape.
The backup fails with "BackupSrvFileToTape failed Tape fatal error. The device is not connected. Tape write error Tape fatal error."

I logged a case with Veeam support, that indicates that the issue is hardware.
That seems to be correct, as we find the following error in the Windows Event at roughly the same time we got the problem
"Event ID 15, source hplto: The device, \Device\Tape0, is not ready for access yet"

I logged a case with HP, and they replaced the tape library, the tape drive, gave me a new set of tapes and also check for firmware and drivers.
However the problem still persists.

I am now in the middle with the problem, with both HP and Veeam telling the problem is software or hardware respectively.

Has anyone got a similar issue with HP LTO6? Or had one and it got solve?
If it is working for you, would you mind letting me know which Drivers, etc you are using?

Any help would be greatly appreciated as we do not have any backup on tape since we switched to Veeam and the new hardware and it is an uncomfortable situation to sit on for too long.

Thanks

[martynuk]

We use Veeam 8 with an HP MSL G3 series tape library and HP LTO Ultrium-6 drives. Never had a problem like this. I actually upgraded the Windows HP drivers for the tape drives to v1.0.7.2 just last week, and still working fine. Our B&R server is Windows 2008 R2.

Let me know if I can provide any further info on our setup which you think might help you.

Martin

[veremin]

Can you post you case number here, so that I can ask QA team to take a look at it?

[Isaksson]

Server 2012 and HP MSL G3 LTO Ultrum-6 I am still using the old 1.0.7.1 driver and i have not seen any issues like that.

[kmbelt]

I am having this exact same issue, only i have an LTO5 drive in my G2 Autoloader. I called and was asked if i was using New tapes or tapes that have been previously used on another system. I'm using tapes that have been previously used. I was then asked to do a Full Erase on them, which is actually running right now.

I have upgraded the tape library and drive to the latest firmware, which fixed nothing.
Server is a fresh 2012 R2 that is fully patched.
I know the tape library is good as we used it Thursday night with NetBackup without issue and then reloaded the server for Veeam.

I'm leaning towards a software issue. at this point, because i have my master server running 2012R2 with a MSL4048 that works fine.

Autoloader driver is HP dated 7/26/2013 and is driver version 3.0.0.3
LTO5 drive is using HP driver dated 7/8/2013 and is version 1.0.7.2

I'd love a resolution to this issue also.

[20100]

Hi

Our case number is 00816784

I have exactly the same drivers as kmbelt (same for LTO5 and LTO6)

Firmware for autoloader: 4.30
Firmware for LTI6: 352W.MSL G3

The HP engineer is coming back today to swap the SAS cable. Will see if that helps...

Regards

[kmbelt]

I have been working on my system all day. I do not believe the issue on my end is HW related as it all worked last week Thursday with NetBackup. Friday i took the backup infrastructure from Server 2008R2 to 2012R2 on the same machine. Everything else stayed the same except for the firmware and drivers.

It has been hit and miss today. One time a job will run, the next time it will not. I have created all disk jobs, media pools, tape jobs from scratch and am getting the same results really.

If i come up with somethign new, i'll be sure to post. I also have a ticket open with Veeam. #00820520. So far, all i've gotten is "have you tried a different tape"...

Edit:

I just thought to compare the library config of my 1/8 Autoloader with my Veeam fully functional MSL4048 I have also. I noticed one difference in the autoloader mode. I made my 1/8 match the settings of my 4048 in that it now is set to Random instead of Automatic (Default). We shall see if this makes a difference. I'll post my outcome.

[20100]

The SAS cable has been replaced and it did not make any difference.
HP does not believe there is a hardware issue. They asked me to try another backup software!
Windows Backup to tape does no longer exists since 2008.

Does anyone knows a quick software that can handle library?
Like kmbelt, I want to establish once of all if it is Veeam, Microsoft or HP

At the end, I am stuck with both of 3 and I really need a working solution.

[seadave]

I've been in a similar boat with an older Dell TL2000. Worked fine under CommVault Simpana, nothing but problems now that it is connected to Veeam. One thing that made the situation slightly better was to increase the timeout period that Veeam uses. Default is 60s. Try this reg entry:

HKLM\SOFTWARE\Veeam\Veeam Backup and Replication:
REG_DWORD
TapeDeviceNotReadyTimeoutSec
Hex=78 or Dec=120

Maybe try increasing it to 180, not sure if that works. To test your library, you might consider standing up a Unitrends VM. I think that will address your tape library as long as you can address it from the VM (would need to be over IP).

http://www.unitrends.com/products/software/download

I think Veeam may come to realize that relying on native Windows drivers is a crap shoot. They need to pick a pool of supported libraries/tapes and write native drivers for them. That is the only way they can maintain quality and consistency.

[tsightler]

The SAS cable has been replaced and it did not make any difference.
HP does not believe there is a hardware issue. They asked me to try another backup software!
Windows Backup to tape does no longer exists since 2008.

Does anyone knows a quick software that can handle library?
Like kmbelt, I want to establish once of all if it is Veeam, Microsoft or HP

At the end, I am stuck with both of 3 and I really need a working solution.

No matter what the problem could be in either the HP provided driver or Veeam, so using another software might not really narrow things down that much as much software on the market drives the tape hardware directly. Using another package might eliminate hardware, but that seems unlikely to be the issue. To me the most likely scenario is that this is somehow related to the HP driver. Looking at the logs on your case it's hard to see how it could be anything else as the error seems very clear. I'm wondering if it could be some type of strange interaction between the SAS connection and the HP failover support in the driver. We've tested the HP hardware pretty well in the HP lab and I've not seen any such issues, however, these all used FC connectivity. What kind of SAS controller are you using to connect to the library?

[tsightler]

If i come up with somethign new, i'll be sure to post. I also have a ticket open with Veeam. #00820520. So far, all i've gotten is "have you tried a different tape"...

Looking at your case notes I see that your tapes have labels that imply they are LTO-4 media. Are you actually using LTO-4 media or just old labels? The HP 6250 LTO-6 drive supports reading LTO-4 tapes, but not writing to them. It only supports writing to LTO-5 and LTO-6. I thought it was worth asking.

[20100]

Hi
Sorry I am not an hardware specialist, so not 100% sure which one is the SAS.
Looking at the device manager, storage controllers we have:
Emulex AK763B/AH403A, PCI SLot 1, Storport Miniport Driver
Emulex LighPulse HBA Storport Miniport Driver
Microsoft ICSCI Target LocalMount Adapter
Microsoft Multi-Path Bus Driver
Microsoft Storage Spaces Controller
Smart Array P420i Controller
Smart Array P822 Controller

I believe the OS drives are on SMart Array controller and the data disks on the other one.
We also have Fiber connectivity to an HP 3PAR

Also, I tried the TapeDeviceNotReadyTimeoutSec set to 180 as mentioned by Seadave, but it did not make a difference. This time it failed after 185Gb

Thanks

[20100]

about the LTO labels and Media: no the tapes are really LTO6 and also the labels are especially for LTO6

If I run the HP Library and Tools utility, the inventory shows LTO6.

I noticed that Veeam detected the tapes as 2.4Tb instead of 6.5T, but you told me it was normal:
http://forums.veeam.com/tape-f29/lto6-c ... 26331.html

[tsightler]

Sorry I am not an hardware specialist, so not 100% sure which one is the SAS.

No problem, we'll work through it. I always like to start out by fully defining the configuration. It should be very easy to figure out via device manager if you select "View...Devices by Connection", you can then drill down through the tree until you find the controllers and then open the controllers and see what specific devices are attached to it.

[20100]

Sorry Tom, I cannot find much. My lack of knowledge is showing up!
When I select the Device by Connection, for Storage I can only see:
Microsoft iSCSI Target LocalMount Adapter
Microsoft Multi-Path Bus Driver
Microsoft Storage Spaces controller

When I open each of them, I get nothing but the standard, General, Driver, Details,Events!

[tsightler]

about the LTO labels and Media: no the tapes are really LTO6 and also the labels are especially for LTO6

If I run the HP Library and Tools utility, the inventory shows LTO6.

I noticed that Veeam detected the tapes as 2.4Tb instead of 6.5T, but you told me it was normal:
http://forums.veeam.com/tape-f29/lto6-c ... 26331.html

The label question was to the other poster in this thread (he's the one who's post I quoted and it was his case notes I was referring to).

[20100]

I think I got it:
It was "Smart Array P822"
I went to properties and selected "Children"
I can see values for the disk volumes and for the LT6 and for the Library autoloader. Looks like it is the same controller

[tsightler]

That's what I was expecting. My immediate concern is that the P822 is not certified by HP for use with tape devices (at least as far as I know, I'm not an HP expert). HP is typically quite picky about what specific SAS controllers are used with tape devices and normally does not even support mixing HDDs and tape devices on the same controller. SAS controllers that have been tested with tape support specifically list this support in their Quickspecs. I'm still researching.

[20100]

Thanks Tom

I will also mention this to HP support case.
They are the one that sold us that config!

[tsightler]

Thanks Tom

I will also mention this to HP support case.
They are the one that sold us that config!

I don't even think that config is allowed to be sold on the website. When you select a tape device they specifically require to to either add a secondary SAS controller or at least dedicate a controller to the tape device. It's at least worth investigating. Sometimes those high end controllers are more sensitive to SCSI timeouts (which is normally a good thing for RAID) and thus are more aggressive at taking devices offline rather than retrying commands. However, for a tape device this is less than ideal, you don't want the controller taking devices offline. I mean, the logs are pretty clear, Veeam is happily writing to the tape device and suddenly it gets a "tape device not available" error and, based on the case notes, it looks like there is also an error in Windows event log. That seems unlikely to be an issue with Veeam, but perhaps not a hardware error either, at least not "broken" hardware, perhaps just hardware that doesn't work that way.

I'm still researching though, it's just another avenue worth looking at for now.

[tsightler]

So I'm having some timeouts trying to access the HP website which is making the research difficult. I'll try again tomorrow. I see that the P822 was recently added to the BURA compatiility document, but I can't tell if that's for tape devices or just some specific devices. I'm trying to find the specific release notes for the firmware versions. The BURA document specifically list firmware 6.1 and driver version 62.28.0.64 so I guess we can check on that in the mean time.

[20100]

Thanks Tom

Will carry on with your info. Driver is the same. Firmware is 6.00
Will try to upgrade and let you know

[kmbelt]

To the OP. I believe i have fixed my issue that is identical to yours. Somewhere during all my searches i saw where someone mentioned the HP Storage Administrator software causing conflicts for some reason. I am assuming your server is HP as mine is also. I uninstalled both my HP Storage Administrator and Administrator CLI from my system and now magically my tape backups work correctly. Go figure, might be worth a shot for you too.

[kmbelt]

Just to clarify what i uninstalled. It was specifically called HP Smart Storage Administrator.

[tsightler]

Just to clarify what i uninstalled. It was specifically called HP Smart Storage Administrator.

Interesting. I've heard of the HP Insight Agents causing similar problems. The agents send polls to the SCSI devices and if they take too long they send SCSI resets to the devices. I had not run across the notes on HP Smart Storage Administrator but I'm guessing the cause might be similar.

[20100]

Unfortunately it did not work for me. I removed both. Also still chasing HP about the firmware 6.1. I cannot find it on their web site

[20100]

The saga continues with HP: the 6.10 firmware does not exists, and now I am told to go back to HP sales, as the autoloader is not a supported configuration for that controller in that particular server (StoreEasy 1840 which is a DL380p Gen8 rebadged with Windows Server Storage 2012R2 + HP software).
That solution was quoted and sold to us by HP!
Seems like the easiest option is to add a separate controller for the tape, unfortunately all the slots are used at the moment with the P822, the HBA for 3PAR connectivity and a 10Ge Nic card.
Looking today at options to find a suitable controller that might fit inside the server directly on the mother board

I'll keep you posted

[sf4692]

Hello all, i think i had the same issue 3 months ago with Win Server 2012R2 and a standalone LTO5 tape. the reason for the non working backup is not Hardware it is the WBEM Driver from the latest SPP Version. (HP also tried here to exchange the drive etc.) this Workaround (officially from HP) worked for me perfectly fine:

First, Uninstall “HP Insight Management WBEM provider” if customer does not use. This is installed by default during SPP installation. This could be unselected during SPP installation.

How to Uninstall please refer Aasim email (with Screenshot). Select “HP Insight Management WBEM provider” and click Uninstall.

So this is fixed…

Second, If the Customer wish to use “HP Insight Management WBEM provider” then downgrade to 9.50

Downgrading WBEM Step by Step…

Open IE and type http://www.hp.com/go/support

Select Drivers & Download..

Add Server Model - Example : DL380 G7

From this select Operating System which is installed.

Expand Software – System Management (21) to choose HP Insight Management WBEM provider Agent

From list select .. you can see the version 10.0.0.0 and previous version 9.5.0.0

Click on revision history,

List of the Version available for download…

HP Insight Management WBEM Providers for Windows Server x64 Editions

Version:10.0.0.0 (9 Sep 2014)

Version:9.5.0.0 (18 Feb 2014)

Version:9.4.0.0 (10 Sep 2013)

http://h20564.www2.hp.com/hpsc/swd/publ ... ab-history

Download and Run as administrator.. you would be able to forcefully downgrade WBEM version Or Uninstall WBEM 10.0.0 and reinstall 9.50.

With kind regards / Mit freundlichen Grüßen,

Velimira Kovacheva

Engineer I HP Storage/ HPSD

GSD Customer Solutions Center Sofia

velimira.kovacheva@hp.com

kovacheva@hp.com

v.kov@hp.com

Hewlett-Packard Company
Business Park Sofia, Building 8

Sofia 1766, Bulgaria

HP

Description: Description: Description: Description: Description: cid:image002.jpg@01CAF03B.A163CF10 Think before you print.

[NowOl]

Hello,
i can confirm the solution from sf4692. We downgraded the WBEM Components to the Version 2014.06 of the HP ServicePacks (i think Version:9.5.0.0 ). We had Contact with HP - Support( also with Velimira Kovacheva )
We are using an autoloader. Since december the backup jobs woks fine.
You can test it also with the HP L&TT Tools > Full - Tape Test.

@ HP : When will there an fixed Version?

Martin

[EdZ]

I have the same issue. I've been working with support for a while with no luck.

HP Autoloader 1x8 G2 with the 4.30 firmware drive firmware U62W.

We've gone through the HP tape tools and they test out fine. Issue is either with Veeam or with how Veeam is passing to WMI as far as we can tell right now.

So erasing tapes and then running is working for some of you? I may try that today because I've got to get a more current backup off site.