HP LTO6 tape issues

Everything about backing up to tape

HP LTO6 tape issues

Veeam Logoby 20100 » Mon Mar 02, 2015 1:06 am

Hi

We have a new server running WIndows 2012R12 with HP 1x8 G2 Autoloader and HP LTO Ultrium-6 drive
We have not been able to perform a full backup on tape since we got it now and it is becoming a big issue.
The backup fails after anything between 50Gb to 150Gb of data copied to tape.
The backup fails with "BackupSrvFileToTape failed Tape fatal error. The device is not connected. Tape write error Tape fatal error."

I logged a case with Veeam support, that indicates that the issue is hardware.
That seems to be correct, as we find the following error in the Windows Event at roughly the same time we got the problem
"Event ID 15, source hplto: The device, \Device\Tape0, is not ready for access yet"

I logged a case with HP, and they replaced the tape library, the tape drive, gave me a new set of tapes and also check for firmware and drivers.
However the problem still persists.

I am now in the middle with the problem, with both HP and Veeam telling the problem is software or hardware respectively.

Has anyone got a similar issue with HP LTO6? Or had one and it got solve?
If it is working for you, would you mind letting me know which Drivers, etc you are using?

Any help would be greatly appreciated as we do not have any backup on tape since we switched to Veeam and the new hardware and it is an uncomfortable situation to sit on for too long.

Thanks
20100
Enthusiast
 
Posts: 69
Liked: 5 times
Joined: Thu Feb 16, 2012 9:23 pm
Full Name: qq

Re: HP LTO6 tape issues

Veeam Logoby martynuk » Mon Mar 02, 2015 11:24 am

We use Veeam 8 with an HP MSL G3 series tape library and HP LTO Ultrium-6 drives. Never had a problem like this. I actually upgraded the Windows HP drivers for the tape drives to v1.0.7.2 just last week, and still working fine. Our B&R server is Windows 2008 R2.

Let me know if I can provide any further info on our setup which you think might help you.

Martin
martynuk
Enthusiast
 
Posts: 48
Liked: 9 times
Joined: Fri Aug 16, 2013 1:34 pm
Full Name: Martin Etheridge

Re: HP LTO6 tape issues

Veeam Logoby v.Eremin » Mon Mar 02, 2015 11:52 am

Can you post you case number here, so that I can ask QA team to take a look at it?
v.Eremin
Veeam Software
 
Posts: 13266
Liked: 968 times
Joined: Fri Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin

Re: HP LTO6 tape issues

Veeam Logoby Isaksson » Mon Mar 02, 2015 1:02 pm

Server 2012 and HP MSL G3 LTO Ultrum-6 I am still using the old 1.0.7.1 driver and i have not seen any issues like that.
Isaksson
Influencer
 
Posts: 15
Liked: 1 time
Joined: Wed Jan 29, 2014 9:24 am
Full Name: Kristoffer Isaksson

Re: HP LTO6 tape issues

Veeam Logoby kmbelt » Mon Mar 02, 2015 2:49 pm

I am having this exact same issue, only i have an LTO5 drive in my G2 Autoloader. I called and was asked if i was using New tapes or tapes that have been previously used on another system. I'm using tapes that have been previously used. I was then asked to do a Full Erase on them, which is actually running right now.

I have upgraded the tape library and drive to the latest firmware, which fixed nothing.
Server is a fresh 2012 R2 that is fully patched.
I know the tape library is good as we used it Thursday night with NetBackup without issue and then reloaded the server for Veeam.

I'm leaning towards a software issue. at this point, because i have my master server running 2012R2 with a MSL4048 that works fine.

Autoloader driver is HP dated 7/26/2013 and is driver version 3.0.0.3
LTO5 drive is using HP driver dated 7/8/2013 and is version 1.0.7.2

I'd love a resolution to this issue also.
kmbelt
Influencer
 
Posts: 10
Liked: never
Joined: Mon Sep 08, 2014 4:08 pm
Full Name: Kevin Belt

Re: HP LTO6 tape issues

Veeam Logoby 20100 » Mon Mar 02, 2015 8:55 pm

Hi

Our case number is 00816784

I have exactly the same drivers as kmbelt (same for LTO5 and LTO6)

Firmware for autoloader: 4.30
Firmware for LTI6: 352W.MSL G3

The HP engineer is coming back today to swap the SAS cable. Will see if that helps...

Regards
20100
Enthusiast
 
Posts: 69
Liked: 5 times
Joined: Thu Feb 16, 2012 9:23 pm
Full Name: qq

Re: HP LTO6 tape issues

Veeam Logoby kmbelt » Mon Mar 02, 2015 9:23 pm

I have been working on my system all day. I do not believe the issue on my end is HW related as it all worked last week Thursday with NetBackup. Friday i took the backup infrastructure from Server 2008R2 to 2012R2 on the same machine. Everything else stayed the same except for the firmware and drivers.

It has been hit and miss today. One time a job will run, the next time it will not. I have created all disk jobs, media pools, tape jobs from scratch and am getting the same results really.

If i come up with somethign new, i'll be sure to post. I also have a ticket open with Veeam. #00820520. So far, all i've gotten is "have you tried a different tape"...

Edit:

I just thought to compare the library config of my 1/8 Autoloader with my Veeam fully functional MSL4048 I have also. I noticed one difference in the autoloader mode. I made my 1/8 match the settings of my 4048 in that it now is set to Random instead of Automatic (Default). We shall see if this makes a difference. I'll post my outcome.
kmbelt
Influencer
 
Posts: 10
Liked: never
Joined: Mon Sep 08, 2014 4:08 pm
Full Name: Kevin Belt

Re: HP LTO6 tape issues

Veeam Logoby 20100 » Mon Mar 02, 2015 11:13 pm

The SAS cable has been replaced and it did not make any difference.
HP does not believe there is a hardware issue. They asked me to try another backup software!
Windows Backup to tape does no longer exists since 2008.

Does anyone knows a quick software that can handle library?
Like kmbelt, I want to establish once of all if it is Veeam, Microsoft or HP

At the end, I am stuck with both of 3 and I really need a working solution.
20100
Enthusiast
 
Posts: 69
Liked: 5 times
Joined: Thu Feb 16, 2012 9:23 pm
Full Name: qq

Re: HP LTO6 tape issues

Veeam Logoby seadave » Tue Mar 03, 2015 12:00 am

I've been in a similar boat with an older Dell TL2000. Worked fine under CommVault Simpana, nothing but problems now that it is connected to Veeam. One thing that made the situation slightly better was to increase the timeout period that Veeam uses. Default is 60s. Try this reg entry:

HKLM\SOFTWARE\Veeam\Veeam Backup and Replication:
REG_DWORD
TapeDeviceNotReadyTimeoutSec
Hex=78 or Dec=120

Maybe try increasing it to 180, not sure if that works. To test your library, you might consider standing up a Unitrends VM. I think that will address your tape library as long as you can address it from the VM (would need to be over IP).

http://www.unitrends.com/products/software/download

I think Veeam may come to realize that relying on native Windows drivers is a crap shoot. They need to pick a pool of supported libraries/tapes and write native drivers for them. That is the only way they can maintain quality and consistency.
seadave
Enthusiast
 
Posts: 35
Liked: 6 times
Joined: Thu Oct 30, 2014 12:43 am

Re: HP LTO6 tape issues

Veeam Logoby tsightler » Tue Mar 03, 2015 1:59 am

20100 wrote:The SAS cable has been replaced and it did not make any difference.
HP does not believe there is a hardware issue. They asked me to try another backup software!
Windows Backup to tape does no longer exists since 2008.

Does anyone knows a quick software that can handle library?
Like kmbelt, I want to establish once of all if it is Veeam, Microsoft or HP

At the end, I am stuck with both of 3 and I really need a working solution.


No matter what the problem could be in either the HP provided driver or Veeam, so using another software might not really narrow things down that much as much software on the market drives the tape hardware directly. Using another package might eliminate hardware, but that seems unlikely to be the issue. To me the most likely scenario is that this is somehow related to the HP driver. Looking at the logs on your case it's hard to see how it could be anything else as the error seems very clear. I'm wondering if it could be some type of strange interaction between the SAS connection and the HP failover support in the driver. We've tested the HP hardware pretty well in the HP lab and I've not seen any such issues, however, these all used FC connectivity. What kind of SAS controller are you using to connect to the library?
tsightler
Veeam Software
 
Posts: 4768
Liked: 1737 times
Joined: Fri Jun 05, 2009 12:57 pm
Full Name: Tom Sightler

Re: HP LTO6 tape issues

Veeam Logoby tsightler » Tue Mar 03, 2015 2:09 am

kmbelt wrote:If i come up with somethign new, i'll be sure to post. I also have a ticket open with Veeam. #00820520. So far, all i've gotten is "have you tried a different tape"...

Looking at your case notes I see that your tapes have labels that imply they are LTO-4 media. Are you actually using LTO-4 media or just old labels? The HP 6250 LTO-6 drive supports reading LTO-4 tapes, but not writing to them. It only supports writing to LTO-5 and LTO-6. I thought it was worth asking.
tsightler
Veeam Software
 
Posts: 4768
Liked: 1737 times
Joined: Fri Jun 05, 2009 12:57 pm
Full Name: Tom Sightler

Re: HP LTO6 tape issues

Veeam Logoby 20100 » Tue Mar 03, 2015 2:11 am

Hi
Sorry I am not an hardware specialist, so not 100% sure which one is the SAS.
Looking at the device manager, storage controllers we have:
Emulex AK763B/AH403A, PCI SLot 1, Storport Miniport Driver
Emulex LighPulse HBA Storport Miniport Driver
Microsoft ICSCI Target LocalMount Adapter
Microsoft Multi-Path Bus Driver
Microsoft Storage Spaces Controller
Smart Array P420i Controller
Smart Array P822 Controller

I believe the OS drives are on SMart Array controller and the data disks on the other one.
We also have Fiber connectivity to an HP 3PAR

Also, I tried the TapeDeviceNotReadyTimeoutSec set to 180 as mentioned by Seadave, but it did not make a difference. This time it failed after 185Gb

Thanks
20100
Enthusiast
 
Posts: 69
Liked: 5 times
Joined: Thu Feb 16, 2012 9:23 pm
Full Name: qq

Re: HP LTO6 tape issues

Veeam Logoby 20100 » Tue Mar 03, 2015 2:17 am

about the LTO labels and Media: no the tapes are really LTO6 and also the labels are especially for LTO6

If I run the HP Library and Tools utility, the inventory shows LTO6.

I noticed that Veeam detected the tapes as 2.4Tb instead of 6.5T, but you told me it was normal:
tape-f29/lto6-capacity-t26331.html
20100
Enthusiast
 
Posts: 69
Liked: 5 times
Joined: Thu Feb 16, 2012 9:23 pm
Full Name: qq

Re: HP LTO6 tape issues

Veeam Logoby tsightler » Tue Mar 03, 2015 2:26 am

20100 wrote:Sorry I am not an hardware specialist, so not 100% sure which one is the SAS.

No problem, we'll work through it. I always like to start out by fully defining the configuration. It should be very easy to figure out via device manager if you select "View...Devices by Connection", you can then drill down through the tree until you find the controllers and then open the controllers and see what specific devices are attached to it.
tsightler
Veeam Software
 
Posts: 4768
Liked: 1737 times
Joined: Fri Jun 05, 2009 12:57 pm
Full Name: Tom Sightler

Re: HP LTO6 tape issues

Veeam Logoby 20100 » Tue Mar 03, 2015 2:42 am

Sorry Tom, I cannot find much. My lack of knowledge is showing up!
When I select the Device by Connection, for Storage I can only see:
Microsoft iSCSI Target LocalMount Adapter
Microsoft Multi-Path Bus Driver
Microsoft Storage Spaces controller

When I open each of them, I get nothing but the standard, General, Driver, Details,Events!
20100
Enthusiast
 
Posts: 69
Liked: 5 times
Joined: Thu Feb 16, 2012 9:23 pm
Full Name: qq

Next

Return to Tape



Who is online

Users browsing this forum: No registered users and 5 guests