Loading Tape Failure HP MSL2024 LTO7 SAS

Everything about backing up to tape

Loading Tape Failure HP MSL2024 LTO7 SAS

Veeam Logoby ratmo » Wed Dec 14, 2016 8:29 am

I've been getting this error with a brand-new MSL 2024 with 2 LTO7 sas drives

Code: Select all
12/14/2016 10:05:23 AM :: [Servername] [TapeService] Failed to LoadTapeToDrive
Loading tape AFI254L7 from Slot 18 to Drive 1 (Server: SIM-VEEAM-NT05, Library: HP MSL G3 Series 6.40, Drive ID: Tape0) Error: Failed to call RPC function 'LoadMedium': The operation completed successfully. DeviceIoControl(ChangerGetElementStatus) with sourceAddress = '17', destAddress = '0', symLink = '\\?\scsi#changer&ven_hp&prod_msl_g3_series#5&6120987&0&000201#{53f56310-b6bf-11d0-94f2-00a0c91efb8b}'. 0:02:07
Failed to call RPC function 'LoadMedium': The operation completed successfully. DeviceIoControl(ChangerGetElementStatus) with sourceAddress = '17', destAddress = '0', symLink = '\\?\scsi#changer&ven_hp&prod_msl_g3_series#5&6120987&0&000201#{53f56310-b6bf-11d0-94f2-00a0c91efb8b}'.
The operation completed successfully. 


I have upgraded all the Drivers and firmware on the server and tape library
Veeam version is 9.5
Windows 2016 Standard

I have also tested moving tapes on the MSL which works fine
I have tried changing to Native SCSI commands and the same error occurs
I have disable Tape Drive 1 in Veeam to force it to use Drive 2 and same error occurs
ratmo
Novice
 
Posts: 7
Liked: never
Joined: Wed Aug 27, 2014 11:28 am
Full Name: Ryan Rutledge

Re: Loading Tape Failure HP MSL2024 LTO7 SAS

Veeam Logoby Dima P. » Wed Dec 14, 2016 9:06 am

Hi Ryan,

You see this error when starting a tape job, right? Have you opened a support case?
Dima P.
Veeam Software
 
Posts: 6517
Liked: 454 times
Joined: Mon Feb 04, 2013 2:07 pm
Location: SPb
Full Name: Dmitry Popov

Re: Loading Tape Failure HP MSL2024 LTO7 SAS

Veeam Logoby ratmo » Wed Dec 14, 2016 2:00 pm

Correct when starting the job and i have opened a case, just though there might be some insights here from the community as well
ratmo
Novice
 
Posts: 7
Liked: never
Joined: Wed Aug 27, 2014 11:28 am
Full Name: Ryan Rutledge

Re: Loading Tape Failure HP MSL2024 LTO7 SAS

Veeam Logoby ratmo » Wed Dec 14, 2016 2:05 pm

case number is 02008483
ratmo
Novice
 
Posts: 7
Liked: never
Joined: Wed Aug 27, 2014 11:28 am
Full Name: Ryan Rutledge

Re: Loading Tape Failure HP MSL2024 LTO7 SAS

Veeam Logoby srw_white » Mon Dec 19, 2016 11:56 am

I am seeing a similar issue on an IBM 3573:
Code: Select all
19/12/2016 10:56:29 :: [server.domain] [TapeService] Failed to LoadTapeToDrive
Failed to call RPC function 'LoadMedium': The operation completed successfully. DeviceIoControl(ChangerGetElementStatus) with sourceAddress = '7', destAddress = '1', symLink = '\\?\scsi#changer&ven_ibm&prod_3573-tl#6&1b148466&0&000201#{53f56310-b6bf-11d0-94f2-00a0c91efb8b}'.
The operation completed successfully. 


Case Ref: 02013552

If I move the tape into the drive using the IBM Tape Utility then the backup to tape starts writing but has an issue when it needs to unload/load the next tape as I only have 1 drive in the library.

Veeam 9.5
Windows 2008 R2
srw_white
Novice
 
Posts: 3
Liked: 4 times
Joined: Fri Feb 03, 2012 9:24 am
Full Name: Stephen White

Re: Loading Tape Failure HP MSL2024 LTO7 SAS

Veeam Logoby srw_white » Wed Dec 21, 2016 9:54 am

After a lot more testing, I'd like to correct/clarify what is happening with my issue.

With no tape in the drive, Veeam is showing the drive as "Drive 2" on device tape0. Starting a backup or trying to erase/inventory a tape in this state results in the error I quoted.
If there is a tape in the drive (moved there using the Tape Web Manager or IBM Tape Util installed on the tape server) then the drive shows in Veeam as "Drive 1" on device tape0 and the backup works, it unloads the tape manually inserted and loads the tape it wants and if it needs to change a tape during the backup it can. So as long as a tape is in the drive it's working fine.
srw_white
Novice
 
Posts: 3
Liked: 4 times
Joined: Fri Feb 03, 2012 9:24 am
Full Name: Stephen White

Re: Loading Tape Failure HP MSL2024 LTO7 SAS

Veeam Logoby stuartmacgreen » Fri Dec 23, 2016 9:15 pm

srw_white wrote:After a lot more testing, I'd like to correct/clarify what is happening with my issue.

With no tape in the drive, Veeam is showing the drive as "Drive 2" on device tape0. Starting a backup or trying to erase/inventory a tape in this state results in the error I quoted.
If there is a tape in the drive (moved there using the Tape Web Manager or IBM Tape Util installed on the tape server) then the drive shows in Veeam as "Drive 1" on device tape0 and the backup works, it unloads the tape manually inserted and loads the tape it wants and if it needs to change a tape during the backup it can. So as long as a tape is in the drive it's working fine.


Further to this. In Veeam Tape Infrastructure Drive view, when a tape is in the drive (State = Loaded) it has the following:

Drive 1 (Drive ID: Tape0)

issue Eject on that drive (via right click)... the tape is ejected back to slot in the library.

However, just moments later the Veeam Drive view changes to

Drive 2 (Drive ID: Tape0)

Thereafter any scheduled job that wishes to perform a tape load/mount operation has problems and reports the LoadMedium error spew above.

This is a IBM TS3100 with a single LTO-4 HH drive. With 23 slots.
The tape Server has the non-exclusive drivers installed runs Windows Server 2008 R2 64bits.
Device Manager has the IBM TotalStorage Medium Changer listed correctly as well as the Tape Drive device.
The Tape0 device is persistent and never changes.

Anyone offer any troubleshooting into this. Support have said to look at connectivity of the library. It working ok from the ITDT package of tools.


UPDATE: recap, I have 2 Tape Libraries/Changers in my Veeam Tape Infrastructure

Deep dive into the SQL VeeamBackup Database table
Interesting entries revealed....

The table [VeeamBackup].[dbo].[Tape.changers] shows

MSL drive_count = 3
MSL first_drive_number = 1
TS3100 drive_count = 2 #very strange ?
TS3100 first_drive_number = 256 #stranger?

This does not seem quite right?
Backup Infrastructure: Veeam B&R 9.5 U2 • vSphere 5.5 U3e • HP MSA 2040 [FC] (Backup Storage) • HP MSL6480 LTO-6 (Tape Storage)
(Veeam Agent for Windows protecting Veeam Proxy / Tape / Repository servers)
stuartmacgreen
Enthusiast
 
Posts: 88
Liked: 16 times
Joined: Tue May 01, 2012 11:56 am
Full Name: Stuart Green

Re: Loading Tape Failure HP MSL2024 LTO7 SAS

Veeam Logoby Dima P. » Mon Dec 26, 2016 2:10 pm

Stuart,

Can you share your case ID please?
TS3100 drive_count = 2 #very strange ?
TS3100 first_drive_number = 256 #stranger?

It’s not the root cause. Drive naming inside the tape library might be different from name in the UI
Dima P.
Veeam Software
 
Posts: 6517
Liked: 454 times
Joined: Mon Feb 04, 2013 2:07 pm
Location: SPb
Full Name: Dmitry Popov

Re: Loading Tape Failure HP MSL2024 LTO7 SAS

Veeam Logoby stuartmacgreen » Wed Dec 28, 2016 8:23 am

Dima P. wrote:Can you share your case ID please?

Same as @srw_white
Case Ref: 02013552
Backup Infrastructure: Veeam B&R 9.5 U2 • vSphere 5.5 U3e • HP MSA 2040 [FC] (Backup Storage) • HP MSL6480 LTO-6 (Tape Storage)
(Veeam Agent for Windows protecting Veeam Proxy / Tape / Repository servers)
stuartmacgreen
Enthusiast
 
Posts: 88
Liked: 16 times
Joined: Tue May 01, 2012 11:56 am
Full Name: Stuart Green

Re: Loading Tape Failure HP MSL2024 LTO7 SAS

Veeam Logoby stuartmacgreen » Fri Jan 06, 2017 9:12 am 2 people like this post

SOLVED Case ID: 02013552

The TS3100 in Veeam was seeing 1 drive at any one time. But a different drive number whether it had a tape media loaded or not. The Tape ID - tape0 - stayed the same throughout.

Further investigation into the library via its management GUI and a Dump Trace file showed a 'ghost' tape drive device existing in its configuration.
The TS3100 did have the attention LED status lit on the front and via the web GUI showing library Ready but with a warning triangle.
A reset to factory defaults of the TS3100 cleared the ghost anomaly and attention LED status, and par for course full firmware updates of library and drives within.

Veeam now sees the 1 drive and never waivers to a different drive number which caused all the issues.

However the signs were there from my previous post..
I rechecked the SQL database table [VeeamBackup].[dbo].[Tape.changers] for the TS3100 and the drive_count now equals 1 not 2.
Backup Infrastructure: Veeam B&R 9.5 U2 • vSphere 5.5 U3e • HP MSA 2040 [FC] (Backup Storage) • HP MSL6480 LTO-6 (Tape Storage)
(Veeam Agent for Windows protecting Veeam Proxy / Tape / Repository servers)
stuartmacgreen
Enthusiast
 
Posts: 88
Liked: 16 times
Joined: Tue May 01, 2012 11:56 am
Full Name: Stuart Green

Re: Loading Tape Failure HP MSL2024 LTO7 SAS

Veeam Logoby profp62 » Tue Apr 04, 2017 11:26 am

Hi

I have a maybe similar problem. HP 1x8 G2 AUTOLDR 4.60.

Inventory of some media is without problem, but some have error like this :

Loading tape Tape 8 from Slot 4 to Drive 1 (Server: xxxxxx, Library: HP 1x8 G2 AUTOLDR 4.60, Drive ID: Tape0) Error: Failed to call RPC function 'LoadMedium': The operation completed successfully. DeviceIoControl(ChangerGetElementStatus) with sourceAddress = '3', destAddress = '0', symLink = '\\?\scsi#changer&ven_hp&prod_1x8_g2_autoldr#5&1db922b&0&000101#{53f56310-b6bf-11d0-94f2-00a0c91efb8b}'.
Error Failed to inventory tape Tape 8 Error: Failed to call RPC function 'LoadMedium': The operation completed successfully. DeviceIoControl(ChangerGetElementStatus) with sourceAddress = '3', destAddress = '0', symLink = '\\?\scsi#changer&ven_hp&prod_1x8_g2_autoldr#5&1db922b&0&000101#{53f56310-b6bf-11d0-94f2-00a0c91efb8b}'.
Error Completed with error at 4. dubna 2017 10:23:37

5 LTO6 in autoloader, 3 without problem, 2 error. Media are OK, If i change the media position, inventory ok. (problem with slot ?!)

Drive count is correct - 1 and no "ghost" drives.

Backup on the "good" media without problem.

Note - on initial discovery, there was physical problem with SAS cable and in the middle of the job the autoloader went offline.

Every next inventory is with problems on some medias.

My idea and question - is it possible to manually delete media from dbo.Tape.tape_mediums and rediscovery ?

No veeam case.

Thanks

F.
profp62
Lurker
 
Posts: 1
Liked: never
Joined: Tue Apr 04, 2017 8:55 am
Full Name: Frantisek Papousek

Re: Loading Tape Failure HP MSL2024 LTO7 SAS

Veeam Logoby Dima P. » Tue Apr 11, 2017 9:36 pm

Hi profp62,
My idea and question - is it possible to manually delete media from dbo.Tape.tape_mediums and rediscovery ?

Unfortunately, we cant advise on any database modifications because it might corrupt you existing installation. Kindly, check with support team regarding errors you got – it’s the best option to get a proper resolution.
Dima P.
Veeam Software
 
Posts: 6517
Liked: 454 times
Joined: Mon Feb 04, 2013 2:07 pm
Location: SPb
Full Name: Dmitry Popov


Return to Tape



Who is online

Users browsing this forum: No registered users and 8 guests