Loading Tape Failure HP MSL2024 LTO7 SAS

[ratmo]

I've been getting this error with a brand-new MSL 2024 with 2 LTO7 sas drives

Code: Select all

12/14/2016 10:05:23 AM :: [Servername] [TapeService] Failed to LoadTapeToDrive
Loading tape AFI254L7 from Slot 18 to Drive 1 (Server: SIM-VEEAM-NT05, Library: HP MSL G3 Series 6.40, Drive ID: Tape0) Error: Failed to call RPC function 'LoadMedium': The operation completed successfully. DeviceIoControl(ChangerGetElementStatus) with sourceAddress = '17', destAddress = '0', symLink = '\\?\scsi#changer&ven_hp&prod_msl_g3_series#5&6120987&0&000201#{53f56310-b6bf-11d0-94f2-00a0c91efb8b}'. 0:02:07
Failed to call RPC function 'LoadMedium': The operation completed successfully. DeviceIoControl(ChangerGetElementStatus) with sourceAddress = '17', destAddress = '0', symLink = '\\?\scsi#changer&ven_hp&prod_msl_g3_series#5&6120987&0&000201#{53f56310-b6bf-11d0-94f2-00a0c91efb8b}'.
The operation completed successfully.  

I have upgraded all the Drivers and firmware on the server and tape library
Veeam version is 9.5
Windows 2016 Standard

I have also tested moving tapes on the MSL which works fine
I have tried changing to Native SCSI commands and the same error occurs
I have disable Tape Drive 1 in Veeam to force it to use Drive 2 and same error occurs

[Dima P.]

Hi Ryan,

You see this error when starting a tape job, right? Have you opened a support case?

[ratmo]

Correct when starting the job and i have opened a case, just though there might be some insights here from the community as well

[ratmo]

case number is 02008483

[srw_white]

I am seeing a similar issue on an IBM 3573:

Code: Select all

19/12/2016 10:56:29 :: [server.domain] [TapeService] Failed to LoadTapeToDrive
Failed to call RPC function 'LoadMedium': The operation completed successfully. DeviceIoControl(ChangerGetElementStatus) with sourceAddress = '7', destAddress = '1', symLink = '\\?\scsi#changer&ven_ibm&prod_3573-tl#6&1b148466&0&000201#{53f56310-b6bf-11d0-94f2-00a0c91efb8b}'.
The operation completed successfully.  

Case Ref: 02013552

If I move the tape into the drive using the IBM Tape Utility then the backup to tape starts writing but has an issue when it needs to unload/load the next tape as I only have 1 drive in the library.

Veeam 9.5
Windows 2008 R2

[srw_white]

After a lot more testing, I'd like to correct/clarify what is happening with my issue.

With no tape in the drive, Veeam is showing the drive as "Drive 2" on device tape0. Starting a backup or trying to erase/inventory a tape in this state results in the error I quoted.
If there is a tape in the drive (moved there using the Tape Web Manager or IBM Tape Util installed on the tape server) then the drive shows in Veeam as "Drive 1" on device tape0 and the backup works, it unloads the tape manually inserted and loads the tape it wants and if it needs to change a tape during the backup it can. So as long as a tape is in the drive it's working fine.

[stuartmacgreen]

After a lot more testing, I'd like to correct/clarify what is happening with my issue.

With no tape in the drive, Veeam is showing the drive as "Drive 2" on device tape0. Starting a backup or trying to erase/inventory a tape in this state results in the error I quoted.
If there is a tape in the drive (moved there using the Tape Web Manager or IBM Tape Util installed on the tape server) then the drive shows in Veeam as "Drive 1" on device tape0 and the backup works, it unloads the tape manually inserted and loads the tape it wants and if it needs to change a tape during the backup it can. So as long as a tape is in the drive it's working fine.

Further to this. In Veeam Tape Infrastructure Drive view, when a tape is in the drive (State = Loaded) it has the following:

Drive 1 (Drive ID: Tape0)

issue Eject on that drive (via right click)... the tape is ejected back to slot in the library.

However, just moments later the Veeam Drive view changes to

Drive 2 (Drive ID: Tape0)

Thereafter any scheduled job that wishes to perform a tape load/mount operation has problems and reports the LoadMedium error spew above.

This is a IBM TS3100 with a single LTO-4 HH drive. With 23 slots.
The tape Server has the non-exclusive drivers installed runs Windows Server 2008 R2 64bits.
Device Manager has the IBM TotalStorage Medium Changer listed correctly as well as the Tape Drive device.
The Tape0 device is persistent and never changes.

Anyone offer any troubleshooting into this. Support have said to look at connectivity of the library. It working ok from the ITDT package of tools.


UPDATE: recap, I have 2 Tape Libraries/Changers in my Veeam Tape Infrastructure

Deep dive into the SQL VeeamBackup Database table
Interesting entries revealed....

The table [VeeamBackup].[dbo].[Tape.changers] shows

MSL drive_count = 3
MSL first_drive_number = 1
TS3100 drive_count = 2 #very strange ?
TS3100 first_drive_number = 256 #stranger?

This does not seem quite right?

[Dima P.]

Stuart,

Can you share your case ID please?

TS3100 drive_count = 2 #very strange ?
TS3100 first_drive_number = 256 #stranger?

It’s not the root cause. Drive naming inside the tape library might be different from name in the UI

[stuartmacgreen]

Can you share your case ID please?

Same as @srw_white
Case Ref: 02013552

[stuartmacgreen]

SOLVED Case ID: 02013552

The TS3100 in Veeam was seeing 1 drive at any one time. But a different drive number whether it had a tape media loaded or not. The Tape ID - tape0 - stayed the same throughout.

Further investigation into the library via its management GUI and a Dump Trace file showed a 'ghost' tape drive device existing in its configuration.
The TS3100 did have the attention LED status lit on the front and via the web GUI showing library Ready but with a warning triangle.
A reset to factory defaults of the TS3100 cleared the ghost anomaly and attention LED status, and par for course full firmware updates of library and drives within.

Veeam now sees the 1 drive and never waivers to a different drive number which caused all the issues.

However the signs were there from my previous post..
I rechecked the SQL database table [VeeamBackup].[dbo].[Tape.changers] for the TS3100 and the drive_count now equals 1 not 2.

[profp62]

Hi

I have a maybe similar problem. HP 1x8 G2 AUTOLDR 4.60.

Inventory of some media is without problem, but some have error like this :

Loading tape Tape 8 from Slot 4 to Drive 1 (Server: xxxxxx, Library: HP 1x8 G2 AUTOLDR 4.60, Drive ID: Tape0) Error: Failed to call RPC function 'LoadMedium': The operation completed successfully. DeviceIoControl(ChangerGetElementStatus) with sourceAddress = '3', destAddress = '0', symLink = '\\?\scsi#changer&ven_hp&prod_1x8_g2_autoldr#5&1db922b&0&000101#{53f56310-b6bf-11d0-94f2-00a0c91efb8b}'.
Error Failed to inventory tape Tape 8 Error: Failed to call RPC function 'LoadMedium': The operation completed successfully. DeviceIoControl(ChangerGetElementStatus) with sourceAddress = '3', destAddress = '0', symLink = '\\?\scsi#changer&ven_hp&prod_1x8_g2_autoldr#5&1db922b&0&000101#{53f56310-b6bf-11d0-94f2-00a0c91efb8b}'.
Error Completed with error at 4. dubna 2017 10:23:37

5 LTO6 in autoloader, 3 without problem, 2 error. Media are OK, If i change the media position, inventory ok. (problem with slot ?!)

Drive count is correct - 1 and no "ghost" drives.

Backup on the "good" media without problem.

Note - on initial discovery, there was physical problem with SAS cable and in the middle of the job the autoloader went offline.

Every next inventory is with problems on some medias.

My idea and question - is it possible to manually delete media from dbo.Tape.tape_mediums and rediscovery ?

No veeam case.

Thanks

F.

[Dima P.]

Hi profp62,

My idea and question - is it possible to manually delete media from dbo.Tape.tape_mediums and rediscovery ?

Unfortunately, we cant advise on any database modifications because it might corrupt you existing installation. Kindly, check with support team regarding errors you got – it’s the best option to get a proper resolution.