After the 9.5 u4a update I can't control my tape unit

[hwilliam]

After installing the 9.5 u4a update, which was suggested to me because of another small issue, Veeam fails when trying to eject a tape. A job will complete successfully if I don't eject the tape after the job completes. Also a rescan of the library never completes, the status just changes from in progress to idle.

A case has been opened with support but I'm not getting much help. The prior version of the software did not have this problem.

Physical server with tape library directly connected with mini sas cables. (2 drives)

[Dima P.]

Hello and welcome to the community hwilliam.

Can you please share your case ID? Thank you!

[hwilliam]

Case ID#
03517202

I had 1 server that would fail the backup and the retry would be successful. I was instructed to upgrade to the latest version, which I did. Now it's way worse.

[hwilliam]

A new case ID# was created for this new problem.

03538320

[aaron@ARB]

I have the same problem

Veeam Support - Case # 03527047 [ ref:_00D30RWR._5000e1ZJFSH:ref ]

30/04/2019 7:44:02 PM :: Error: Failed to call RPC function 'UnloadMediumByScsi': The operation completed successfully. Cannot get status of ChangerSlot #1571729563.

I suspect the issue is that the changer slot information is screwed up as the changer (Dell TL2000) only has slots 1-23

[hwilliam]

yeah I'm seeing the same error.
4/30/2019 3:55:13 PM Error: Failed to call RPC function 'UnloadMediumByScsi': The operation completed successfully. Cannot get status of ChangerSlot #282035342.

I also have a Dell TL2000

Isn't that funny!

[aaron@ARB]

I dont know if funny is what I would use to describe it, at the moment I have a few other choice words that you could use instead

Hopefully it is going to be fixed soon. I'm going to have to go out today and purchase some external HDD's to offload the VBK's and VIB's to something

[Dima P.]

Hello folks,

Thanks for sharing your support cases. I've asked QA folks to review the details. Stay tuned.

[aaron@ARB]

Thanks Dmitry, that would be appreciated, we are dead in the water here at the moment. I can confirm that the library and drives work fine in 3a which I downloaded yesterday, the slots are reporting correctly.

The issue is that when you unload a tape you get

29/04/2019 9:51:17 AM Error Failed to eject tape CJH022L6 Error: Failed to call RPC function 'UnloadMediumByScsi': The operation completed successfully. Cannot get status of ChangerSlot #1571729563.

29/04/2019 9:49:38 AM Error Unloading tape CJH022L6 from Drive 1 (Server: ARBBACKUP, Library: IBM 3573-TL C.30, Drive ID: Tape1) to Slot 1571729564 Error: Failed to call RPC function 'UnloadMedium': The indicated element does not exist. DeviceIoControl(ChangerGetElementStatus) with eAddress = '45211' and number = '1', symLink = '\\.\Changer0'.

There is no changer slot #1571729563 obviously.. whereas on version 3a

2/05/2019 12:43:42 PM Unloading tape CJH017L6 from Drive 1 (Server: KIL096, Library: IBM 3573-TL F.11, Drive ID: Tape1) to Slot 2

[aaron@ARB]

Speaking with the local tech support in sydney who found similar from another case, it looks like the upgrade to 4a has mangled the 'original_slot' in the database

Code: Select all

barcode		original_slot
CJH050L6	1571729563
CJH040L6	1571729563
CJH048L6	1571729563
CJH167L6	1571729563
CJH025L6	1571729563
CJH047L6	1571729563
CJH017L6	1571729563
CJH024L6	1571729563
CJH049L6	1571729563
CJH046L6	1571729563
CJH023L6	1571729563
CJH051L6	1571729563
CJG985L6	1571729563
CJH053L6	1571729563
000401L6	22
CJH104L6	22
CJI432L6	22
CJI449L6	22

The 'fix' / 'workaround' is to offline the tapes (remove them from the changer), delete them from the catalog, re-install them into the changer and re-inventory the tapes.

when I get access to the location where my equipment is later today I will test this and report back.

[Dima P.]

Thanks Aaron, unfortunately I do not have any news to share as it's still being investigated.

[aaron@ARB]

Speaking with the local tech support in sydney who found similar from another case, it looks like the upgrade to 4a has mangled the 'original_slot' in the database

Code: Select all

barcode		original_slot
CJH050L6	1571729563
CJH040L6	1571729563
CJH048L6	1571729563
CJH167L6	1571729563
CJH025L6	1571729563
CJH047L6	1571729563
CJH017L6	1571729563
CJH024L6	1571729563
CJH049L6	1571729563
CJH046L6	1571729563
CJH023L6	1571729563
CJH051L6	1571729563
CJG985L6	1571729563
CJH053L6	1571729563
000401L6	22
CJH104L6	22
CJI432L6	22
CJI449L6	22

The 'fix' / 'workaround' is to offline the tapes (remove them from the changer), delete them from the catalog, re-install them into the changer and re-inventory the tapes.

when I get access to the location where my equipment is later today I will test this and report back.

Just coming back to confirm that this appears to be working, backups are running again.

[hongt2k]

I have this problem too, support is still looking at this case. The first time I open this case, support told me it was because of bad tape. I keep saying that this happen after 4a update.

[aaron@ARB]

Refer support to my case number (further up in the thread) and if its the same issue, its a really easy fix.

[DFEX]

Hi there,


I had a similar issue after the update.
I unplugged the power of the tapedrive - plugged in again - did a rescan inside devicemanager - waited a few minutes in the VEEAM B&R-console until it reappeard.
The next backup2tape-job run (nearly) successfully again. Realy, because there is a warning that the statistics have been lost.
Since the steps above everything works fine again.
Hope this helps someone.


Cheers


Didi

[rayhenry88]

I am getting the same error as well, I also have a Dell TL2000. I have a support case opened and a call scheduled with a Veeam tech today at 11AM.

Unloading tape 000038L6 from Drive 1 (Server: Local Tape Server, Library: IBM 3573-TL C.30, Drive ID: Tape1) to Slot 498677149 Error: Failed to call RPC function 'UnloadMedium': The indicated element does not exist. DeviceIoControl(ChangerGetElementStatus) with eAddress = '13724' and number = '1', symLink = '\\.\Changer0'.
Full backup: Failed to continue tape backup session on the next tape medium [GAVEEAM.iccur.com] [TapeService] Failed to UnloadTapeFromDrive Failed to call RPC function 'UnloadMedium': The indicated element does not exist. DeviceIoControl(ChangerGetElementStatus) with eAddress = '13724' and number = '1', symLink = '\\.\Changer0'. The indicated element does not exist.

I thought I was alone in this but it seems to be a consistent issue after upgrading to 4a.

[goldstar4]

I have a similar issue with VEEAM 9.5 update 4 and have my own case number 03537269. The server is a Dell PE r530 connected via SAS to an Overland NEOs T24 with an IBM drive, running the latest firmware that Overland has posted: 4/24/2019 6:57 AM 9471344 LTO6_JAX1.sas_hh.fmrz
The tape library is on the latest firmware.
The server uses the Windows generic LTO Tape drive driver (6/21/2006). I have tried the specific driver with this setup but I still have issues.

Since the update I have had issues with this being stable. A backup to tape would work, the next one wouldn't finish.
Greg

[rayhenry88]

The 'fix' / 'workaround' is to offline the tapes (remove them from the changer), delete them from the catalog, re-install them into the changer and re-inventory the tapes.

Thanks aaron@ARB, this fixed my backups. I've been beating my head against the wall on this one for the last week.

[Moebius]

I too had some tape issues immediately after upgrading to v9.5 u4a from 9.5 u2.
I'm using GFS tape backups that run on the weekends. The job runs on the first weekend produced errors never seen before (from being unable to unload/eject tapes, to a "Transmission pipeline hanged" error. (Case ID 03525846). Don't know if this is actually related to this thread.

Without really changing anything, the next two weekends ran without errors. I have no clues about what it was. I'm still waiting to see if this can classify as a one-time, upgrade-related issue that fixed itself

[dariusz.tyka]

We are having the same problem with our Dell TL2000 tape library. Case number: 03549800
Tape library: Library name
Used tapes: 000013L6 [000013L6]
Unloading tape 000013L6 from Drive 1 (Server: server name, Library: library name, Drive ID: Tape0) to Slot 210697636 Error: Failed to call RPC function 'UnloadMedium': The indicated element does not exist. DeviceIoControl(ChangerGetElementStatus) with eAddress = '64931' and number = '1', symLink = '\\.\Changer0'.
Failed to eject tape 000013L6 Details: Failed to call RPC function 'UnloadMedium': The indicated element does not exist. DeviceIoControl(ChangerGetElementStatus) with eAddress = '64931' and number = '1', symLink = '\\.\Changer0'.

[goldstar4]

Follow up: I tried the following
The 'fix' / 'workaround' is to offline the tapes (remove them from the changer), delete them from the catalog, re-install them into the changer and re-inventory the tapes.

Now I can't inventory the library:
5/8/2019 7:58:49 AM Error Failed to inventory tape EKZ888L6 Error: Timed out waiting for tape drive lock

[dariusz.tyka]

I tried this fix so I stopped Veeam tape Access service to have my tape library offline. Then I removed 3 affected tapes from catalog, reenabled Veeam tape service and those tapes were rediscovered as unknown. I move them to free media pool. One of those tapes were in TL2000 drive so I tried to unload it to slot but it ended with following error:
5/8/2019 2:54:23 PM Error Unloading tape 000013L6 from Drive 1 (Server: name, Library: name, Drive ID: Tape0) to Slot 234 Error: Failed to call RPC function 'UnloadMedium': The indicated element does not exist. DeviceIoControl(ChangerGetElementStatus) with eAddress = '233' and number = '1', symLink = '\\.\Changer0'.
So the fix is not working for me unfortunately.

* an update - some features are working OK. Like tape erase - Veeam can grab tape from slot, erase it and unload from the drive. But starting a B2T job fails with an error.

[Dima P.]

Hello folks,

We've identify the issue and now working on a private fix (it's going to be available via support team once we confirm that it is stable). If you confirm the same behavior in your environment please raise a support case on the first place (support team must confirm that your Veeam B&R is affected by the same issue before proposing the workaround). Cheers!

[orlando_rubino]

I have the same error, where can I download the fix? Thank you

[aaron@ARB]

I tried this fix so I stopped Veeam tape Access service to have my tape library offline. Then I removed 3 affected tapes from catalog, reenabled Veeam tape service and those tapes were rediscovered as unknown. I move them to free media pool. One of those tapes were in TL2000 drive so I tried to unload it to slot but it ended with following error:
5/8/2019 2:54:23 PM Error Unloading tape 000013L6 from Drive 1 (Server: name, Library: name, Drive ID: Tape0) to Slot 234 Error: Failed to call RPC function 'UnloadMedium': The indicated element does not exist. DeviceIoControl(ChangerGetElementStatus) with eAddress = '233' and number = '1', symLink = '\\.\Changer0'.
So the fix is not working for me unfortunately.

* an update - some features are working OK. Like tape erase - Veeam can grab tape from slot, erase it and unload from the drive. But starting a B2T job fails with an error.

Sounds like veeam may not have deleted the DB records properly as its perhaps picking up the label / barcode / tape ID or what not as the issue is you are still getting "The indicated element does not exist." so I reckon if you were to check the tape_mediums table in your veeam database and see which slot it thinks it belongs in

something like

Code: Select all

SELECT name,barcode,original_slot FROM [VeeamBackup].[dbo].[Tape.tape_mediums] where barcode = 'CJH019L6'

Just change your barcode to whatever your tape is called (or name if you are not using barcodes) and see where the 'original_slot' is

Code: Select all

name		barcode		original_slot
CJH019L6	CJH019L6	13

for an example, you will find a slot that you dont actually have. Erasing and such like would work as once the tape is IN the drive it will work fine, its just the tape operations that rely on its slot (unloading for example)

[dariusz.tyka]

Hi aaron@ARB,

tested it and got results that 2 affected tapes are in the same slot 210697635. So it looks like still wrong slot numbers are present in Veeam database.

[dariusz.tyka]

Ok it worked finally. But fix requires to physically remove and reinsert tapes to library. Simple veeam tape service off/remove from catalog/service on was not enough. Awaiting for fix from Veeam R&D.

[Avi Milbaum]

I have the same issue with HP MSL2024 G3 Series 7.00

Unloading tape 000193L4 from Drive 1 (Server: PACKER-DP.packer.steels, Library: HP MSL G3 Series 7.00, Drive ID: Tape1) to Slot -1507724375 Error: Failed to call RPC function 'UnloadMedium': The indicated element does not exist. DeviceIoControl(ChangerGetElementStatus) with eAddress = '65535' and number = '1', symLink = '\\.\Changer0'.
Failed to eject tape 000193L4 Details: Failed to call RPC function 'UnloadMedium': The indicated element does not exist. DeviceIoControl(ChangerGetElementStatus) with eAddress = '65535' and number = '1', symLink = '\\.\Changer0'.

Case # 03556160

Is there an idea to bypass this issue. until fix will be available ?

Thanks
Avi Milbaum

[Dima P.]

Hello Avi,

Have your tried the workaround proposed in this thread? Thanks!

[Gostev]

While there could be some exceptions, in vast majority of support cases following the Update 4a release, the issue was caused by not having the latest updates installed on Windows Server 2012 R2. The actual bug sits in JIT (Just In Time) compiler logic of .NET Framework 4.6. This bug is also the critical vulnerability > Microsoft Security Bulletin MS15-092