Discussions specific to tape backups
hwilliam
Novice
Posts: 4
Liked: never
Joined: Apr 30, 2019 4:53 pm
Full Name: Heath Williams
Contact:

After the 9.5 u4a update I can't control my tape unit

Post by hwilliam » Apr 30, 2019 5:04 pm

After installing the 9.5 u4a update, which was suggested to me because of another small issue, Veeam fails when trying to eject a tape. A job will complete successfully if I don't eject the tape after the job completes. Also a rescan of the library never completes, the status just changes from in progress to idle.

A case has been opened with support but I'm not getting much help. The prior version of the software did not have this problem.

Physical server with tape library directly connected with mini sas cables. (2 drives)

Dima P.
Product Manager
Posts: 10709
Liked: 881 times
Joined: Feb 04, 2013 2:07 pm
Full Name: Dmitry Popov
Location: Prague
Contact:

Re: After the 9.5 u4a update I can't control my tape unit

Post by Dima P. » Apr 30, 2019 6:39 pm

Hello and welcome to the community hwilliam.

Can you please share your case ID? Thank you!

hwilliam
Novice
Posts: 4
Liked: never
Joined: Apr 30, 2019 4:53 pm
Full Name: Heath Williams
Contact:

Re: After the 9.5 u4a update I can't control my tape unit

Post by hwilliam » Apr 30, 2019 7:35 pm

Case ID#
03517202

I had 1 server that would fail the backup and the retry would be successful. I was instructed to upgrade to the latest version, which I did. Now it's way worse.

hwilliam
Novice
Posts: 4
Liked: never
Joined: Apr 30, 2019 4:53 pm
Full Name: Heath Williams
Contact:

Re: After the 9.5 u4a update I can't control my tape unit

Post by hwilliam » Apr 30, 2019 9:39 pm

A new case ID# was created for this new problem.

03538320

aaron@ARB
Expert
Posts: 132
Liked: 14 times
Joined: Feb 21, 2014 3:12 am
Full Name: ARBCorporationPtyLtd
Contact:

Re: After the 9.5 u4a update I can't control my tape unit

Post by aaron@ARB » May 01, 2019 5:36 am

I have the same problem

Veeam Support - Case # 03527047 [ ref:_00D30RWR._5000e1ZJFSH:ref ]

30/04/2019 7:44:02 PM :: Error: Failed to call RPC function 'UnloadMediumByScsi': The operation completed successfully. Cannot get status of ChangerSlot #1571729563.

I suspect the issue is that the changer slot information is screwed up as the changer (Dell TL2000) only has slots 1-23

hwilliam
Novice
Posts: 4
Liked: never
Joined: Apr 30, 2019 4:53 pm
Full Name: Heath Williams
Contact:

Re: After the 9.5 u4a update I can't control my tape unit

Post by hwilliam » May 01, 2019 4:47 pm

yeah I'm seeing the same error.
4/30/2019 3:55:13 PM Error: Failed to call RPC function 'UnloadMediumByScsi': The operation completed successfully. Cannot get status of ChangerSlot #282035342.

I also have a Dell TL2000

Isn't that funny!

aaron@ARB
Expert
Posts: 132
Liked: 14 times
Joined: Feb 21, 2014 3:12 am
Full Name: ARBCorporationPtyLtd
Contact:

Re: After the 9.5 u4a update I can't control my tape unit

Post by aaron@ARB » May 01, 2019 10:17 pm

I dont know if funny is what I would use to describe it, at the moment I have a few other choice words that you could use instead :)

Hopefully it is going to be fixed soon. I'm going to have to go out today and purchase some external HDD's to offload the VBK's and VIB's to something

Dima P.
Product Manager
Posts: 10709
Liked: 881 times
Joined: Feb 04, 2013 2:07 pm
Full Name: Dmitry Popov
Location: Prague
Contact:

Re: After the 9.5 u4a update I can't control my tape unit

Post by Dima P. » May 02, 2019 9:16 am

Hello folks,

Thanks for sharing your support cases. I've asked QA folks to review the details. Stay tuned.

aaron@ARB
Expert
Posts: 132
Liked: 14 times
Joined: Feb 21, 2014 3:12 am
Full Name: ARBCorporationPtyLtd
Contact:

Re: After the 9.5 u4a update I can't control my tape unit

Post by aaron@ARB » May 03, 2019 12:32 am

Thanks Dmitry, that would be appreciated, we are dead in the water here at the moment. I can confirm that the library and drives work fine in 3a which I downloaded yesterday, the slots are reporting correctly.

The issue is that when you unload a tape you get
29/04/2019 9:51:17 AM Error Failed to eject tape CJH022L6 Error: Failed to call RPC function 'UnloadMediumByScsi': The operation completed successfully. Cannot get status of ChangerSlot #1571729563.

29/04/2019 9:49:38 AM Error Unloading tape CJH022L6 from Drive 1 (Server: ARBBACKUP, Library: IBM 3573-TL C.30, Drive ID: Tape1) to Slot 1571729564 Error: Failed to call RPC function 'UnloadMedium': The indicated element does not exist. DeviceIoControl(ChangerGetElementStatus) with eAddress = '45211' and number = '1', symLink = '\\.\Changer0'.

There is no changer slot #1571729563 obviously.. whereas on version 3a
2/05/2019 12:43:42 PM Unloading tape CJH017L6 from Drive 1 (Server: KIL096, Library: IBM 3573-TL F.11, Drive ID: Tape1) to Slot 2

aaron@ARB
Expert
Posts: 132
Liked: 14 times
Joined: Feb 21, 2014 3:12 am
Full Name: ARBCorporationPtyLtd
Contact:

Re: After the 9.5 u4a update I can't control my tape unit

Post by aaron@ARB » May 03, 2019 2:52 am 1 person likes this post

Speaking with the local tech support in sydney who found similar from another case, it looks like the upgrade to 4a has mangled the 'original_slot' in the database

Code: Select all

barcode		original_slot
CJH050L6	1571729563
CJH040L6	1571729563
CJH048L6	1571729563
CJH167L6	1571729563
CJH025L6	1571729563
CJH047L6	1571729563
CJH017L6	1571729563
CJH024L6	1571729563
CJH049L6	1571729563
CJH046L6	1571729563
CJH023L6	1571729563
CJH051L6	1571729563
CJG985L6	1571729563
CJH053L6	1571729563
000401L6	22
CJH104L6	22
CJI432L6	22
CJI449L6	22
The 'fix' / 'workaround' is to offline the tapes (remove them from the changer), delete them from the catalog, re-install them into the changer and re-inventory the tapes.

when I get access to the location where my equipment is later today I will test this and report back.

Dima P.
Product Manager
Posts: 10709
Liked: 881 times
Joined: Feb 04, 2013 2:07 pm
Full Name: Dmitry Popov
Location: Prague
Contact:

Re: After the 9.5 u4a update I can't control my tape unit

Post by Dima P. » May 03, 2019 10:16 am

Thanks Aaron, unfortunately I do not have any news to share as it's still being investigated.

aaron@ARB
Expert
Posts: 132
Liked: 14 times
Joined: Feb 21, 2014 3:12 am
Full Name: ARBCorporationPtyLtd
Contact:

Re: After the 9.5 u4a update I can't control my tape unit

Post by aaron@ARB » May 03, 2019 10:55 pm 1 person likes this post

aaron@ARB wrote:
May 03, 2019 2:52 am
Speaking with the local tech support in sydney who found similar from another case, it looks like the upgrade to 4a has mangled the 'original_slot' in the database

Code: Select all

barcode		original_slot
CJH050L6	1571729563
CJH040L6	1571729563
CJH048L6	1571729563
CJH167L6	1571729563
CJH025L6	1571729563
CJH047L6	1571729563
CJH017L6	1571729563
CJH024L6	1571729563
CJH049L6	1571729563
CJH046L6	1571729563
CJH023L6	1571729563
CJH051L6	1571729563
CJG985L6	1571729563
CJH053L6	1571729563
000401L6	22
CJH104L6	22
CJI432L6	22
CJI449L6	22
The 'fix' / 'workaround' is to offline the tapes (remove them from the changer), delete them from the catalog, re-install them into the changer and re-inventory the tapes.

when I get access to the location where my equipment is later today I will test this and report back.
Just coming back to confirm that this appears to be working, backups are running again.

hongt2k
Lurker
Posts: 2
Liked: never
Joined: May 09, 2018 7:35 pm
Full Name: Tony
Contact:

Re: After the 9.5 u4a update I can't control my tape unit

Post by hongt2k » May 06, 2019 12:46 am

I have this problem too, support is still looking at this case. The first time I open this case, support told me it was because of bad tape. I keep saying that this happen after 4a update.

aaron@ARB
Expert
Posts: 132
Liked: 14 times
Joined: Feb 21, 2014 3:12 am
Full Name: ARBCorporationPtyLtd
Contact:

Re: After the 9.5 u4a update I can't control my tape unit

Post by aaron@ARB » May 06, 2019 2:09 am

Refer support to my case number (further up in the thread) and if its the same issue, its a really easy fix.

DFEX
Novice
Posts: 9
Liked: never
Joined: Sep 01, 2014 11:20 am
Contact:

Re: After the 9.5 u4a update I can't control my tape unit

Post by DFEX » May 06, 2019 7:51 am

Hi there,


I had a similar issue after the update.
I unplugged the power of the tapedrive - plugged in again - did a rescan inside devicemanager - waited a few minutes in the VEEAM B&R-console until it reappeard.
The next backup2tape-job run (nearly) successfully again. Realy, because there is a warning that the statistics have been lost.
Since the steps above everything works fine again.
Hope this helps someone.


Cheers


Didi

rayhenry88
Lurker
Posts: 2
Liked: never
Joined: May 14, 2014 2:00 pm
Full Name: Ray Henry
Contact:

Re: After the 9.5 u4a update I can't control my tape unit

Post by rayhenry88 » May 06, 2019 12:46 pm

I am getting the same error as well, I also have a Dell TL2000. I have a support case opened and a call scheduled with a Veeam tech today at 11AM.

Unloading tape 000038L6 from Drive 1 (Server: Local Tape Server, Library: IBM 3573-TL C.30, Drive ID: Tape1) to Slot 498677149 Error: Failed to call RPC function 'UnloadMedium': The indicated element does not exist. DeviceIoControl(ChangerGetElementStatus) with eAddress = '13724' and number = '1', symLink = '\\.\Changer0'.
Full backup: Failed to continue tape backup session on the next tape medium [GAVEEAM.iccur.com] [TapeService] Failed to UnloadTapeFromDrive Failed to call RPC function 'UnloadMedium': The indicated element does not exist. DeviceIoControl(ChangerGetElementStatus) with eAddress = '13724' and number = '1', symLink = '\\.\Changer0'. The indicated element does not exist.

I thought I was alone in this but it seems to be a consistent issue after upgrading to 4a.

goldstar4
Novice
Posts: 5
Liked: never
Joined: Mar 19, 2018 3:48 pm
Full Name: gregory filter
Contact:

Re: After the 9.5 u4a update I can't control my tape unit

Post by goldstar4 » May 06, 2019 7:32 pm

I have a similar issue with VEEAM 9.5 update 4 and have my own case number 03537269. The server is a Dell PE r530 connected via SAS to an Overland NEOs T24 with an IBM drive, running the latest firmware that Overland has posted: 4/24/2019 6:57 AM 9471344 LTO6_JAX1.sas_hh.fmrz
The tape library is on the latest firmware.
The server uses the Windows generic LTO Tape drive driver (6/21/2006). I have tried the specific driver with this setup but I still have issues.

Since the update I have had issues with this being stable. A backup to tape would work, the next one wouldn't finish.
Greg

rayhenry88
Lurker
Posts: 2
Liked: never
Joined: May 14, 2014 2:00 pm
Full Name: Ray Henry
Contact:

Re: After the 9.5 u4a update I can't control my tape unit

Post by rayhenry88 » May 06, 2019 9:13 pm

The 'fix' / 'workaround' is to offline the tapes (remove them from the changer), delete them from the catalog, re-install them into the changer and re-inventory the tapes.

Thanks aaron@ARB, this fixed my backups. I've been beating my head against the wall on this one for the last week.

Moebius
Veeam ProPartner
Posts: 167
Liked: 21 times
Joined: Jun 09, 2009 2:48 pm
Full Name: Lucio Mazzi
Location: Reggio Emilia, Italy
Contact:

Re: After the 9.5 u4a update I can't control my tape unit

Post by Moebius » May 07, 2019 1:05 pm

I too had some tape issues immediately after upgrading to v9.5 u4a from 9.5 u2.
I'm using GFS tape backups that run on the weekends. The job runs on the first weekend produced errors never seen before (from being unable to unload/eject tapes, to a "Transmission pipeline hanged" error. (Case ID 03525846). Don't know if this is actually related to this thread.

Without really changing anything, the next two weekends ran without errors. I have no clues about what it was. I'm still waiting to see if this can classify as a one-time, upgrade-related issue that fixed itself :-)

dariusz.tyka
Influencer
Posts: 18
Liked: 2 times
Joined: Jan 21, 2019 1:38 pm
Full Name: Dariusz Tyka
Contact:

Re: After the 9.5 u4a update I can't control my tape unit

Post by dariusz.tyka » May 08, 2019 9:15 am

We are having the same problem with our Dell TL2000 tape library. Case number: 03549800
Tape library: Library name
Used tapes: 000013L6 [000013L6]
Unloading tape 000013L6 from Drive 1 (Server: server name, Library: library name, Drive ID: Tape0) to Slot 210697636 Error: Failed to call RPC function 'UnloadMedium': The indicated element does not exist. DeviceIoControl(ChangerGetElementStatus) with eAddress = '64931' and number = '1', symLink = '\\.\Changer0'.
Failed to eject tape 000013L6 Details: Failed to call RPC function 'UnloadMedium': The indicated element does not exist. DeviceIoControl(ChangerGetElementStatus) with eAddress = '64931' and number = '1', symLink = '\\.\Changer0'.

goldstar4
Novice
Posts: 5
Liked: never
Joined: Mar 19, 2018 3:48 pm
Full Name: gregory filter
Contact:

Re: After the 9.5 u4a update I can't control my tape unit

Post by goldstar4 » May 08, 2019 12:03 pm

Follow up: I tried the following
The 'fix' / 'workaround' is to offline the tapes (remove them from the changer), delete them from the catalog, re-install them into the changer and re-inventory the tapes.

Now I can't inventory the library:
5/8/2019 7:58:49 AM Error Failed to inventory tape EKZ888L6 Error: Timed out waiting for tape drive lock

dariusz.tyka
Influencer
Posts: 18
Liked: 2 times
Joined: Jan 21, 2019 1:38 pm
Full Name: Dariusz Tyka
Contact:

Re: After the 9.5 u4a update I can't control my tape unit

Post by dariusz.tyka » May 08, 2019 12:59 pm

I tried this fix so I stopped Veeam tape Access service to have my tape library offline. Then I removed 3 affected tapes from catalog, reenabled Veeam tape service and those tapes were rediscovered as unknown. I move them to free media pool. One of those tapes were in TL2000 drive so I tried to unload it to slot but it ended with following error:
5/8/2019 2:54:23 PM Error Unloading tape 000013L6 from Drive 1 (Server: name, Library: name, Drive ID: Tape0) to Slot 234 Error: Failed to call RPC function 'UnloadMedium': The indicated element does not exist. DeviceIoControl(ChangerGetElementStatus) with eAddress = '233' and number = '1', symLink = '\\.\Changer0'.
So the fix is not working for me unfortunately.

* an update - some features are working OK. Like tape erase - Veeam can grab tape from slot, erase it and unload from the drive. But starting a B2T job fails with an error.

Dima P.
Product Manager
Posts: 10709
Liked: 881 times
Joined: Feb 04, 2013 2:07 pm
Full Name: Dmitry Popov
Location: Prague
Contact:

Re: After the 9.5 u4a update I can't control my tape unit

Post by Dima P. » May 09, 2019 2:10 pm

Hello folks,

We've identify the issue and now working on a private fix (it's going to be available via support team once we confirm that it is stable). If you confirm the same behavior in your environment please raise a support case on the first place (support team must confirm that your Veeam B&R is affected by the same issue before proposing the workaround). Cheers!

orlando_rubino
Lurker
Posts: 1
Liked: never
Joined: May 09, 2019 7:42 pm
Full Name: Orlando Rubino
Contact:

Re: After the 9.5 u4a update I can't control my tape unit

Post by orlando_rubino » May 09, 2019 7:44 pm

I have the same error, where can I download the fix? Thank you

aaron@ARB
Expert
Posts: 132
Liked: 14 times
Joined: Feb 21, 2014 3:12 am
Full Name: ARBCorporationPtyLtd
Contact:

Re: After the 9.5 u4a update I can't control my tape unit

Post by aaron@ARB » May 10, 2019 2:25 am

dariusz.tyka wrote:
May 08, 2019 12:59 pm
I tried this fix so I stopped Veeam tape Access service to have my tape library offline. Then I removed 3 affected tapes from catalog, reenabled Veeam tape service and those tapes were rediscovered as unknown. I move them to free media pool. One of those tapes were in TL2000 drive so I tried to unload it to slot but it ended with following error:
5/8/2019 2:54:23 PM Error Unloading tape 000013L6 from Drive 1 (Server: name, Library: name, Drive ID: Tape0) to Slot 234 Error: Failed to call RPC function 'UnloadMedium': The indicated element does not exist. DeviceIoControl(ChangerGetElementStatus) with eAddress = '233' and number = '1', symLink = '\\.\Changer0'.
So the fix is not working for me unfortunately.

* an update - some features are working OK. Like tape erase - Veeam can grab tape from slot, erase it and unload from the drive. But starting a B2T job fails with an error.
Sounds like veeam may not have deleted the DB records properly as its perhaps picking up the label / barcode / tape ID or what not as the issue is you are still getting "The indicated element does not exist." so I reckon if you were to check the tape_mediums table in your veeam database and see which slot it thinks it belongs in

something like

Code: Select all

SELECT name,barcode,original_slot FROM [VeeamBackup].[dbo].[Tape.tape_mediums] where barcode = 'CJH019L6'
Just change your barcode to whatever your tape is called (or name if you are not using barcodes) and see where the 'original_slot' is

Code: Select all

name		barcode		original_slot
CJH019L6	CJH019L6	13
for an example, you will find a slot that you dont actually have. Erasing and such like would work as once the tape is IN the drive it will work fine, its just the tape operations that rely on its slot (unloading for example)

dariusz.tyka
Influencer
Posts: 18
Liked: 2 times
Joined: Jan 21, 2019 1:38 pm
Full Name: Dariusz Tyka
Contact:

Re: After the 9.5 u4a update I can't control my tape unit

Post by dariusz.tyka » May 10, 2019 8:03 am

Hi aaron@ARB,

tested it and got results that 2 affected tapes are in the same slot 210697635. So it looks like still wrong slot numbers are present in Veeam database.

dariusz.tyka
Influencer
Posts: 18
Liked: 2 times
Joined: Jan 21, 2019 1:38 pm
Full Name: Dariusz Tyka
Contact:

Re: After the 9.5 u4a update I can't control my tape unit

Post by dariusz.tyka » May 10, 2019 11:10 am

Ok it worked finally. But fix requires to physically remove and reinsert tapes to library. Simple veeam tape service off/remove from catalog/service on was not enough. Awaiting for fix from Veeam R&D.

Avi Milbaum
Novice
Posts: 5
Liked: never
Joined: Jul 17, 2016 3:15 pm
Full Name: Avi Milbaum
Contact:

Re: After the 9.5 u4a update I can't control my tape unit

Post by Avi Milbaum » May 12, 2019 9:11 am

I have the same issue with HP MSL2024 G3 Series 7.00

Unloading tape 000193L4 from Drive 1 (Server: PACKER-DP.packer.steels, Library: HP MSL G3 Series 7.00, Drive ID: Tape1) to Slot -1507724375 Error: Failed to call RPC function 'UnloadMedium': The indicated element does not exist. DeviceIoControl(ChangerGetElementStatus) with eAddress = '65535' and number = '1', symLink = '\\.\Changer0'.
Failed to eject tape 000193L4 Details: Failed to call RPC function 'UnloadMedium': The indicated element does not exist. DeviceIoControl(ChangerGetElementStatus) with eAddress = '65535' and number = '1', symLink = '\\.\Changer0'.

Case # 03556160

Is there an idea to bypass this issue. until fix will be available ?

Thanks
Avi Milbaum

Dima P.
Product Manager
Posts: 10709
Liked: 881 times
Joined: Feb 04, 2013 2:07 pm
Full Name: Dmitry Popov
Location: Prague
Contact:

Re: After the 9.5 u4a update I can't control my tape unit

Post by Dima P. » May 12, 2019 11:29 am 1 person likes this post

Hello Avi,

Have your tried the workaround proposed in this thread? Thanks!

Gostev
SVP, Product Management
Posts: 25122
Liked: 3685 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: After the 9.5 u4a update I can't control my tape unit

Post by Gostev » May 12, 2019 9:38 pm 1 person likes this post

While there could be some exceptions, in vast majority of support cases following the Update 4a release, the issue was caused by not having the latest updates installed on Windows Server 2012 R2. The actual bug sits in JIT (Just In Time) compiler logic of .NET Framework 4.6. This bug is also the critical vulnerability > Microsoft Security Bulletin MS15-092

Post Reply

Who is online

Users browsing this forum: George Boncea and 8 guests