Using tape as a backup target
Post Reply
rreed
Expert
Posts: 354
Liked: 72 times
Joined: Jun 30, 2015 6:06 pm
Contact:

Bug: "Insert tape" fails job when pulling library magazine

Post by rreed »

Afternoon, everyone. Respectfully, I would consider this a legitimate bug please. File to tape job, been running for a few days, spanning quite a lot of TB across several tapes. Library eventually runs out of free tapes, fine. Veeam/tape backup job pauses and says "Insert a valid tape into the library," fine.

Get to the data center, pull the magazine to insert a valid tape,

Code: Select all

 "Unloading tape blah from Drive 1 (Server: blah, Library: blah, Drive ID: Tape0) to Slot 13 Error: The indicated element is part of a magazine that is not present. <some other error code stuff here> Timed out waiting for device ready status on ChangerGetElementStatus operation, element 'Change
6/20/2016 10:49:54 AM :: Task <my backup file here> failed. Error: The indicated element is part of a magazine that is not present.
<some other error code stuff here>
--tr:Timed out waiting for device ready status on ChangerGetElementStatus operation, element 'ChangerDrive' ?'0'
The indicated element is part  
6/20/2016 10:49:54 AM ::  [TapeService] Failed to UnloadTapeFromDrive
The indicated element is part of a magazine that is not present.
--tr:Timed out waiting for device ready status on ChangerGetElementStatus operation, element 'ChangerDrive' ?'0'
The indicated element is part of a magazine that "
Looks like some of the error messages in the Statistics also get cut off. Anyways, after a few days, several TB, and a handful of tapes, when I go to give it the more tapes for which it's asking, it fails my job when I do. As far as this "timed out," it was enough time to eject the magazine, throw it up on the bench, load a handful of tapes into it, and slide it back in. Couldn't have been more than a few minutes. I didn't pull the magazine and go to lunch, let alone leave it out for several days, etc. Not expecting my job to fail while I loaded more tapes it was asking for, I didn't think to watch the screen to see if it bombed out the moment I pulled the magazine. Either way, it should #1 not time out at all, really, and #2, if for whatever reason it needs to time out while I'm swapping tapes, at least give me enough time to do so please?
VMware 6
Veeam B&R v9
Dell DR4100's
EMC DD2200's
EMC DD620's
Dell TL2000 via PE430 (SAS)

rreed
Expert
Posts: 354
Liked: 72 times
Joined: Jun 30, 2015 6:06 pm
Contact:

Re: Bug: "Insert tape" fails job when pulling library magazi

Post by rreed »

Are the spanned tapes it had created in this job trash (Free pool fodder) or are each of them usable, each having files written to them?
VMware 6
Veeam B&R v9
Dell DR4100's
EMC DD2200's
EMC DD620's
Dell TL2000 via PE430 (SAS)

veremin
Product Manager
Posts: 17834
Liked: 1662 times
Joined: Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin
Contact:

Re: Bug: "Insert tape" fails job when pulling library magazi

Post by veremin »

If I'm not mistaken the default timeout is set to 10 minutes. Was it the case in your situation?

The said value is configurable via registry by the way:

Code: Select all

ChangerElementFillCompletionTimeoutSec 
<DWORD>
Thanks.

rreed
Expert
Posts: 354
Liked: 72 times
Joined: Jun 30, 2015 6:06 pm
Contact:

Re: Bug: "Insert tape" fails job when pulling library magazi

Post by rreed »

I can't imagine that it took me 10 minutes to pull a magazine, stuff a couple of tapes into it, then shove it back in. I was sitting right there next to it. However, I have no evidence to support my case. What is the maximum I can set that to, please? This is applied on the VBR server, correct?

What about my written tapes in the set up to the point it bombed out? Are the files it wrote to the other tapes good and usable or do I need to move all the tapes it wrote in this job back to Free and start all over?
VMware 6
Veeam B&R v9
Dell DR4100's
EMC DD2200's
EMC DD620's
Dell TL2000 via PE430 (SAS)

veremin
Product Manager
Posts: 17834
Liked: 1662 times
Joined: Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin
Contact:

Re: Bug: "Insert tape" fails job when pulling library magazi

Post by veremin »

What is the maximum I can set that to, please?
I don't think it has maximum value.
This is applied on the VBR server, correct?
Correct.
What about my written tapes in the set up to the point it bombed out?
Files already written to tapes should not be corrupted whatsoever. No need to re-write those cassettes.

Thanks.

rreed
Expert
Posts: 354
Liked: 72 times
Joined: Jun 30, 2015 6:06 pm
Contact:

Re: Bug: "Insert tape" fails job when pulling library magazi

Post by rreed »

Very awesome, thanks. Next question, so now that I've loaded up w/ free tapes, what happens when I run an incremental backup? Since the job bombed after writing a bunch of tapes, I'd like for it to pick up where it left off.
VMware 6
Veeam B&R v9
Dell DR4100's
EMC DD2200's
EMC DD620's
Dell TL2000 via PE430 (SAS)

veremin
Product Manager
Posts: 17834
Liked: 1662 times
Joined: Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin
Contact:

Re: Bug: "Insert tape" fails job when pulling library magazi

Post by veremin »

Already archived files should not be copied again. Unless, they have changed since the latest run. Thanks.

rreed
Expert
Posts: 354
Liked: 72 times
Joined: Jun 30, 2015 6:06 pm
Contact:

Re: Bug: "Insert tape" fails job when pulling library magazi

Post by rreed »

Awesome as usual, thanks guys. Let's see what it does w/ four days' worth of wait time for me to throw in more tapes.
VMware 6
Veeam B&R v9
Dell DR4100's
EMC DD2200's
EMC DD620's
Dell TL2000 via PE430 (SAS)

rreed
Expert
Posts: 354
Liked: 72 times
Joined: Jun 30, 2015 6:06 pm
Contact:

Re: Bug: "Insert tape" fails job when pulling library magazi

Post by rreed »

Same thing happened again this morning. We were out of tapes over the weekend again, fine. Veeam was waiting patiently for tapes this morning, fine. Pulled one of the magazines, "The indicated element is part of a magazine that is not present" and my job(s) failed. Happened very shortly after I pulled the magazine to insert new tapes.
VMware 6
Veeam B&R v9
Dell DR4100's
EMC DD2200's
EMC DD620's
Dell TL2000 via PE430 (SAS)

veremin
Product Manager
Posts: 17834
Liked: 1662 times
Joined: Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin
Contact:

Re: Bug: "Insert tape" fails job when pulling library magazi

Post by veremin »

I'm wondering whether you've specified a new value for changer fill timeout. If so, and it didn't have any effect, please, open a ticket with our support team.

There are different ways to increase various timeouts that support team should be aware of. Chances are, once those are tweaked properly, you won't come across the said problem any longer.

Thanks.

rreed
Expert
Posts: 354
Liked: 72 times
Joined: Jun 30, 2015 6:06 pm
Contact:

Re: Bug: "Insert tape" fails job when pulling library magazi

Post by rreed »

On our VBR server, DWORD, ChangerElementFillCompletionTimeoutSec, 259200 decimal. It and our tape server have been rebooted since adding this key if that should matter.

I'll get a support ticket going and report back.
VMware 6
Veeam B&R v9
Dell DR4100's
EMC DD2200's
EMC DD620's
Dell TL2000 via PE430 (SAS)

cac8907
Veeam Software
Posts: 10
Liked: 6 times
Joined: Mar 14, 2013 10:09 pm
Full Name: Adam Congdon
Contact:

Re: Bug: "Insert tape" fails job when pulling library magazi

Post by cac8907 »

When you pull the Magazine, does Windows lose connection to the device(s)? In Device manager do they disappear? There are a couple of keys we can try for sure but IIRC removing the magazine makes the device go "offline" and thus does indeed fail the job. Might be able to extend the check so there is more time, but if you just happen to overlap that timer then the problem could still exist. Certainly worth a try! Let us know the case number so we can follow along!

rreed
Expert
Posts: 354
Liked: 72 times
Joined: Jun 30, 2015 6:06 pm
Contact:

Re: Bug: "Insert tape" fails job when pulling library magazi

Post by rreed »

Interesting you ask that, because while there were a stack of a bunch of jobs waiting for tape, when I pull a magazine (there are two, BTW so not sure why the whole thing disappears), some of the jobs failed w/ this error and some failed w/ "does not contain any enabled tape drives." I was too busy loading tapes to check to see if device mangler lost the drives/library.
VMware 6
Veeam B&R v9
Dell DR4100's
EMC DD2200's
EMC DD620's
Dell TL2000 via PE430 (SAS)

rreed
Expert
Posts: 354
Liked: 72 times
Joined: Jun 30, 2015 6:06 pm
Contact:

Re: Bug: "Insert tape" fails job when pulling library magazi

Post by rreed » 1 person likes this post

#01846039
VMware 6
Veeam B&R v9
Dell DR4100's
EMC DD2200's
EMC DD620's
Dell TL2000 via PE430 (SAS)

pverburg
Novice
Posts: 5
Liked: never
Joined: Jan 29, 2016 1:12 am
Full Name: Paul Verburg
Contact:

Re: Bug: "Insert tape" fails job when pulling library magazi

Post by pverburg »

I think you can only use the letterbox/mailbox feature so it can identify the tape inserted and removed.
Had similar problems here but since only using the mailbox option it works every time.

Paul V.

rreed
Expert
Posts: 354
Liked: 72 times
Joined: Jun 30, 2015 6:06 pm
Contact:

Re: Bug: "Insert tape" fails job when pulling library magazi

Post by rreed »

To be sure I'm clear; if I need to swap out 23 tapes, I'm to use the mail slot instead of pulling a magazine? Is that the expectation?
VMware 6
Veeam B&R v9
Dell DR4100's
EMC DD2200's
EMC DD620's
Dell TL2000 via PE430 (SAS)

pverburg
Novice
Posts: 5
Liked: never
Joined: Jan 29, 2016 1:12 am
Full Name: Paul Verburg
Contact:

Re: Bug: "Insert tape" fails job when pulling library magazi

Post by pverburg »

rreed wrote:To be sure I'm clear; if I need to swap out 23 tapes, I'm to use the mail slot instead of pulling a magazine? Is that the expectation?
Just giving you answer what I have found and worked for me.
Maybe your library is to small or your job to big ??

rreed
Expert
Posts: 354
Liked: 72 times
Joined: Jun 30, 2015 6:06 pm
Contact:

Re: Bug: "Insert tape" fails job when pulling library magazi

Post by rreed »

Many thanks Paul, do appreciate it. Just cringed at the thought of changing 23 tapes via mail slot. :wink:

Not sure it's the number of jobs/library capacity, it seems more of a deep config issue to flush out. Support gave me some reg hack timeouts to set, once this round of tape jobs is done (some time next week?) I'll apply, bounce, and we'll see what happens.
VMware 6
Veeam B&R v9
Dell DR4100's
EMC DD2200's
EMC DD620's
Dell TL2000 via PE430 (SAS)

Post Reply

Who is online

Users browsing this forum: No registered users and 9 guests