Bug: "Insert tape" fails job when pulling library magazine

[rreed]

Afternoon, everyone. Respectfully, I would consider this a legitimate bug please. File to tape job, been running for a few days, spanning quite a lot of TB across several tapes. Library eventually runs out of free tapes, fine. Veeam/tape backup job pauses and says "Insert a valid tape into the library," fine.

Get to the data center, pull the magazine to insert a valid tape,

Code: Select all

 "Unloading tape blah from Drive 1 (Server: blah, Library: blah, Drive ID: Tape0) to Slot 13 Error: The indicated element is part of a magazine that is not present. <some other error code stuff here> Timed out waiting for device ready status on ChangerGetElementStatus operation, element 'Change
6/20/2016 10:49:54 AM :: Task <my backup file here> failed. Error: The indicated element is part of a magazine that is not present.
<some other error code stuff here>
--tr:Timed out waiting for device ready status on ChangerGetElementStatus operation, element 'ChangerDrive' ?'0'
The indicated element is part  
6/20/2016 10:49:54 AM ::  [TapeService] Failed to UnloadTapeFromDrive
The indicated element is part of a magazine that is not present.
--tr:Timed out waiting for device ready status on ChangerGetElementStatus operation, element 'ChangerDrive' ?'0'
The indicated element is part of a magazine that "

Looks like some of the error messages in the Statistics also get cut off. Anyways, after a few days, several TB, and a handful of tapes, when I go to give it the more tapes for which it's asking, it fails my job when I do. As far as this "timed out," it was enough time to eject the magazine, throw it up on the bench, load a handful of tapes into it, and slide it back in. Couldn't have been more than a few minutes. I didn't pull the magazine and go to lunch, let alone leave it out for several days, etc. Not expecting my job to fail while I loaded more tapes it was asking for, I didn't think to watch the screen to see if it bombed out the moment I pulled the magazine. Either way, it should #1 not time out at all, really, and #2, if for whatever reason it needs to time out while I'm swapping tapes, at least give me enough time to do so please?

[rreed]

Are the spanned tapes it had created in this job trash (Free pool fodder) or are each of them usable, each having files written to them?

[veremin]

If I'm not mistaken the default timeout is set to 10 minutes. Was it the case in your situation?

The said value is configurable via registry by the way:

Code: Select all

ChangerElementFillCompletionTimeoutSec 
<DWORD>

Thanks.

[rreed]

I can't imagine that it took me 10 minutes to pull a magazine, stuff a couple of tapes into it, then shove it back in. I was sitting right there next to it. However, I have no evidence to support my case. What is the maximum I can set that to, please? This is applied on the VBR server, correct?

What about my written tapes in the set up to the point it bombed out? Are the files it wrote to the other tapes good and usable or do I need to move all the tapes it wrote in this job back to Free and start all over?

[veremin]

What is the maximum I can set that to, please?

I don't think it has maximum value.

This is applied on the VBR server, correct?

Correct.

What about my written tapes in the set up to the point it bombed out?

Files already written to tapes should not be corrupted whatsoever. No need to re-write those cassettes.

Thanks.

[rreed]

Very awesome, thanks. Next question, so now that I've loaded up w/ free tapes, what happens when I run an incremental backup? Since the job bombed after writing a bunch of tapes, I'd like for it to pick up where it left off.

[veremin]

Already archived files should not be copied again. Unless, they have changed since the latest run. Thanks.

[rreed]

Awesome as usual, thanks guys. Let's see what it does w/ four days' worth of wait time for me to throw in more tapes.

[rreed]

Same thing happened again this morning. We were out of tapes over the weekend again, fine. Veeam was waiting patiently for tapes this morning, fine. Pulled one of the magazines, "The indicated element is part of a magazine that is not present" and my job(s) failed. Happened very shortly after I pulled the magazine to insert new tapes.

[veremin]

I'm wondering whether you've specified a new value for changer fill timeout. If so, and it didn't have any effect, please, open a ticket with our support team.

There are different ways to increase various timeouts that support team should be aware of. Chances are, once those are tweaked properly, you won't come across the said problem any longer.

Thanks.

[rreed]

On our VBR server, DWORD, ChangerElementFillCompletionTimeoutSec, 259200 decimal. It and our tape server have been rebooted since adding this key if that should matter.

I'll get a support ticket going and report back.

[adam_c]

When you pull the Magazine, does Windows lose connection to the device(s)? In Device manager do they disappear? There are a couple of keys we can try for sure but IIRC removing the magazine makes the device go "offline" and thus does indeed fail the job. Might be able to extend the check so there is more time, but if you just happen to overlap that timer then the problem could still exist. Certainly worth a try! Let us know the case number so we can follow along!

[rreed]

Interesting you ask that, because while there were a stack of a bunch of jobs waiting for tape, when I pull a magazine (there are two, BTW so not sure why the whole thing disappears), some of the jobs failed w/ this error and some failed w/ "does not contain any enabled tape drives." I was too busy loading tapes to check to see if device mangler lost the drives/library.

[rreed]

#01846039

[pverburg]

I think you can only use the letterbox/mailbox feature so it can identify the tape inserted and removed.
Had similar problems here but since only using the mailbox option it works every time.

Paul V.

[rreed]

To be sure I'm clear; if I need to swap out 23 tapes, I'm to use the mail slot instead of pulling a magazine? Is that the expectation?

[pverburg]

To be sure I'm clear; if I need to swap out 23 tapes, I'm to use the mail slot instead of pulling a magazine? Is that the expectation?

Just giving you answer what I have found and worked for me.
Maybe your library is to small or your job to big ??

[rreed]

Many thanks Paul, do appreciate it. Just cringed at the thought of changing 23 tapes via mail slot.

Not sure it's the number of jobs/library capacity, it seems more of a deep config issue to flush out. Support gave me some reg hack timeouts to set, once this round of tape jobs is done (some time next week?) I'll apply, bounce, and we'll see what happens.