Bug: "Insert tape" fails job when pulling library magazine

Everything about backing up to tape

Bug: "Insert tape" fails job when pulling library magazine

Veeam Logoby rreed » Mon Jun 20, 2016 6:47 pm

Afternoon, everyone. Respectfully, I would consider this a legitimate bug please. File to tape job, been running for a few days, spanning quite a lot of TB across several tapes. Library eventually runs out of free tapes, fine. Veeam/tape backup job pauses and says "Insert a valid tape into the library," fine.

Get to the data center, pull the magazine to insert a valid tape,
Code: Select all
 "Unloading tape blah from Drive 1 (Server: blah, Library: blah, Drive ID: Tape0) to Slot 13 Error: The indicated element is part of a magazine that is not present. <some other error code stuff here> Timed out waiting for device ready status on ChangerGetElementStatus operation, element 'Change
6/20/2016 10:49:54 AM :: Task <my backup file here> failed. Error: The indicated element is part of a magazine that is not present.
<some other error code stuff here>
--tr:Timed out waiting for device ready status on ChangerGetElementStatus operation, element 'ChangerDrive' ?'0'
The indicated element is part 
6/20/2016 10:49:54 AM ::  [TapeService] Failed to UnloadTapeFromDrive
The indicated element is part of a magazine that is not present.
--tr:Timed out waiting for device ready status on ChangerGetElementStatus operation, element 'ChangerDrive' ?'0'
The indicated element is part of a magazine that "

Looks like some of the error messages in the Statistics also get cut off. Anyways, after a few days, several TB, and a handful of tapes, when I go to give it the more tapes for which it's asking, it fails my job when I do. As far as this "timed out," it was enough time to eject the magazine, throw it up on the bench, load a handful of tapes into it, and slide it back in. Couldn't have been more than a few minutes. I didn't pull the magazine and go to lunch, let alone leave it out for several days, etc. Not expecting my job to fail while I loaded more tapes it was asking for, I didn't think to watch the screen to see if it bombed out the moment I pulled the magazine. Either way, it should #1 not time out at all, really, and #2, if for whatever reason it needs to time out while I'm swapping tapes, at least give me enough time to do so please?
VMware 6
Veeam B&R v9
Dell DR4100's
EMC DD2200's
EMC DD620's
Dell TL2000 via PE430 (SAS)
rreed
Expert
 
Posts: 354
Liked: 72 times
Joined: Tue Jun 30, 2015 6:06 pm

Re: Bug: "Insert tape" fails job when pulling library magazi

Veeam Logoby rreed » Mon Jun 20, 2016 7:03 pm

Are the spanned tapes it had created in this job trash (Free pool fodder) or are each of them usable, each having files written to them?
VMware 6
Veeam B&R v9
Dell DR4100's
EMC DD2200's
EMC DD620's
Dell TL2000 via PE430 (SAS)
rreed
Expert
 
Posts: 354
Liked: 72 times
Joined: Tue Jun 30, 2015 6:06 pm

Re: Bug: "Insert tape" fails job when pulling library magazi

Veeam Logoby v.Eremin » Tue Jun 21, 2016 12:59 pm

If I'm not mistaken the default timeout is set to 10 minutes. Was it the case in your situation?

The said value is configurable via registry by the way:

Code: Select all
ChangerElementFillCompletionTimeoutSec
<DWORD>


Thanks.
v.Eremin
Veeam Software
 
Posts: 13266
Liked: 968 times
Joined: Fri Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin

Re: Bug: "Insert tape" fails job when pulling library magazi

Veeam Logoby rreed » Tue Jun 21, 2016 1:04 pm

I can't imagine that it took me 10 minutes to pull a magazine, stuff a couple of tapes into it, then shove it back in. I was sitting right there next to it. However, I have no evidence to support my case. What is the maximum I can set that to, please? This is applied on the VBR server, correct?

What about my written tapes in the set up to the point it bombed out? Are the files it wrote to the other tapes good and usable or do I need to move all the tapes it wrote in this job back to Free and start all over?
VMware 6
Veeam B&R v9
Dell DR4100's
EMC DD2200's
EMC DD620's
Dell TL2000 via PE430 (SAS)
rreed
Expert
 
Posts: 354
Liked: 72 times
Joined: Tue Jun 30, 2015 6:06 pm

Re: Bug: "Insert tape" fails job when pulling library magazi

Veeam Logoby v.Eremin » Tue Jun 21, 2016 2:47 pm

What is the maximum I can set that to, please?

I don't think it has maximum value.

This is applied on the VBR server, correct?

Correct.

What about my written tapes in the set up to the point it bombed out?

Files already written to tapes should not be corrupted whatsoever. No need to re-write those cassettes.

Thanks.
v.Eremin
Veeam Software
 
Posts: 13266
Liked: 968 times
Joined: Fri Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin

Re: Bug: "Insert tape" fails job when pulling library magazi

Veeam Logoby rreed » Tue Jun 21, 2016 3:38 pm

Very awesome, thanks. Next question, so now that I've loaded up w/ free tapes, what happens when I run an incremental backup? Since the job bombed after writing a bunch of tapes, I'd like for it to pick up where it left off.
VMware 6
Veeam B&R v9
Dell DR4100's
EMC DD2200's
EMC DD620's
Dell TL2000 via PE430 (SAS)
rreed
Expert
 
Posts: 354
Liked: 72 times
Joined: Tue Jun 30, 2015 6:06 pm

Re: Bug: "Insert tape" fails job when pulling library magazi

Veeam Logoby v.Eremin » Wed Jun 22, 2016 8:33 am

Already archived files should not be copied again. Unless, they have changed since the latest run. Thanks.
v.Eremin
Veeam Software
 
Posts: 13266
Liked: 968 times
Joined: Fri Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin

Re: Bug: "Insert tape" fails job when pulling library magazi

Veeam Logoby rreed » Wed Jun 22, 2016 1:20 pm

Awesome as usual, thanks guys. Let's see what it does w/ four days' worth of wait time for me to throw in more tapes.
VMware 6
Veeam B&R v9
Dell DR4100's
EMC DD2200's
EMC DD620's
Dell TL2000 via PE430 (SAS)
rreed
Expert
 
Posts: 354
Liked: 72 times
Joined: Tue Jun 30, 2015 6:06 pm

Re: Bug: "Insert tape" fails job when pulling library magazi

Veeam Logoby rreed » Tue Jul 05, 2016 6:41 pm

Same thing happened again this morning. We were out of tapes over the weekend again, fine. Veeam was waiting patiently for tapes this morning, fine. Pulled one of the magazines, "The indicated element is part of a magazine that is not present" and my job(s) failed. Happened very shortly after I pulled the magazine to insert new tapes.
VMware 6
Veeam B&R v9
Dell DR4100's
EMC DD2200's
EMC DD620's
Dell TL2000 via PE430 (SAS)
rreed
Expert
 
Posts: 354
Liked: 72 times
Joined: Tue Jun 30, 2015 6:06 pm

Re: Bug: "Insert tape" fails job when pulling library magazi

Veeam Logoby v.Eremin » Wed Jul 06, 2016 8:32 am

I'm wondering whether you've specified a new value for changer fill timeout. If so, and it didn't have any effect, please, open a ticket with our support team.

There are different ways to increase various timeouts that support team should be aware of. Chances are, once those are tweaked properly, you won't come across the said problem any longer.

Thanks.
v.Eremin
Veeam Software
 
Posts: 13266
Liked: 968 times
Joined: Fri Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin

Re: Bug: "Insert tape" fails job when pulling library magazi

Veeam Logoby rreed » Wed Jul 06, 2016 1:57 pm

On our VBR server, DWORD, ChangerElementFillCompletionTimeoutSec, 259200 decimal. It and our tape server have been rebooted since adding this key if that should matter.

I'll get a support ticket going and report back.
VMware 6
Veeam B&R v9
Dell DR4100's
EMC DD2200's
EMC DD620's
Dell TL2000 via PE430 (SAS)
rreed
Expert
 
Posts: 354
Liked: 72 times
Joined: Tue Jun 30, 2015 6:06 pm

Re: Bug: "Insert tape" fails job when pulling library magazi

Veeam Logoby cac8907 » Wed Jul 06, 2016 2:07 pm

When you pull the Magazine, does Windows lose connection to the device(s)? In Device manager do they disappear? There are a couple of keys we can try for sure but IIRC removing the magazine makes the device go "offline" and thus does indeed fail the job. Might be able to extend the check so there is more time, but if you just happen to overlap that timer then the problem could still exist. Certainly worth a try! Let us know the case number so we can follow along!
cac8907
Veeam Software
 
Posts: 8
Liked: 5 times
Joined: Thu Mar 14, 2013 10:09 pm
Full Name: Adam Congdon

Re: Bug: "Insert tape" fails job when pulling library magazi

Veeam Logoby rreed » Wed Jul 06, 2016 2:17 pm

Interesting you ask that, because while there were a stack of a bunch of jobs waiting for tape, when I pull a magazine (there are two, BTW so not sure why the whole thing disappears), some of the jobs failed w/ this error and some failed w/ "does not contain any enabled tape drives." I was too busy loading tapes to check to see if device mangler lost the drives/library.
VMware 6
Veeam B&R v9
Dell DR4100's
EMC DD2200's
EMC DD620's
Dell TL2000 via PE430 (SAS)
rreed
Expert
 
Posts: 354
Liked: 72 times
Joined: Tue Jun 30, 2015 6:06 pm

Re: Bug: "Insert tape" fails job when pulling library magazi

Veeam Logoby rreed » Wed Jul 06, 2016 3:01 pm 1 person likes this post

#01846039
VMware 6
Veeam B&R v9
Dell DR4100's
EMC DD2200's
EMC DD620's
Dell TL2000 via PE430 (SAS)
rreed
Expert
 
Posts: 354
Liked: 72 times
Joined: Tue Jun 30, 2015 6:06 pm

Re: Bug: "Insert tape" fails job when pulling library magazi

Veeam Logoby pverburg » Thu Jul 07, 2016 5:28 am

I think you can only use the letterbox/mailbox feature so it can identify the tape inserted and removed.
Had similar problems here but since only using the mailbox option it works every time.

Paul V.
pverburg
Novice
 
Posts: 5
Liked: never
Joined: Fri Jan 29, 2016 1:12 am
Full Name: Paul Verburg

Next

Return to Tape



Who is online

Users browsing this forum: No registered users and 7 guests

cron