Un Retire Tapes

[IHeartCats]

Hi Folks,

I found a 3 year old thread asking this question but I thought I'd start a new one, incase there is a different answer now.

Full backups to tape this weekend.
2 Tapes have become retired.
Looked at the Veeam logs, it looks like both tapes were retired while using the same drive.
Tape library logs show that drive required cleaning a couple of times on that day.

Since that point the drive has been used again and has had no further problems.

Hence I believe the two retired tapes are 99% like to be OK (They aren't old or anything) I think just a dirty drive caused the write issues.

How can I give them a second chance and un retire them?

If the tapes are dodgy I'm assuming they will get retired again and then I'll actually remove them.

Thanks
H

[DGrinev]

Hi Helen,

Briefly discussed your issue with the QA team and here is a short guide for you:
1. Make export of the tapes.
2. They should appear as offline in the catalog, so you can initiate Remove from catalog.
3. Insert them back in the Library, they will be recognized like a new tape in Unrecognized Media Pool.
4. You can try to use them again. Thanks!

[IHeartCats]

Hi,

Thanks for the info.

I should elaborate, currently they have backup images on them which are still required. Everything I have read suggests that you should still be able to use images on the retired tape for restore where possible, until the data expires?

I'm assuming the process you describe looses any data on them?

Ejecting them from the library will be fun. Our library is offsite

Do you think there will every be the option to "un retire" tapes without needing this manual intervention? It's something we can do easily in our other backup software!

Thanks
Helen

[csydas]

Hi Helen,

I think we goofed with this as well at some point in our environment and had Veeam Support help us. "Retired" is just some boolean in the database. Take a backup of your Veeam Database, look for the tapes section, and I think you can just flip the bit there. But obviously not official

[DGrinev]

I'm assuming the process you describe looses any data on them?

No, all the data will stay untouched, but the thing you should do when the tapes are back is right click on them and select "Catalog Tape" to preserve content.

Do you think there will every be the option to "un retire" tapes without needing this manual intervention?

I think the chances are low, since a tape gets Retired flag when the time has come and not accidentally, according to our experience.

Please keep in mind, It's not an official/unsupported solution that Harvey mentions, so make sure you have a backup before any changes in the database. Thanks!

[tecserve]

When do you guys expect the fix to be out for this?
I mean this is a pretty ridiculous exercise just to get our tapes back, not to mention the idiotic auto-retirement instead of complaining about the need for cleaning...

[andyman3000]

I've had this issue many times and am up to a good stack of "Retired" tapes. I'm using Quantum i6 LTO7's and I think we've sorted this out but hope I'm not speaking too soon.

I believe if you are copying a large number of VM's (in my case fulls are about 1,300 VM's to 10-12 LTO7's), use application managed cleaning NOT tape library managed.

If you do library managed cleanings, tape jobs seemed to get interrupted for cleanings and tapes get expired as no "EOD" marker is written to tape. Usually I would get errors like "Tape Error 21: Device in Use" in Veeam which correlate to when the library attempted to do cleanings (pulling out active tape, loading cleaning tape, then attempting to reload the original tape during the job), and then intermittently the library would show as offline unless I re scanned the library which would significantly cut into my backup-to-tape windows as jobs needed to re-enumerate.

Also if you do application managed cleanings, You need to have the cleaning tapes in the normal partition slots (i.e. not the cleaning partitions) so that Veeam can see them and use them during a job and not cause the job to fail. Also disable Library Managed cleanings on the i6.

As far as expired tapes, I think the cleaning configuration has caused some but maybe not all of my problems. I will try the solution above to reclaim some of my Expired tapes, now that I'm hoping we fixed the cleaning tape configuration. I'm working with the Quantum vendor and have had to replace drives on multiple occasion, we are also suspecting Humidity/Heat may be causing some issues for us as there are more tracks on the LTO7 than older generations, which can also cause tracking & read write issues that cause tapes to expire. I am going to see if our Vaulting company can add the tapes into the original cases as well for added protection. Implementing tape for Veeam has not been all that easy, but I think I'm getting closer to weeding out all these issues.

[pshute]

Hi Helen,

Briefly discussed your issue with the QA team and here is a short guide for you:
1. Make export of the tapes.
2. They should appear as offline in the catalog, so you can initiate Remove from catalog.
3. Insert them back in the Library, they will be recognized like a new tape in Unrecognized Media Pool.
4. You can try to use them again. Thanks!

I have this issue too. Our tape drive had some kind of firmware error last night, and the error code suggested powering the drive off and on. Now I can do tape operations on other tapes, but the tape that was in it has been marked Retired.

The tape could genuinely have problems, as it's been in use for a while, so before I try unretiring it, is there a way to get statistics about the tape? Eg. how many errors it's had, etc.

And if I do decide to try unretiring it, the instructions above don't work - If I right click on it, the Export Tape option is greyed out.

[pshute]

And if I do decide to try unretiring it, the instructions above don't work - If I right click on it, the Export Tape option is greyed out.

I worked it out. I have a standalone drive, so I needed to eject the tape instead of export it. Then I was able to Remove From Catalog, etc. I'm currently doing a test backup on it.

So I just need to know if there's a way to get statistics for the tape's error history, assuming I haven't wiped any history that existed when I un-retired it.

[Dima P.]

Guys,

Have you tried to Inventory that tape from the drive? I am under impression that inventory operation on tape media should update the expired stated, so if you faced this issue please check it and update this thread with the results. Thank you in advance.

[pshute]

I've unretired it now (and it worked fine for a 2TB backup after), so I can't check, but I thought I tried an inventory operation on it. I thought it it just refused, saying it was retired. Can't remember now. It's a pity the history entries for inventory jobs don't say which tape it found in the drive.

[Dima P.]

Noted, thanks Peter!

[RZRnd]

After a "dirty" drive I got the same problem. One existing good media was marked as retired and there was no option to mark it as good again.

So I look in the sql database with ssms and watched all the tables which could be relevant. I did the following changes

/removed by moderator/

Refresh the GUI and the media was a good one again and I could use it directly for the next backup job.

[Dima P.]

Hello and welcome to the community RZRnd.

Thank you for sharing the hint, but unfortunately I had to remove part of your post as we do not recommend any modifications of the backup server database (unless its performed with our support team).

Next time you notice that tape is being marked as retired - please raise a support case (this would also help us to investigate the root cause of this issue and make the required adjustments in the product). Cheers!

[RZRnd]

Sorry, but if there will be no other option to unretire a tape, where is the problem to unretire it directly in DB?
I think every guy who is "not retired" could imagine that it should be handled with care when working in DB direct.

And (sorry) I will not open any support case for some such easy error. I think the veeam support team has some more difficult and tricky tickets than un-retiring tapes. OR what would the supporters do in that case?

But thank you anyway for your post.

[Dima P.]

Generally, Veeam B&R does not mark tapes as retried on it's own - this information is received from the tape device and then such tape is being marked as retired in Veeam B&R database. If tape media is being accidentally detected as retired we would love to investigate the root cause based on the collected debug logs. Cheers!

[RZRnd]

So I have all the log files and could transmit it to you / support anyway.

The tape drive was a little bit dusted. On two tries to write a backup to tape it was not possible - CRC read failure also suspected hardware drive failure. So I opened it and there was "a lot of" dust in the shaft and drive inside. After I cleaned the drive I started a test reading and then I unretired the tapes, marked them as "free" and started the backup again. Worked without any failure or something else. Also the verfication of data.
So the reason for the retired-flag it looked so was only the little bit dust inside and the write and read errors.

[Regnor]

Generally, Veeam B&R does not mark tapes as retried on it's own - this information is received from the tape device and then such tape is being marked as retired in Veeam B&R database. If tape media is being accidentally detected as retired we would love to investigate the root cause based on the collected debug logs. Cheers!

We often see firmware or hardware problems with the drive or library itself, which cause IO or CRC errors.
And because of those errors Veeam markes all used tapes as retired.

[Dima P.]

Thanks a lot folks! Now I understood the issue, consider the improvement request noted.

[Regnor]

Great, thanks.
It's really annoying if a whole set of tapes gets retired

[RZRnd]

...which was the reason for me to look at my own for a solution to un-retire brandnew and good tapes.

[Regnor]

I've just noticed that you can catalog a retired tape, which moves them to the "Imported" media pool. From there the tape can be moved to any other media pool and the retired flag is gone. So no need to export/remove from catalog or manipulation of the retired flag.
Does this look like a valid workaround?

[busitech]

Changing the tape status away from Retired and Locked in order to make many tapes usable at once should definitely be supported within the management UI. We had a tape drive with a defective lock sensor screw up several boxes worth of tapes, and there was no easy recovery process. We opted for database level SQL instead.

[Vodochnik]

I've just noticed that you can catalog a retired tape, which moves them to the "Imported" media pool. From there the tape can be moved to any other media pool and the retired flag is gone. So no need to export/remove from catalog or manipulation of the retired flag.
Does this look like a valid workaround?

Trying your solution... If I try to move a tape from "Imported" to original pool, veeam want to mark tape as free.


Do I have to recatalog it second time?

[Dima P.]

John,

Right. When the tape is moved to another media pool we need to remove it from the catalog, however, that does not erase the data on tape. The data will be overwritten either by the job run when such tape is used during the backup or by the erase operation which can be started manually. Thank you!