Fatal Veeam Tape Infrastructure behavior ...

[Didi7]

Hello Guys,

today we were forced to replace our HPE MSL2024 tape library with another chassis, the controller had to be replaced as well and so VBR recognizes another tape library now and the old one is no longer available.

So far so good. I thought no problem. Let‘s remove the old library in the tape inventory and rename the new one. Of course this didn‘t work, first I had to reconfigure every media set and replace the old (no more available) library with the new one.

Afterwards, I was able to remove the old library from the tape infrastructure. The result was, that all offline and online tapes were offline now and they all were no longer associated with a tape library.

I did a scan on the new library from the inventory, which resulted in duplicate barcode label entries. I removed the duplicate entries, which were offline. Now all tapes, which are available online were associated to the new library.

Unfortunately all tapes, which are offline, like monthly or yearly tapes are still offline, which is fine, but they cannot be re-associated to the new library, which is the feature that‘s missing in Veeam B&R.

Reconfiguring media sets with offline tapes that were originally associated with another tape library from the tape infrastructure and are now associated to no tape library results in an error that has to be acknowledged first before configuration of this media set is possible and can continue and this error will come every time you start to configure this media set, as long as those offline tapes are not associated to an existing tape library, which is not configurable at the moment.

The only way I know to clean up these media sets is to completely remove those tapes from the tape catalog and re-insert those tapes again (which btw is very time consuming) and re-inventory those tapes again and move it to the original media sets manually and finally physically remove those tapes again from the library, as in this case those tapes are associated to a library from the tape infrastructure again.

So, what‘s missing in VBR now, when you need to replace a library for whatever reason? A feature, which enables you to associate tapes (originally associated to a former used tape library, which may be dead now or which needs to be replaced with another type of library now) to another recently added library in the tape infrastructure.

And what‘s still annoying after years of using VBR now? The fact, that the catalog is removed from tapes, which are manually associated to another media set. Why does VBR not ask, if the catalog for a tape should be removed or not, when it‘s moved to another media set than the ‚Free‘ media set?

Still missing as well since years is the ability to move expired tapes to the ‚Free‘ media set automatically.

Gostev is talking about the importance of AIR-GAPPED backups. Tape is still not dead after years and especially since Ransomware is very popular and known for encrypting backup files as well.

So, when will Veeam add those long awaited features to their tape infrastructure features finally?

Best regards,
Didi7

[Dima P.]

Hello Didi7,

To clarify, when saying media set you were referring to media pool? When the tape library was removed from the media pool did you remove offline tape media from the media pool? If you replace one library with another from the media pool properties and then import tapes from your new library catalog operation is not required. Thanks!

[Didi7]

To clarify, when saying media set you were referring to media pool?

Yes, exactly.

When the tape library was removed from the media pool did you remove offline tape media from the media pool?

No, I just replaced the removed Tape-Library with the new one.

If you replace one library with another from the media pool properties and then import tapes from your new library catalog operation is not required.

Import tapes from my new library? You mean into my new library? How is that possible, if I remove offline tape media, when I replace the old Tape-Library in the media pool with a new Tape-Library? In this case, all offline media should be gone and nothing can be imported?

As I said above, I did not remove any tape media, when I replaced the old Tape-Library with the new one in every media pool. Inventoring media, which is located in the library is no problem. I get duplicate barcode labels and can remove those, which are not associated with any Tape-Library.

If I replace the old Tape-Library in every media pool with a new Tape-Library and remove offline tape media from the media pool, all those offline tape media would be completely removed from the catalog, so how can I import this media again, when replacing the old with the new Tape-Library?

The only way I see is to re-insert those offline tape media and re-invent it again or do I miss something?

And in this case I miss the feature to associate all tape media, that was associated to the old Tape-Library to the new one, when replacing Tape-Library in the media pool.

Best regards,
Didi7

[Dima P.]

Thanks for the clarification, now I see the problem.

I get duplicate barcode labels and can remove

It looks like a bug and needs to be investigated by our support team. Kindly, open support case and share the case Id with us. Thank you!

[Didi7]

It looks like a bug and needs to be investigated by our support team.

Is it really a bug or just a missing feature? To me VBR behaves logically. All tape media is associated to a Tape-library of the Tape Infrastructure. Replacing a defective Tape-library or upgrading to a new Tape-library results in an offline Tape-library and one new Tape-library. I don't know, if it's possible to move tape media from one to another Tape-library, when both are online and you just add the same or a similar Tape-library from the same vendor, that's able to read the same tape media.

Maybe you should try this step in your own Veeam labs? Is it possible to move tape media from one Tape-library to another Tape-library? If that's not possible, then a lot of your customers (who e.g. need to replace a Tape-library for whatever reason) will have lots of work re-associating offline tape media to another Tape-library by deleting those offline tape media, because it is no longer associated to a Tape-library, because the original Tape-library needs to be removed (worthless entry) and a new Tape-library won't replace the existing one, even if it's the same model. Customers need to re-insert this offline tape media again, do an inventory in order to have those tape media in the catalog again.

You can call it a bug, I call it a missing feature (re-associating tape media to a new or additional Tape-library).

What I also miss is the feature to keep the tape media contents, if individual tape media is moved from one to another media pool manually! This is common in traditional backup software from other vendors but VBR deletes the tape media contents. Re-cataloging the tape media contents is necessary in this case. Why? And why does Veeam not add the feature to move expired tape media to the Free media pool automatically?

Please be informed, that in this case VBR 9.5 U3 is still used, as this server wasn't upgraded to U4 until now.

Best regards,
Didi7

[Dima P.]

Hello Didi7,

Is it really a bug or just a missing feature?

Catalog of used tape media is not linked to the tape library (this improvement was made in v8 if I am not mistaken), so the change of the tape device should not produce duplicated tape media records. Instead Veeam B&R must detect that it is a same tape media that was used before.

Maybe you should try this step in your own Veeam labs?

Can confirm that it should work as I described above.

Is it possible to move tape media from one Tape-library to another Tape-library?

Yes that is possible and does not require additional care.

You can call it a bug, I call it a missing feature (re-associating tape media to a new or additional Tape-library).

Most likely you faced an infrastructure specific issue and your tapes were not correctly recognized. That is the reason I've asked you to submit support case and investigate this behavior with our support team.

What I also miss is the feature to keep the tape media contents, if individual tape media is moved from one to another media pool manually! This is common in traditional backup software from other vendors but VBR deletes the tape media contents. Re-cataloging the tape media contents is necessary in this case. Why? And why does Veeam not add the feature to move expired tape media to the Free media pool automatically?

Information about tape sequence, backup set, media set and retention is linked to the media pool, so currently the only way to break this 'bond' - perform mark as free operation. I'll add your vote to this improvement request. Cheers!

[nathano]

hmmm, I've had a similar issue when some tapes were moved to a new tape library - still have the old library as well. I got duplicate barcode warnings until I removed the media from the old library. It didn't recognize the media being moved from one library to another.

[Dima P.]

Nathan,

If you can confirm this behavior with latest Veeam B&R 9.5 U4a please open a support case and share the case ID. I'd love to ask our RnD folks to investigate the debug logs. Cheers!

[nathano]

I've just tried moving a tape and it worked OK. it showed up as unrecognized until I did an inventory and then it detected the media. no duplicate barcode warnings. so what I have seen previously must have been a bug that's now resolved.

[Dima P.]

Hi folks,

Thanks to QA team we found the root cause: old tape device must be removed from Veeam B&R server before you start the rescan of your need library (otherwise you will see tape media duplicates, since existing tape media will be 'linked' to the old tape library. We will create a Help Center article to make sure the emigration procedure is well explained and wont cause further issues. Cheers!

[nathano]

In our case we added a new library but also kept the old one so we would not be removing it.

[Dima P.]

Nathan,

You kept old library registered or added to the media pool (or both)? Thanks!

[nathano]

yep, kept the old library registered and moved the tapes to the new library.... was nearly 3 years ago now.. so my memory is fading a bit

Will you be at VeeamOn?

[Dima P.]

Thanks for clarification! Yup, I have session dedicated to tape support and will be glad to have a chat with you

[nathano]

That session is on my schedule

[leedudmish1]

hi All,

hope you can help me, our autoloader and tape drive failed and was replaced with another device, have got this all connected to the server however when we put media which has already been written to tape into the device it is detecting a duplicate barcode rather than seeing the tape which already exists in the catalog.

I have changed the media set association over so that its now looking at the new tape drive and have removed the old drive from the console however still getting the same issues above.

under the tape library section on all offline and previous media nothing is listed however dont see a way to associate this with the drive so it recognises the media as whats previously been written and more importantly whats already been cataloged in the database.

for good measure i upgraded from 9.5u2 to 9.5u4a however still dont have a fix for this.

reading around suggestions have been that the tapes may need to be removed/readded to catalog however this seems like a massive task for something as simple as a tape drive switch over and with over 200 tapes in the catalog it will take forever to do this.

Luck would have it this is legacy data system and not full prod but ideally would not want to recatalog the entire lot of tapes.

any assistance is gratefully received on this

thanks

Lee

[Dima P.]

Hello and welcome to the community Lee,

I've merged your post to the existing discussion. Aside from the proposed workaround I can only suggest to raise a support case and let our team investigate the application debug logs. Thank you!

[lyapkost]

Hi Lee.

There are two ways to change an old library to a new one without cataloging all the tapes and having duplicate barcodes.

1. You need to have at least two libraries in media pools.
a) Disconnect and remove old library from media pools and VBR (but don't remove offline tapes from infrastructure)
b) Move tapes from the old library to the new one
c) Connect the new library and rescan it - tapes should appear in their media pools
d) Add the new library to media pools

2. No pre-requirements.
a) Connect new library, rescan your tape server
b) Add the new library to media pools
c) Remove old library from media pools
d) Disconnect and remove the old library from VBR (but don't remove offline tapes from infrastructure)
e) Move tapes from the old library to the new one
f) Rescan the new library - tapes should appear in their media pools