Inventory vs. catalog tape: difference?

[Moebius]

Probably a trivial question, but this is something I've never really understood from the beginning: what is the difference between cataloging and inventorying a tape? What does each operation really do? And when should one use one or the other?

[Egor Yakovlev]

Hi Lucio.

Inventory - tells Veeam which tape cartridges you have in your library - we scan them and create an inventory which you can later group in Media Pools. Example: Tape001 and Tape002 or tape cartridge barcodes, if you use them. You should run inventory whenever you bring new cartridges for Veeam to use.

Catalog - tells Veeam what kind of DATA is already on the tape. When Veeam Server finds a new tape, it doesn't know if that tape is fresh and empty or it was used before and some critical data sits there(Veeam backups made on another server?). Catalog scans tape for content and if some backups are present on the tape, Veeam will see it, can use it to restore, and append new data on the tape. You should run Catalog when you bring non-fresh tape with some data on it.

Feel free to read about Inventory and Catalog processes in our documentation.

Hope that helps!

[Moebius]

Thanks Egor, it sure does. Things are clearer now.

[lyapkost]

That's not quite correct. Both inventory and cataloging processes require tape to be loaded to drive and read. The difference is that inventory job only reads tape header which contains some information in MTF format and looks like this if was written with Veeam B&R:

Code: Select all

MTF Media Label|1.0|Veeam|Backup & Replication 9.5|2019/11/13.12:27:40|Veeam 2019/11/13.12:27:40|1|{1d674f3e-57dc-4d73-9ea5-c21cebad6724}|{5b609aa8-a92d-4daf-9c22-add2ce8b9075}|

The header's reading takes several seconds and gives brief information about tape usage but it does not actually give an entire information about the tape content. Cataloging process reads catalog from the tape (which is being written at the end of each session) or scans tape content in case there's no catalog on the tape (for example if we're cataloging only the first of two tapes in a media set, the catalog is located on the second one and can not be read). This process requires tape rewind and may take a lot of time, especially scanning, depending on the tape's content.

Both operations save gathered data to the Veeam B&R database.

[AlexeyG]

I do understand the topic is almost 4 years old, but tapes will always be actual
Thank you Konstantin for the additional details. "Other" backup software can do kind of "quick scan" of the library only using the barcode scanner, without moving every tape to a drive and reading it. Even with 4 drives for a mid size library moving every tape and reading a header can be very time consuming process. Is there any way to do such a "quick scan"? Or that information is supposed to be "pushed" from the library to VBR than the scanning activity performed by the hardware?

Cheers,

[AlexeyG]

I think "Rescan" will synchronize "physical" inventory in VBR with library, so if any tapes were inserted they will appear in the VBR list, correct?

[Vodochnik]

@AlexeyG short answer is yes, rescan asks library itself which cartridges are present, without loading them to drive.

[david.domask]

I can confirm @Vodochnik's answer; rescan just polls the tape hardware on the tape server and asks "what is the state of your tape slots + drives + tapes right now?" This is the same as the discovery process that runs every 180 seconds, and it's done to ensure Veeam knows the current state of the library and what is in slot/drive/IESlot.

This process should be very fast; inventory is done when you have inserted/removed tapes from the device without using the Mail Slots or any other manual actions, and it just synchronizes the state of the Hardware with Veeam's undersanding in the configuration database.

In most cases, you don't need to inventory/catalog tapes unless you're cycling them in/out without using the mail slot or are using the remove from catalog feature frequently; inventory tells us information about the tape like you see in Konstantin's post, Catalog tells us what backups are actually on the tape.

[paruccone]

I've opened Case #07021574 for most probably this: trying to reload old tapes and move everything to LTFS (I need file backup, no VMs or anything else) but in the process it's taking ages (meaning tens of hours for each LTO-4 tape written by Veeam B&R) to rebuild the catalog.

I thought drives/tapes were having issue but no: I've been able to create copies of entire tapes (hence reading all content) in a few hours per tape while Veeam keeps shoeshining the tapes: a lot (like at least 12h) for a tape with about 150x 4 to 8GB files and days for tapes with smaller files. Translated : if that's by design, better to avoid using Veeam to tape for file backups (unless you are 100% sure you'll never loose the machine, OS and DB containing the catalog not to mention the proportional amount of space to store the logs of agent and job... :'(

[paruccone]

17 hours now and going... I strongly recommend file backup/restore to only be available for community edition : paying customers could be quite angry at that. In the meanwhile another LTO4's catalog (same media set) in another drive completed in 1h31... Looking forward to discover more within my case on how catalogs are stored. Veeam is great (the product, the support, etc.) but for everything file I'll use LTFS from now on

[david.domask]

Hi Alessandro ( @paruccone ). On your second post, do I get it right that using another drive it went faster? If you were to catalog the tape that took 17+ hours in same drive, is it also faster?

Cataloging itself is a bit lengthy, but especially for File to Tape jobs, you have to consider not just the tape drive performance, but configuration database performance as there can be a lot to track. I'll check the case next week and give the engineer some ideas on how to isolate this further, but if you can clarify the above would be great. Please report within the case.

[Gostev]

I'm wondering if the tapes in question were written by V12 or earlier versions?

[paruccone]

earlier, 9 or 10 and I also tried to uninstall 12 and install 10: same behavior. Tape has been spinning for more than 26 hours now and more than 12MB of gzipped logs (don't wanna look the unzipped size) so far... I'm fearing for the drive's health (tape cartridge has already been mirrored should it wear out because of the stress)

[Gostev]

Things is, we did not have proper general-purpose file to tape backup capability until V12. What we had there is the engine built to export relatively few of very large files (our backup files) to tape. We did not even license this back then because we did not expect customers to use it as general purpose file to tape backup given its scalability and performance limits for such scenario. And we even added the corresponding recommendation to the UI eventually.

So your comment about this being only available for community edition... is sort of spot on, in a way we gave it away for free knowing you could not do much with it. As such, please don't expect great results with performance from those old tape backups.

It's only V12 where we finally delivered the new, scalable file to tape backup engine that can protect billions of files (or 1000 times more than the original engine).

[paruccone]

very nice of you. I really enjoy Veeam as a brand before their products. But I still have tapes I can't restore is there any chance I can point T&R to a file to do the catalog scanning (I imaged the "problematic" tapes with a do/while dd from /dev/nst* using the appropriate block size so everything is in there) or any other alternative?

[paruccone]

and indeed after 30h of non-stop scraping the tape:
23.11.23 16:32:43.20 TRC/CST <84 15 02 14 > Drive Warn or Crit Tape Alert flag
23.11.23 16:32:43.19 TRC/CST <84 15 02 05 > Drive Warn or Crit Tape Alert flag
23.11.23 16:32:43.18 TRC/CST <84 15 02 04 > Drive Warn or Crit Tape Alert flag
23.11.23 16:32:43.17 TRC/CST <84 15 02 03 > Drive Warn or Crit Tape Alert flag
23.11.23 16:32:43.16 TRC/CST <84 15 02 01 > Drive Warn or Crit Tape Alert flag
23.11.23 16:32:43.15 TRC/CST <82 02 > Drive Cleaning request

[paruccone]

@gostev my questions would be : why some tapes (given that all are readable without errors) are catalogued in 1h30 and some others don't even after more than 27h ? Would it be possible to enable some sort "restore everything that's on tape without going through the catalog?"

Plus, something is still wrong because if I cancel the operation, Veeam keeps sending the tape back and forth, only way to stop that is restarting the system.

[Gostev]

As I noted above, you need to use V12 for fast and scalable file to tape backup and restore. It's too late to do anything about previous versions and their backups, add new options to handle those etc.

As for 27h and the issue with cancelling, you should investigate these with our support engineers as obviously it's not normal.

The log above seems to indicate a physical issue with a tape medium or a tape drive or both. This would explain no results in 27h... if we can't read data from tape, then even eternity would not be enough to finish the restore

[RobTurk]

@gostev my questions would be : why some tapes (given that all are readable without errors) are catalogued in 1h30 and some others don't even after more than 27h ?

Some background for the technically inclined might be in order.

A full tape in perfect condition will take anywhere from ~2 hours (LTO-3) to over 12 hours (LTO-9) to read. Any bottleneck in reading will increase this time.
Tapes are not necessarily full, so it may take less time before the end of data is reached.

When considering our block size on tape (usually 256KB) and the amount of data on tape (up to 18TB native on LTO-9), you can see that it takes a lot of read commands to read the full length of a tape. Hundreds of millions.
When there are weak spots on the tape, a read command may need to retry several times to get the data from the tape. The drive goes through a great number of recovery steps to get you your data back.
The drive manufacturer specifies how long software needs to wait for each command before giving up. On LTO tape drives, the manufacturers specify in the order of 15 minutes or more for read commands.

A weak spot on the tape may impact a lot of blocks, each of which is allowed 15+ minutes to try read that block.
If the recovery succeeds after a few seconds or minutes, the block is sent to the software which then proceeds with the next block. No error is logged, as it's all within specification.

As you can imagine, a few weak spots or damage from abuse is enough to have significant impact on the full read time without ever flagging any error.
When the drive determines that the error or retry rate is beyond acceptable, it will warn the software using a Tape Alert message, which is what you see in the logs. This criteria too, is set by the drive manufacturer.