Restore from Tape - Invalid Data

[PariPari]

Hey,

we have an open case #00749517 since over a month now.

When trying to restore from tape from last december, the restore fails with "Restore error ERROR: Cannot Restore file. invalid data". It seems like Veeam is unable to read the start of the tape (its own catalogue).
The Tape in Question contains the vbk file.
Veeam Support tried to mark the tapes free and re-cataloge them, but that didn't work. The Full restore point is missing now in the chain and the increments are worthless without.

The Backup to tape job did run successful back in december.

We have two major issues now:
Corrupt Backup without veeam saying so
Restore not working at all...

Is there any way we can read the vkb from the tape?

regards

[Dima P.]

PariPari,
I’ve requested an update regarding your case. Meanwhile, kindly, continue to work with the support team, I am aware QA and DEV teams are involved, so your case has already been escalated to the highest level of support.

I’ll follow up as soon as I hear anything. Thank you for your patience and sorry for any inconvenience caused.

[PariPari]

According to Support 2 out of 4 Tapes are "empty". This is a serious issue because the initial backup to tape job was dark green successfull. In addition, it was the annual backup and the only backup containing the required data to restore.
The customer is now questioning the support renewal (licence will run out in 30days)
Does the catalog gets verified after writing? It would be nice to have an explanation about the actions Veeam takes to ensure data integrity... So at least i can tell the customer something...

[Dima P.]

Hello,

Does the catalog gets verified after writing? It would be nice to have an explanation about the actions Veeam takes to ensure data integrity... So at least i can tell the customer something...

Yes it does. Each subsequent tape knows the previously used media and keeps the information that backup was not completed (next tape will be used during backup). Last tape in the sequence keeps the list of all media in the sequence. Such approach ensures the restore possibility even if the VBR database was corrupted or lost.

According to Support 2 out of 4 Tapes are "empty".

Here is what I've got from your case - two tapes from the backup sequence (out of four) are missing and that is the reason why you can’t complete the restore. Can you, please, clarify was it actually erased or, maybe, there is a chance it was used by other tape software? In addition, it might be caused by retention – first two tapes were taken and overwritten by tape job while other two tapes remain untouched. Also, was the said media labeled with barcodes? Thank you.

[PariPari]

Hey, thanks for your answer.

Does Veeam compare every block written on tape afterwards? Thats what i.e. backup exec does. Its an optional feature, because it takes significant additional time for the tape job.

The 4 tapes were taken out to a safe location (annual backup). We cyclically overwrite the tapes (50 Tapes are used), the GFS system is maintained manually. We need the full + daily incs for the whole week on one tapeset, so we use the settings "always continue using current media set". Every tuesday a new media set starts and all tapes get exchanged. The customer is really thoughtful about his backup. All tapes are labeled with barcodes and Veeam is the only backup software used.

All 4 tapes have data on them according to "capacity/free", but on 2 tapes there is nothing to see under "properties -> files" which is really strange. During one of the remote sessions Veeam support marked all 4 tapes as free and started several catalog operations (none of them successful). I guess thats the reason there is nothing to see there now? I am 100% certain there was content under "properties -> files" on all tapes before said remote session.
The vbk in question spanned over 3 Tapes. Tape #2 is empty as it seems. The other empty tape must have contained some .vibs.

There must have been some sort of error (human/software) during the media set for that week and thats why there are 2 empty tapes now. I guess, there is no option to clarify what has happened.

Btw is there an option to extend the retention for log files in veeam?

[Dima P.]

PariPari,

Its an optional feature, because it takes significant additional time for the tape job.

We have this feature in our long-term list but since it was not requested much - we left if for the next versions. The backup files are verified by the original backup job (in addition it could be tested via SureBackup) - tape verification is mostly needed to check the media integrity to avoid the magnetic tape corruption.

During one of the remote sessions Veeam support marked all 4 tapes as free and started several catalog operations (none of them successful).

Mark as free does not delete the data – only clears the catalogue inside the VBR database, once the catalogue operation is performed again, the said media will be dropped to the imported media pool and all the data could be restored (some users do that to avoid tape being overwritten by retention) so that should not be the case.

There must have been some sort of error (human/software) during the media set for that week and thats why there are 2 empty tapes now. I guess, there is no option to clarify what has happened.

If it is possible - you can search the tape logs for any operation on these erased media – you can find something if the tape was properly labeled, search for any mentioning of this barcode in the tape logs. Since I’ve access to your support case details – can you post here the barcodes of these tape? Do my best to check your logs, while support team is still investigating.

is there an option to extend the retention for log files in veeam?

This article should help you to change the log retention, max size, etc: How to Change the Veeam Backup & Replication Log Files Folder Location, File Size and Count
By the way, we have a special protect switch that could be enabled on the tape to avoid its being erased or overwritten. Same as hardware write protect mechanism on the media itself, but on the VBR level.

[PariPari]

Hey,

my last mail to the support case should contain all Barcodes.
The whole Log folder is over 1GB in size. I have uploaded the tape job logs (into folder 26.03.2015).
If there is anything missing please let me know and i will upload it.

tape verification is mostly needed to check the media integrity to avoid the magnetic tape corruption

Feature Request I am not really concerned about the integrity of the original files, I feel Veeam does a good job there.
But if a tape is bad and Veeam doesn't notice this, its an issue because the tape backup is worthless then.

[Dima P.]

But if a tape is bad and Veeam doesn't notice this, its an issue because the tape backup is worthless then.

Agree with you. Counted as feature request!

my last mail to the support case should contain all Barcodes

Ok, great. I’ll search and update this post based on the results of my finings. Thank you.

[PariPari]

Hi,

i have another question regarding this case.

When "cataloging" a tape, Veeam trys to read the Catalog from Tape.
Is there another operation thats reads everything from tape, no matter what the catalog says?

Like, in Data Portector you can "cataloge" and "Import" a Tape. "Catalog" Operation reads the catalog from tape. "Import" is a extensive operation (takes several hours) that reads everything(!) from tape and lists the data it has found.

So, assumed the catalog from the media is defective: is there another way import the whole tape?
Or is that not possible because the tape is from a spanned media set?

[Dima P.]

Hello PariPari,

Like, in Data Portector you can "cataloge" and "Import" a Tape. "Catalog" Operation reads the catalog from tape. "Import" is a extensive operation (takes several hours) that reads everything(!) from tape and lists the data it has found.

It good to know, thanks! We read only the header and tape catalogue. For me, it looks like a part of the verification request you proposed before

P.S. Still checking your logs with the support team.

[Dima P.]

Hello,
Just wanted to follow up and check how things are going? I’ve noticed that you received a private fix for the media in the question – did it help you to restore?