How to verify tape backup content and integrity

[EnerconIT]

Case ID:01786446

We have Veeam 9.0 update 1 running on a VMware Esxi 5.1 environment with a Dell TL2000 Tape drive connected with SAS to a Windows Server 2012 physical server. We recently replaced an LTO4 tape drive with an LTO6 tape drive. We tried to do a monthly backup were not able to and are working with Veeam tech to determine cause of the backups stalling and then resuming but no errors reported. In the mean time, we reran the monthly backup on the LTO4 drive and it used several tapes less than usual. Is there a tool or process to verify the contents and the integrity of that backup to determine if it is reliable for us to use?

[Dima P.]

Hi EnerconIT,

Is there a tool or process to verify the contents and the integrity of that backup to determine if it is reliable for us to use?

You can manually start a recovery process to check that backup files are operational.

[EnerconIT]

If we start a manual recovery process and it is not a complete backup how do we stop the process so we do not overwrite the good data that we have?

[Dima P.]

You can restore the backup to the staging repository and then start a file level recovery (there is no need to perform a full VM recovery). If you are using a file to tape backup – just use non-original destination for restore purposes.

[dweide]

There must be a better way to verify backup tapes!

Your proposed procedure requires sufficient space on the staging repository - this is simply not available for the huge fileservers in our backup Scenario.

[Dima P.]

dweide,

Unfortunately, that is the only way to verify that you backup is fully operational in current version.

[dweide]

Hello Dima,

yes, if the subject is to verify "fully operational" backups.

But my point is to verify and check the function of the LTO tape drives.
Any plans for the upcoming releases?

[Dima P.]

dweide,

LTO drive performs an automated verification itself during write operations. When data is written to the tape it is verified by the read heads that are positioned behind the write heads.

[folerx]

All tape drives have this???

[Dima P.]

Can’t say about all, but latest LTO drives have it.

[zoltank]

What's the status on this feature?

I'm in the middle of migrating my tape backup to Veeam, but not having tape verification is a complete show stopper for us. I've had tapes fail verification before, and the thought of an unverified tape failing in a DR situation makes me nauseous.

[veremin]

There is no such feature at the moment.

However, providing you're using combination of SureBackup and latest LTO drives, this should be enough to get required confidence. SureBackup confirms recoverability of backup data, and built-in drive mechanism verifies that data archived to tapes does not change anyhow.

Thanks.

[zoltank]

No, it's not. LTO verification only verifies that what the tape drive received is what's written to the tape, and SureBackup only verifies the backup on the hard drive is good. Tape verification makes sure it was also read from the hard drive correctly and transmitted to the tape drive correctly. It's actually shocking to me that Veeam doesn't have verification and is saying it's not needed.

However, it's my understanding tape verification is expected in the next major release.

[clovella]

Hello All,

Apologies for any rule-breaking or confusion, as this is my first time posting here.

I work for a smaller business with around 200 users and we're looking to streamline our processes and save some space. I've been tasked with documenting the Restore from Backup procedures, but before I begin, I was assigned the sub-task of configuring a way to utilize our Virtual Labs as Backup Repositories for Restore from Tape and Restore from File backup recoveries.

I have been reading, and reading and reading and reading.. so much so that these letters are beginning to run together.

Is this even possible?

Help!

Thanks!

[veremin]

Hi, Cassey,

Welcome to our community forums.

Answering your question, there is no such thing as SureBackup for tape backups. You can either confirm VMs recoverability by restoring VMs from tapes or validate backups before they get archived (recommended approach).

Thanks.

[zoltank]

Any word on the verification for tapes?

It was supposed to be released in Veeam 10, but since Veeam changed their development cycle I'm no longer sure when we might see it. I'm still having to use and maintain Backup Exec due to the lack of tape verification in Veeam.

[acatic]

I am also looking for this functionality. We backup to disk, and then write the vbk's to tape. Tape library is HPE MSL2024. Is there any sort of integrity check that we can refer to?

[Dima P.]

Hi folks,

To put the correct expectations: this feature was never planned for v10 of Backup & Replication, but we are researching the possibilities to provide this functionality in next versions.

My old comment in this thread remain valid: LTO drive performs an automated verification itself after every single write operation. When data is written to the tape it has been verified by the read heads that are positioned behind the write heads. That said you do not have to do the verification right after tape media is being written.

Please let us know, how frequently you plan to verify your tape media? Are you going to load you vaulted tape media to tape library to perform verification or you want to periodically check the tapes that remain in the library.

Thank you in advance!

[zoltank]

I run verify on my tape after every backup.

Backup Exec calculates the checksum of the files as it writes them to the tape, it then also writes that checksum to the tape. Verify then go and recalculates the checksum of the data written to the tape and compares it against the original checksum. Therefore if something was corrupted in transit (something a dodgy SCSI controller and cable can easily do) then the verify will catch it. Yes, it's extra wear on the tape and tape drive, but that's a small price to pay for knowing your tapes are viable.

So far as I know, this is something Veeam's tape support doesn't do but is a critical step in protecting your data with tapes. Until Veeam adds this feature I'd be shirking my responsibilities to my company by using Veeam for our tape backups, regardless of how much I want to get rid of Backup Exec.

[Dima P.]

May I ask if you verify your tape media after it's being written with your current solution or it's basically one time operation? Thank you in advance!

[csydas]

Hi zoltank,

To be clear, the reason I have such an issue with this is that if you have bad data written, the checksum isn't going to help you here. We hit ESXi CBT bugs which wrote bad data, but CRC returned no errors, and rightfully so -- Veeam wrote exactly what CBT fed it, and there was no reason for the software to second guess the data provided to it. I've had this argument with other forum users, but I see such a request as the emperor's clothes; you can verify the data on disk is what got written to tape, but ultimately, you're relying on the fact that what is on disk is legitimate; this is a foregone assumption, imo.,,m ad to argue it is just an exercise more than anything.

Avoiding data corruption for real is just and odd game. You cannot truly prevent it, you can only mitigate the effects.

[zoltank]

May I ask if you verify your tape media after it's being written with your current solution or it's basically one time operation? Thank you in advance!

Yes. We automatically verify every tape that's written.

[zoltank]

Hi zoltank,

To be clear, the reason I have such an issue with this is that if you have bad data written, the checksum isn't going to help you here. We hit ESXi CBT bugs which wrote bad data, but CRC returned no errors, and rightfully so -- Veeam wrote exactly what CBT fed it, and there was no reason for the software to second guess the data provided to it. I've had this argument with other forum users, but I see such a request as the emperor's clothes; you can verify the data on disk is what got written to tape, but ultimately, you're relying on the fact that what is on disk is legitimate; this is a foregone assumption, imo.,,m ad to argue it is just an exercise more than anything.

Avoiding data corruption for real is just and odd game. You cannot truly prevent it, you can only mitigate the effects.

Don't assume because I want tape verify that I haven't already take other measures to ensure the backup data is viable and not corrupt; all my backups automatically go through SureBackup to insure viability. Tape verify is just a tool to insure that data then gets onto the tape correctly and readably.

Tape verify helps protect you from a bad SCSI controller, cable, or tape drive, etc. It does not protect you from writing data that's already bad to the tape, but then it was never meant to do that. Just because there are scenarios it doesn't protect you doesn't mean there aren't scenarios where it does protect you, or that it's not worth having. Really, you're just playing that straw man game here, coming up with things outside the scope of tape verification, then saying it's worthless having it because it doesn't protect you in those situations. Please stop muddying the waters.

[csydas]

Well, I never really said that. What I'm saying is that the payout is not worth the buy-in cost. CRC checks and, well, regular tape maintenance will catch the grand majority of these things because they will present themselves as hardware errors or CRC errors.

I get entirely what you're saying, but it still seems like a strange proposition to me -- this is a situation which ought be flagged by many other safe-guards, and you'd require that these safe-guards also fail in order for there to be a large value-proposition from what you're asking. Is it really worth torturing your hardware for this?

[zoltank]

What do you think tape verify is? It's a CRC check of the data on the tape.

It's not "torturing your hardware", it's just a read and checksum of the data written to the tape. Backups have been doing it for years, and it's caught several bad tapes for me personally. Tape verify is the safeguard against a flaky controller, cable, tape drive, or tape. I find it hard to believe that you're actually arguing against a mechanism to insure your backup data was written correctly to the tape.

[Marco31]

I agree 100% with zoltank. Missing tape verify is the only thing preventing us form using Veeam instead of ArcServe for Backup to Tape. And 10 years using ArcServe for backup to tape tells me you do NOT find out bad tapes when they are written by the drive! I configured ArcServe to verify the backup Jobs after writing and when tapes reach the end of their Lifetime, we often have the "Problem" that Backup ist successfull, but verification fails with "Tape Error - Tape could not be read". So tell me, how can i be sure that my backup is written to tape AND can be successfully restored if such tape problems cannot be detected by Veeam?

[icetrain]

Any update on this matter from Veeam give that this feature has not been added in V10? I know.. it has never been promised.

[Dima P.]

Tape verification functionality is not in v10 but it's planned for later versions. Cheers!

[zoltank]

So we have a major version release and one of the most basic tape functions is still missing?

Come on guys, Backup Exec has had tape verify for over two decades, as has pretty much every other tape backup vendor. Verify is a very basic and very critical feature of tape backup servers.

Looks like I'll need to renew our Backup Exec support for yet another year.

[RobTurk]

Personal experience here, not company party line;

A couple of decades ago, a verify pass was the right thing to do, as tape drives didn't have much sophistication in terms of error handling. A scratch, a dropout or a bad connection would be enough to hit a media error on tape. We're talking QIC, DLT, 4mm DAT and 8mm Exabyte times, and parallel SCSI flatcables with questionable termination creating ugly error patterns. A verify pass was the only end-to-end test to ensure your data was actually on tape, and skipping it was a no-no.

Not anymore. All modern tape drives (LTO, IBM Enterprise) have full read-after-write verification built in. The head consists of a center write section, and two read heads next to it (one for each tape direction). Any write is immediately read back and compared as it is written to tape. If a tape hits a dubious section and the block doesn't properly check out, that same block is written to tape again, a little further down the track. All this is done inside the tape drive, without any involvement from the tape server. This ensures that data that was reported as good on write, will always be readable. Even if the tape were to degrade slightly over time, there's a lot of ECC error correction built in to accommodate for degraded bits, if any. It takes a tremendous amount of abuse to modern tape to induce a read error. Also, the FC and SAS data transports have built-in mechanisms to ensure data transport between the server and the drive is reliable. Tape data integrity surpasses that of harddisks these days.

Verify passes on tape today will add to peace of mind, but frankly it's been years since I have seen a tape throw a media error on read. Media errors on write usually mean a problem with the tape drive and are detected during write. I've spent the last 30 years writing tape drive diagnostics software, and I am very keen on getting my hands on some LTO tapes with real read media errors on then, as I just can't find any.