Using tape as a backup target
EnerconIT
Influencer
Posts: 13
Liked: never
Joined: Dec 02, 2014 3:21 pm
Full Name: Enercon Engineering
Contact:

How to verify tape backup content and integrity

Post by EnerconIT »

Case ID:01786446

We have Veeam 9.0 update 1 running on a VMware Esxi 5.1 environment with a Dell TL2000 Tape drive connected with SAS to a Windows Server 2012 physical server. We recently replaced an LTO4 tape drive with an LTO6 tape drive. We tried to do a monthly backup were not able to and are working with Veeam tech to determine cause of the backups stalling and then resuming but no errors reported. In the mean time, we reran the monthly backup on the LTO4 drive and it used several tapes less than usual. Is there a tool or process to verify the contents and the integrity of that backup to determine if it is reliable for us to use?

Dima P.
Product Manager
Posts: 12501
Liked: 1166 times
Joined: Feb 04, 2013 2:07 pm
Full Name: Dmitry Popov
Location: Prague
Contact:

Re: How to verify tape backup content and integrity

Post by Dima P. »

Hi EnerconIT,
Is there a tool or process to verify the contents and the integrity of that backup to determine if it is reliable for us to use?
You can manually start a recovery process to check that backup files are operational.

EnerconIT
Influencer
Posts: 13
Liked: never
Joined: Dec 02, 2014 3:21 pm
Full Name: Enercon Engineering
Contact:

Re: How to verify tape backup content and integrity

Post by EnerconIT »

If we start a manual recovery process and it is not a complete backup how do we stop the process so we do not overwrite the good data that we have?

Dima P.
Product Manager
Posts: 12501
Liked: 1166 times
Joined: Feb 04, 2013 2:07 pm
Full Name: Dmitry Popov
Location: Prague
Contact:

Re: How to verify tape backup content and integrity

Post by Dima P. »

You can restore the backup to the staging repository and then start a file level recovery (there is no need to perform a full VM recovery). If you are using a file to tape backup – just use non-original destination for restore purposes.

dweide
Enthusiast
Posts: 35
Liked: 6 times
Joined: Mar 29, 2012 1:57 pm
Full Name: D. Weide
Contact:

Re: How to verify tape backup content and integrity

Post by dweide »

There must be a better way to verify backup tapes!

Your proposed procedure requires sufficient space on the staging repository - this is simply not available for the huge fileservers in our backup Scenario.

Dima P.
Product Manager
Posts: 12501
Liked: 1166 times
Joined: Feb 04, 2013 2:07 pm
Full Name: Dmitry Popov
Location: Prague
Contact:

Re: How to verify tape backup content and integrity

Post by Dima P. »

dweide,

Unfortunately, that is the only way to verify that you backup is fully operational in current version.

dweide
Enthusiast
Posts: 35
Liked: 6 times
Joined: Mar 29, 2012 1:57 pm
Full Name: D. Weide
Contact:

Re: How to verify tape backup content and integrity

Post by dweide »

Hello Dima,

yes, if the subject is to verify "fully operational" backups.

But my point is to verify and check the function of the LTO tape drives.
Any plans for the upcoming releases?

Dima P.
Product Manager
Posts: 12501
Liked: 1166 times
Joined: Feb 04, 2013 2:07 pm
Full Name: Dmitry Popov
Location: Prague
Contact:

Re: How to verify tape backup content and integrity

Post by Dima P. »

dweide,

LTO drive performs an automated verification itself during write operations. When data is written to the tape it is verified by the read heads that are positioned behind the write heads.

folerx
Expert
Posts: 107
Liked: 8 times
Joined: Jun 22, 2016 9:47 pm
Full Name: Daniel Kaiser
Contact:

Re: How to verify tape backup content and integrity

Post by folerx »

All tape drives have this???

Dima P.
Product Manager
Posts: 12501
Liked: 1166 times
Joined: Feb 04, 2013 2:07 pm
Full Name: Dmitry Popov
Location: Prague
Contact:

Re: How to verify tape backup content and integrity

Post by Dima P. »

Can’t say about all, but latest LTO drives have it.

zoltank
Expert
Posts: 229
Liked: 41 times
Joined: Feb 18, 2011 5:01 pm
Contact:

[MERGED] Veeam 7 tape and verify

Post by zoltank »

What's the status on this feature?

I'm in the middle of migrating my tape backup to Veeam, but not having tape verification is a complete show stopper for us. I've had tapes fail verification before, and the thought of an unverified tape failing in a DR situation makes me nauseous.

veremin
Product Manager
Posts: 18413
Liked: 1819 times
Joined: Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin
Contact:

Re: How to verify tape backup content and integrity

Post by veremin »

There is no such feature at the moment.

However, providing you're using combination of SureBackup and latest LTO drives, this should be enough to get required confidence. SureBackup confirms recoverability of backup data, and built-in drive mechanism verifies that data archived to tapes does not change anyhow.

Thanks.

zoltank
Expert
Posts: 229
Liked: 41 times
Joined: Feb 18, 2011 5:01 pm
Contact:

Re: How to verify tape backup content and integrity

Post by zoltank »

No, it's not. LTO verification only verifies that what the tape drive received is what's written to the tape, and SureBackup only verifies the backup on the hard drive is good. Tape verification makes sure it was also read from the hard drive correctly and transmitted to the tape drive correctly. It's actually shocking to me that Veeam doesn't have verification and is saying it's not needed.

However, it's my understanding tape verification is expected in the next major release.

clovella
Lurker
Posts: 1
Liked: never
Joined: Jun 21, 2017 4:55 pm
Full Name: Casey Lovell
Contact:

[MERGED] Restore Tape Backup to Virtual Lab?

Post by clovella »

Hello All,

Apologies for any rule-breaking or confusion, as this is my first time posting here.

I work for a smaller business with around 200 users and we're looking to streamline our processes and save some space. I've been tasked with documenting the Restore from Backup procedures, but before I begin, I was assigned the sub-task of configuring a way to utilize our Virtual Labs as Backup Repositories for Restore from Tape and Restore from File backup recoveries.

I have been reading, and reading and reading and reading.. so much so that these letters are beginning to run together.

Is this even possible?

Help!

Thanks!

veremin
Product Manager
Posts: 18413
Liked: 1819 times
Joined: Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin
Contact:

Re: How to verify tape backup content and integrity

Post by veremin »

Hi, Cassey,

Welcome to our community forums.

Answering your question, there is no such thing as SureBackup for tape backups. You can either confirm VMs recoverability by restoring VMs from tapes or validate backups before they get archived (recommended approach).

Thanks.

zoltank
Expert
Posts: 229
Liked: 41 times
Joined: Feb 18, 2011 5:01 pm
Contact:

Re: How to verify tape backup content and integrity

Post by zoltank »

Any word on the verification for tapes?

It was supposed to be released in Veeam 10, but since Veeam changed their development cycle I'm no longer sure when we might see it. I'm still having to use and maintain Backup Exec due to the lack of tape verification in Veeam.

acatic
Enthusiast
Posts: 39
Liked: 3 times
Joined: Oct 07, 2013 3:08 pm
Contact:

Re: How to verify tape backup content and integrity

Post by acatic »

I am also looking for this functionality. We backup to disk, and then write the vbk's to tape. Tape library is HPE MSL2024. Is there any sort of integrity check that we can refer to?

Dima P.
Product Manager
Posts: 12501
Liked: 1166 times
Joined: Feb 04, 2013 2:07 pm
Full Name: Dmitry Popov
Location: Prague
Contact:

Re: How to verify tape backup content and integrity

Post by Dima P. »

Hi folks,

To put the correct expectations: this feature was never planned for v10 of Backup & Replication, but we are researching the possibilities to provide this functionality in next versions.

My old comment in this thread remain valid: LTO drive performs an automated verification itself after every single write operation. When data is written to the tape it has been verified by the read heads that are positioned behind the write heads. That said you do not have to do the verification right after tape media is being written.

Please let us know, how frequently you plan to verify your tape media? Are you going to load you vaulted tape media to tape library to perform verification or you want to periodically check the tapes that remain in the library.

Thank you in advance!

zoltank
Expert
Posts: 229
Liked: 41 times
Joined: Feb 18, 2011 5:01 pm
Contact:

Re: How to verify tape backup content and integrity

Post by zoltank »

I run verify on my tape after every backup.

Backup Exec calculates the checksum of the files as it writes them to the tape, it then also writes that checksum to the tape. Verify then go and recalculates the checksum of the data written to the tape and compares it against the original checksum. Therefore if something was corrupted in transit (something a dodgy SCSI controller and cable can easily do) then the verify will catch it. Yes, it's extra wear on the tape and tape drive, but that's a small price to pay for knowing your tapes are viable.

So far as I know, this is something Veeam's tape support doesn't do but is a critical step in protecting your data with tapes. Until Veeam adds this feature I'd be shirking my responsibilities to my company by using Veeam for our tape backups, regardless of how much I want to get rid of Backup Exec.

Dima P.
Product Manager
Posts: 12501
Liked: 1166 times
Joined: Feb 04, 2013 2:07 pm
Full Name: Dmitry Popov
Location: Prague
Contact:

Re: How to verify tape backup content and integrity

Post by Dima P. »

May I ask if you verify your tape media after it's being written with your current solution or it's basically one time operation? Thank you in advance!

csydas
Expert
Posts: 193
Liked: 47 times
Joined: Jan 16, 2018 5:14 pm
Full Name: Harvey Carel
Contact:

Re: How to verify tape backup content and integrity

Post by csydas »

Hi zoltank,

To be clear, the reason I have such an issue with this is that if you have bad data written, the checksum isn't going to help you here. We hit ESXi CBT bugs which wrote bad data, but CRC returned no errors, and rightfully so -- Veeam wrote exactly what CBT fed it, and there was no reason for the software to second guess the data provided to it. I've had this argument with other forum users, but I see such a request as the emperor's clothes; you can verify the data on disk is what got written to tape, but ultimately, you're relying on the fact that what is on disk is legitimate; this is a foregone assumption, imo.,,m ad to argue it is just an exercise more than anything.

Avoiding data corruption for real is just and odd game. You cannot truly prevent it, you can only mitigate the effects.

zoltank
Expert
Posts: 229
Liked: 41 times
Joined: Feb 18, 2011 5:01 pm
Contact:

Re: How to verify tape backup content and integrity

Post by zoltank »

Dima P. wrote: Apr 09, 2019 6:56 pm May I ask if you verify your tape media after it's being written with your current solution or it's basically one time operation? Thank you in advance!
Yes. We automatically verify every tape that's written.

zoltank
Expert
Posts: 229
Liked: 41 times
Joined: Feb 18, 2011 5:01 pm
Contact:

Re: How to verify tape backup content and integrity

Post by zoltank »

csydas wrote: Apr 10, 2019 9:56 pm Hi zoltank,

To be clear, the reason I have such an issue with this is that if you have bad data written, the checksum isn't going to help you here. We hit ESXi CBT bugs which wrote bad data, but CRC returned no errors, and rightfully so -- Veeam wrote exactly what CBT fed it, and there was no reason for the software to second guess the data provided to it. I've had this argument with other forum users, but I see such a request as the emperor's clothes; you can verify the data on disk is what got written to tape, but ultimately, you're relying on the fact that what is on disk is legitimate; this is a foregone assumption, imo.,,m ad to argue it is just an exercise more than anything.

Avoiding data corruption for real is just and odd game. You cannot truly prevent it, you can only mitigate the effects.
Don't assume because I want tape verify that I haven't already take other measures to ensure the backup data is viable and not corrupt; all my backups automatically go through SureBackup to insure viability. Tape verify is just a tool to insure that data then gets onto the tape correctly and readably.

Tape verify helps protect you from a bad SCSI controller, cable, or tape drive, etc. It does not protect you from writing data that's already bad to the tape, but then it was never meant to do that. Just because there are scenarios it doesn't protect you doesn't mean there aren't scenarios where it does protect you, or that it's not worth having. Really, you're just playing that straw man game here, coming up with things outside the scope of tape verification, then saying it's worthless having it because it doesn't protect you in those situations. Please stop muddying the waters.

csydas
Expert
Posts: 193
Liked: 47 times
Joined: Jan 16, 2018 5:14 pm
Full Name: Harvey Carel
Contact:

Re: How to verify tape backup content and integrity

Post by csydas »

Well, I never really said that. What I'm saying is that the payout is not worth the buy-in cost. CRC checks and, well, regular tape maintenance will catch the grand majority of these things because they will present themselves as hardware errors or CRC errors.

I get entirely what you're saying, but it still seems like a strange proposition to me -- this is a situation which ought be flagged by many other safe-guards, and you'd require that these safe-guards also fail in order for there to be a large value-proposition from what you're asking. Is it really worth torturing your hardware for this?

zoltank
Expert
Posts: 229
Liked: 41 times
Joined: Feb 18, 2011 5:01 pm
Contact:

Re: How to verify tape backup content and integrity

Post by zoltank »

What do you think tape verify is? It's a CRC check of the data on the tape.

It's not "torturing your hardware", it's just a read and checksum of the data written to the tape. Backups have been doing it for years, and it's caught several bad tapes for me personally. Tape verify is the safeguard against a flaky controller, cable, tape drive, or tape. I find it hard to believe that you're actually arguing against a mechanism to insure your backup data was written correctly to the tape.

Marco31
Novice
Posts: 7
Liked: never
Joined: Jun 01, 2017 11:01 am
Contact:

Re: How to verify tape backup content and integrity

Post by Marco31 »

I agree 100% with zoltank. Missing tape verify is the only thing preventing us form using Veeam instead of ArcServe for Backup to Tape. And 10 years using ArcServe for backup to tape tells me you do NOT find out bad tapes when they are written by the drive! I configured ArcServe to verify the backup Jobs after writing and when tapes reach the end of their Lifetime, we often have the "Problem" that Backup ist successfull, but verification fails with "Tape Error - Tape could not be read". So tell me, how can i be sure that my backup is written to tape AND can be successfully restored if such tape problems cannot be detected by Veeam?

icetrain
Influencer
Posts: 15
Liked: 2 times
Joined: Sep 05, 2017 9:02 am
Full Name: ECE
Contact:

Re: How to verify tape backup content and integrity

Post by icetrain »

Any update on this matter from Veeam give that this feature has not been added in V10? I know.. it has never been promised.

Dima P.
Product Manager
Posts: 12501
Liked: 1166 times
Joined: Feb 04, 2013 2:07 pm
Full Name: Dmitry Popov
Location: Prague
Contact:

Re: How to verify tape backup content and integrity

Post by Dima P. » 1 person likes this post

Tape verification functionality is not in v10 but it's planned for later versions. Cheers!

zoltank
Expert
Posts: 229
Liked: 41 times
Joined: Feb 18, 2011 5:01 pm
Contact:

Re: How to verify tape backup content and integrity

Post by zoltank »

So we have a major version release and one of the most basic tape functions is still missing?

Come on guys, Backup Exec has had tape verify for over two decades, as has pretty much every other tape backup vendor. Verify is a very basic and very critical feature of tape backup servers.

Looks like I'll need to renew our Backup Exec support for yet another year.

RobTurk
Veeam Software
Posts: 120
Liked: 20 times
Joined: Aug 07, 2019 10:05 am
Full Name: Rob Turk
Contact:

Re: How to verify tape backup content and integrity

Post by RobTurk »

Personal experience here, not company party line;

A couple of decades ago, a verify pass was the right thing to do, as tape drives didn't have much sophistication in terms of error handling. A scratch, a dropout or a bad connection would be enough to hit a media error on tape. We're talking QIC, DLT, 4mm DAT and 8mm Exabyte times, and parallel SCSI flatcables with questionable termination creating ugly error patterns. A verify pass was the only end-to-end test to ensure your data was actually on tape, and skipping it was a no-no.

Not anymore. All modern tape drives (LTO, IBM Enterprise) have full read-after-write verification built in. The head consists of a center write section, and two read heads next to it (one for each tape direction). Any write is immediately read back and compared as it is written to tape. If a tape hits a dubious section and the block doesn't properly check out, that same block is written to tape again, a little further down the track. All this is done inside the tape drive, without any involvement from the tape server. This ensures that data that was reported as good on write, will always be readable. Even if the tape were to degrade slightly over time, there's a lot of ECC error correction built in to accommodate for degraded bits, if any. It takes a tremendous amount of abuse to modern tape to induce a read error. Also, the FC and SAS data transports have built-in mechanisms to ensure data transport between the server and the drive is reliable. Tape data integrity surpasses that of harddisks these days.

Verify passes on tape today will add to peace of mind, but frankly it's been years since I have seen a tape throw a media error on read. Media errors on write usually mean a problem with the tape drive and are detected during write. I've spent the last 30 years writing tape drive diagnostics software, and I am very keen on getting my hands on some LTO tapes with real read media errors on then, as I just can't find any.

Post Reply

Who is online

Users browsing this forum: No registered users and 7 guests