Veeam 7 tape and verify

[dlieshout@client.nl]

Tape-offload works fine and fast in our environment.

But what about a verify of the tape. I can not find anything about that.
Is it not implemented or will it be added in feature release?

[veremin]

The best idead in this case would be to verify source backup files prior to copying them to tape appliance. In order to do it, you might want to run SureBackup jobs first and only then copy given backup files to tape.

Otherwise, you can restore a backup file from a tape media, import it to a backup console and use it, then, a source for an application group that will be verified within a SureBackup job.

Thanks.

[dlieshout@client.nl]

Yes, but still I do not know if the job is writing rubbish to the tape. Only a restore and test it will verify that the tape was written right.

[tsightler]

There is currently no tape verify function in V7 but I do believe that this feature is being considered for a future version. I remember the days when verify operations were quite common, but modern tape devices have become quite reliable, and many "tape" devices are actually VTL appliances backed by disks so I don't see a lot of customers running verify jobs anymore. Many customers simply don't have to time to get it done. Still, I know some companies have policies that even require verification so I agree it's something I'd like to see added going forward.

[dannythake]

Is this available yet?

I've just found the horror of trying to restore from tape, and finding that they are corrupt

[Dima P.]

Hello Danny,
It’s in the long term feature list and won’t be available in the next release. If you have any issues with restores, it’s worth checking with support team.

[dannythake]

Thanks Dima,

I have to say that this is a real oversight on Veeams part. Offloading backup files to tapes is pretty useless if there is no way to verify that those files are recoverable, as we have just found. Backups files on disk are absolutely fine, but when sent to tape they're unusable. We waited for the tape functionality in v7 before moving from a different vendor to veeam, and this has caused some serious concern in our Backup/DR plans.

We are now at a point where we cant trust the Veeam tape jobs, and will likely need to spend further money to buy a separate solution to do this. Really, really disappointing.

[dlieshout@client.nl]

I agree with Danny. For Veeam the Tape Job is something they did not want to build in but for many customers the Tape Job is a "must have" for there rescue-procedures.
So please Veeam, you do a lot of very good work, place this feature higher at the list.

[veremin]

I have to say that this is a real oversight on Veeams part. Offloading backup files to tapes is pretty useless if there is no way to verify that those files are recoverable, as we have just found.

Did you open a case with our support regarding inability to restore from tapes? Was it confirmed to be drive/library specific problem or something? Thanks.

[dannythake]

Hi Vlad,

Ticket was raised this morning, after further unsuccessful restores from this weekends backup. Logs have been sent

[veremin]

Are only tape backups affected? Or both disk and tape backups seems to be unrecoverable? Can you also post a ticket number, so that we can follow the case?

[dannythake]

Yes, only tape backups affected. Disk backups successfully restore at the production site. Raised ticket 00890328 - I have an engineer calling me tomorrow to discuss further.

For info, the errors we are getting are:

Code: Select all

4/27/2015 1:09:10 PM          Processing started at Monday, April 27, 2015 1:09:09 PM
4/27/2015 1:09:14 PM          Drive 1 (Drive ID: Tape0) locked successfully
4/27/2015 1:09:15 PM          Performing inventory for tape in Drive 1 (Drive ID: Tape0)
4/27/2015 1:09:37 PM          Reading tape Tape 1 catalog
4/27/2015 1:09:38 PM          Moving tape Tape 1 to media pool Imported
4/27/2015 1:10:57 PM Error    Failed to scan tape Tape 1 catalog:  Error: bad allocation
4/27/2015 1:10:57 PM Error    Failed to scan tape Tape 1 Error: bad allocation
4/27/2015 1:10:57 PM          Reading tape metadata
4/27/2015 1:10:57 PM Error    Completed with error at Monday, April 27, 2015 1:10:57 PM

This happens across nearly all tapes. Tapes were new last near so nowhere near there MTBF. At this stage i cannot rule out an actual Drive problem, but i will know either way when i do a test restore in the morning at the production site. We only started getting this problem at v8, of which we had to apply some custom dll's in order to get it to backup correctly. We did multiple restore on v7 without issue. Again, it may not be relevant, but thats my observation.

[veremin]

Your problem might not be necessarily related to backup unrecoverability, but to allocation of required amount of memory during catalogue. We have a regkey forcing start of 64-bit component on tape server; it might be helpful in your case. I'll try to approach your engineer and share the said key with him; let's see whether or not it applies to your problem.

Thanks.

[dannythake]

Vlad,

Thanks for the info. Now here's something interesting, I managed to find a backup tape that was created by v7 around 12 months ago, stuck it in the drive at the DR site and it restored successfully!

Now the strange part, if I put the most recent tapes into our production multi-loader (the same machine that put them onto tape), they catalog correctly and restore correctly. My gut feeling is that this is actually due to the custom dll we were provided to resolve a different tape issue.

I have a call with engineer in 2 hours, so will mention patch 2 as suggested. For the record, its still a +1 on the Verify feature in a future release

[veremin]

We've worked in close collaboration with your support engineer and our QA and dev teams to identify the root cause of the experienced behaviour.

We were able to find out the problem, and are going to implement a private fix for you. For now, be aware to update to Update 2, as private fix when it's ready will be applicable to that version.

Thanks.

[dannythake]

Thanks Vlad,

To confirm, I need to update to Patch 2 before I install the private fix?

Thanks

[veremin]

Yep, you need to install update 2 first.

As to the private fix, it's being developed at the moment. You will be provided with it, when it's ready.

Thanks.

[dannythake]

Thanks for your assistance Vlad

[veremin]

Hi, Danny, as far as I'm aware, the private fix has been just developed, so you should be provided with it pretty soon. Thanks.

[dannythake]

Hi Vlad,

Tested fix, still cannot complete a catalog:

5/6/2015 4:21:38 PM Processing started at Wednesday, May 06, 2015 4:21:34 PM
5/6/2015 4:21:46 PM Drive 1 (Drive ID: Tape0) locked successfully
5/6/2015 4:22:02 PM Performing inventory for tape in Drive 1 (Drive ID: Tape0)
5/6/2015 4:24:44 PM Reading tape Tape 3 catalog
5/6/2015 4:24:51 PM Moving tape Tape 3 to media pool Imported
5/6/2015 4:24:51 PM Reading tape metadata
5/6/2015 4:25:04 PM Error: No connection could be made because the target machine actively refused it 192.168.100.12:443 <------------This is our enterprise Manager Server
5/6/2015 4:25:04 PM Error Completed with error at Wednesday, May 06, 2015 4:25:04 PM

Not entirely sure why is needs to communicate with a server at our other site (that would be unavailable if we were in DR situation!)

Support Engineer notified.

[Dima P.]

Danny,
It looks like a separate issue – could you, please, attach a new portion of logs to the existing case? I’ll ask dev team to take a look once done.

[dannythake]

Hi Dmitry,

It was indeed a separate issue from the original issue. Unfortunately we didn't find the second issue until the private fix was installed. I had an Engineer run a cleanup of the SQL Server DB, which cleared the problem. Looks like some stale records of an old vCenter installation were causing it.

I have managed to now restore a very small VM successfully, but will test further in the morning with our Production Backup Set

[Dima P.]

Danny,

Glad to hear you are up and running now. Keep us posted if you face any issue or questions.

[icetrain]

Do I understand you correctly, that there is no verification available for backup to tape jobs?

[veremin]

Correct. However, most of the tape devices have quite comprehensive write verification mechanisms. So, you should be good, as long as you verify source backup data via SureBackup, before it gets archived to tapes. Thanks.

[icetrain]

This means there are no definite plans to support this software wise?

[veremin]

Not at the moment.

[icetrain]

Maybe I have missed something, but I still don't understand why you would skip adding such a feature.

There is always the possibility of a bit error, and even NRE - even on tape. How checking the source backup would prevent those NRE happening on tape is out of my mind right now.

[veremin]

Since we don't have infinite development and QA resources, we always have to skip this or that particular functionality. Less demanded features are prone to such destiny. Thanks.

[pshute]

Are there still no plans to implement this? The first tape drive I ever bought, back in 1987, was advertised as having a built in read after write feature, but the backups were still often unreadable, probably because the backup software was getting something wrong. Built in tape drive verification only makes sure it's backed up what it's been told to backup. It doesn't know if it's correct or not.

In those days we were backing up directly from filesystem to tape, so the verification had to be done overnight before people started work in the morning, changing files and causing them fail the software verification. These days we backup to a single file on disk, and the backup file is available all the next day for comparison with the file on the tape. It's surely not a difficult feature to implement. It's just reading from tape while comparing to a file on disk. It doesn't need to do a full restore first, it could compare a few GB at a time and discard the verified data as it goes to save disk space.