Using tape as a backup target
dlieshout@client.nl
Service Provider
Posts: 43
Liked: 14 times
Joined: Dec 15, 2009 12:41 pm
Full Name: DaLi
Contact:

Veeam 7 tape and verify

Post by dlieshout@client.nl » 3 people like this post

Tape-offload works fine and fast in our environment.

But what about a verify of the tape. I can not find anything about that.
Is it not implemented or will it be added in feature release?

veremin
Product Manager
Posts: 18786
Liked: 1893 times
Joined: Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin
Contact:

Re: Veeam 7 tape and verify

Post by veremin »

The best idead in this case would be to verify source backup files prior to copying them to tape appliance. In order to do it, you might want to run SureBackup jobs first and only then copy given backup files to tape.

Otherwise, you can restore a backup file from a tape media, import it to a backup console and use it, then, a source for an application group that will be verified within a SureBackup job.

Thanks.

dlieshout@client.nl
Service Provider
Posts: 43
Liked: 14 times
Joined: Dec 15, 2009 12:41 pm
Full Name: DaLi
Contact:

Re: Veeam 7 tape and verify

Post by dlieshout@client.nl » 1 person likes this post

Yes, but still I do not know if the job is writing rubbish to the tape. Only a restore and test it will verify that the tape was written right.

tsightler
VP, Product Management
Posts: 5901
Liked: 2756 times
Joined: Jun 05, 2009 12:57 pm
Full Name: Tom Sightler
Contact:

Re: Veeam 7 tape and verify

Post by tsightler »

There is currently no tape verify function in V7 but I do believe that this feature is being considered for a future version. I remember the days when verify operations were quite common, but modern tape devices have become quite reliable, and many "tape" devices are actually VTL appliances backed by disks so I don't see a lot of customers running verify jobs anymore. Many customers simply don't have to time to get it done. Still, I know some companies have policies that even require verification so I agree it's something I'd like to see added going forward.

dannythake
Enthusiast
Posts: 39
Liked: 10 times
Joined: May 28, 2013 8:41 am
Full Name: Danny Thake
Contact:

Re: Veeam 7 tape and verify

Post by dannythake »

Is this available yet?

I've just found the horror of trying to restore from tape, and finding that they are corrupt

Dima P.
Product Manager
Posts: 12938
Liked: 1231 times
Joined: Feb 04, 2013 2:07 pm
Full Name: Dmitry Popov
Location: Prague
Contact:

Re: Veeam 7 tape and verify

Post by Dima P. »

Hello Danny,
It’s in the long term feature list and won’t be available in the next release. If you have any issues with restores, it’s worth checking with support team.

dannythake
Enthusiast
Posts: 39
Liked: 10 times
Joined: May 28, 2013 8:41 am
Full Name: Danny Thake
Contact:

Re: Veeam 7 tape and verify

Post by dannythake »

Thanks Dima,

I have to say that this is a real oversight on Veeams part. Offloading backup files to tapes is pretty useless if there is no way to verify that those files are recoverable, as we have just found. Backups files on disk are absolutely fine, but when sent to tape they're unusable. We waited for the tape functionality in v7 before moving from a different vendor to veeam, and this has caused some serious concern in our Backup/DR plans.

We are now at a point where we cant trust the Veeam tape jobs, and will likely need to spend further money to buy a separate solution to do this. Really, really disappointing.

dlieshout@client.nl
Service Provider
Posts: 43
Liked: 14 times
Joined: Dec 15, 2009 12:41 pm
Full Name: DaLi
Contact:

Re: Veeam 7 tape and verify

Post by dlieshout@client.nl » 1 person likes this post

I agree with Danny. For Veeam the Tape Job is something they did not want to build in but for many customers the Tape Job is a "must have" for there rescue-procedures.
So please Veeam, you do a lot of very good work, place this feature higher at the list.

veremin
Product Manager
Posts: 18786
Liked: 1893 times
Joined: Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin
Contact:

Re: Veeam 7 tape and verify

Post by veremin »

I have to say that this is a real oversight on Veeams part. Offloading backup files to tapes is pretty useless if there is no way to verify that those files are recoverable, as we have just found.
Did you open a case with our support regarding inability to restore from tapes? Was it confirmed to be drive/library specific problem or something? Thanks.

dannythake
Enthusiast
Posts: 39
Liked: 10 times
Joined: May 28, 2013 8:41 am
Full Name: Danny Thake
Contact:

Re: Veeam 7 tape and verify

Post by dannythake »

Hi Vlad,

Ticket was raised this morning, after further unsuccessful restores from this weekends backup. Logs have been sent

veremin
Product Manager
Posts: 18786
Liked: 1893 times
Joined: Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin
Contact:

Re: Veeam 7 tape and verify

Post by veremin »

Are only tape backups affected? Or both disk and tape backups seems to be unrecoverable? Can you also post a ticket number, so that we can follow the case?

dannythake
Enthusiast
Posts: 39
Liked: 10 times
Joined: May 28, 2013 8:41 am
Full Name: Danny Thake
Contact:

Re: Veeam 7 tape and verify

Post by dannythake »

Yes, only tape backups affected. Disk backups successfully restore at the production site. Raised ticket 00890328 - I have an engineer calling me tomorrow to discuss further.

For info, the errors we are getting are:

Code: Select all

4/27/2015 1:09:10 PM          Processing started at Monday, April 27, 2015 1:09:09 PM
4/27/2015 1:09:14 PM          Drive 1 (Drive ID: Tape0) locked successfully
4/27/2015 1:09:15 PM          Performing inventory for tape in Drive 1 (Drive ID: Tape0)
4/27/2015 1:09:37 PM          Reading tape Tape 1 catalog
4/27/2015 1:09:38 PM          Moving tape Tape 1 to media pool Imported
4/27/2015 1:10:57 PM Error    Failed to scan tape Tape 1 catalog:  Error: bad allocation
4/27/2015 1:10:57 PM Error    Failed to scan tape Tape 1 Error: bad allocation
4/27/2015 1:10:57 PM          Reading tape metadata
4/27/2015 1:10:57 PM Error    Completed with error at Monday, April 27, 2015 1:10:57 PM
This happens across nearly all tapes. Tapes were new last near so nowhere near there MTBF. At this stage i cannot rule out an actual Drive problem, but i will know either way when i do a test restore in the morning at the production site. We only started getting this problem at v8, of which we had to apply some custom dll's in order to get it to backup correctly. We did multiple restore on v7 without issue. Again, it may not be relevant, but thats my observation.

veremin
Product Manager
Posts: 18786
Liked: 1893 times
Joined: Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin
Contact:

Re: Veeam 7 tape and verify

Post by veremin »

Your problem might not be necessarily related to backup unrecoverability, but to allocation of required amount of memory during catalogue. We have a regkey forcing start of 64-bit component on tape server; it might be helpful in your case. I'll try to approach your engineer and share the said key with him; let's see whether or not it applies to your problem.

Thanks.

dannythake
Enthusiast
Posts: 39
Liked: 10 times
Joined: May 28, 2013 8:41 am
Full Name: Danny Thake
Contact:

Re: Veeam 7 tape and verify

Post by dannythake »

Vlad,

Thanks for the info. Now here's something interesting, I managed to find a backup tape that was created by v7 around 12 months ago, stuck it in the drive at the DR site and it restored successfully!

Now the strange part, if I put the most recent tapes into our production multi-loader (the same machine that put them onto tape), they catalog correctly and restore correctly. My gut feeling is that this is actually due to the custom dll we were provided to resolve a different tape issue.

I have a call with engineer in 2 hours, so will mention patch 2 as suggested. For the record, its still a +1 on the Verify feature in a future release 8)

veremin
Product Manager
Posts: 18786
Liked: 1893 times
Joined: Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin
Contact:

Re: Veeam 7 tape and verify

Post by veremin »

We've worked in close collaboration with your support engineer and our QA and dev teams to identify the root cause of the experienced behaviour.

We were able to find out the problem, and are going to implement a private fix for you. For now, be aware to update to Update 2, as private fix when it's ready will be applicable to that version.

Thanks.

dannythake
Enthusiast
Posts: 39
Liked: 10 times
Joined: May 28, 2013 8:41 am
Full Name: Danny Thake
Contact:

Re: Veeam 7 tape and verify

Post by dannythake »

Thanks Vlad,

To confirm, I need to update to Patch 2 before I install the private fix?

Thanks

veremin
Product Manager
Posts: 18786
Liked: 1893 times
Joined: Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin
Contact:

Re: Veeam 7 tape and verify

Post by veremin »

Yep, you need to install update 2 first.

As to the private fix, it's being developed at the moment. You will be provided with it, when it's ready.

Thanks.

dannythake
Enthusiast
Posts: 39
Liked: 10 times
Joined: May 28, 2013 8:41 am
Full Name: Danny Thake
Contact:

Re: Veeam 7 tape and verify

Post by dannythake » 3 people like this post

Thanks for your assistance Vlad

veremin
Product Manager
Posts: 18786
Liked: 1893 times
Joined: Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin
Contact:

Re: Veeam 7 tape and verify

Post by veremin »

Hi, Danny, as far as I'm aware, the private fix has been just developed, so you should be provided with it pretty soon. Thanks.

dannythake
Enthusiast
Posts: 39
Liked: 10 times
Joined: May 28, 2013 8:41 am
Full Name: Danny Thake
Contact:

Re: Veeam 7 tape and verify

Post by dannythake »

Hi Vlad,

Tested fix, still cannot complete a catalog:

5/6/2015 4:21:38 PM Processing started at Wednesday, May 06, 2015 4:21:34 PM
5/6/2015 4:21:46 PM Drive 1 (Drive ID: Tape0) locked successfully
5/6/2015 4:22:02 PM Performing inventory for tape in Drive 1 (Drive ID: Tape0)
5/6/2015 4:24:44 PM Reading tape Tape 3 catalog
5/6/2015 4:24:51 PM Moving tape Tape 3 to media pool Imported
5/6/2015 4:24:51 PM Reading tape metadata
5/6/2015 4:25:04 PM Error: No connection could be made because the target machine actively refused it 192.168.100.12:443 <------------This is our enterprise Manager Server
5/6/2015 4:25:04 PM Error Completed with error at Wednesday, May 06, 2015 4:25:04 PM

Not entirely sure why is needs to communicate with a server at our other site (that would be unavailable if we were in DR situation!)

Support Engineer notified.

Dima P.
Product Manager
Posts: 12938
Liked: 1231 times
Joined: Feb 04, 2013 2:07 pm
Full Name: Dmitry Popov
Location: Prague
Contact:

Re: Veeam 7 tape and verify

Post by Dima P. »

Danny,
It looks like a separate issue – could you, please, attach a new portion of logs to the existing case? I’ll ask dev team to take a look once done.

dannythake
Enthusiast
Posts: 39
Liked: 10 times
Joined: May 28, 2013 8:41 am
Full Name: Danny Thake
Contact:

Re: Veeam 7 tape and verify

Post by dannythake » 1 person likes this post

Hi Dmitry,

It was indeed a separate issue from the original issue. Unfortunately we didn't find the second issue until the private fix was installed. I had an Engineer run a cleanup of the SQL Server DB, which cleared the problem. Looks like some stale records of an old vCenter installation were causing it.

I have managed to now restore a very small VM successfully, but will test further in the morning with our Production Backup Set

Dima P.
Product Manager
Posts: 12938
Liked: 1231 times
Joined: Feb 04, 2013 2:07 pm
Full Name: Dmitry Popov
Location: Prague
Contact:

Re: Veeam 7 tape and verify

Post by Dima P. »

Danny,

Glad to hear you are up and running now. Keep us posted if you face any issue or questions.

icetrain
Influencer
Posts: 15
Liked: 2 times
Joined: Sep 05, 2017 9:02 am
Full Name: ECE
Contact:

Re: Veeam 7 tape and verify

Post by icetrain »

Do I understand you correctly, that there is no verification available for backup to tape jobs?

veremin
Product Manager
Posts: 18786
Liked: 1893 times
Joined: Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin
Contact:

Re: Veeam 7 tape and verify

Post by veremin »

Correct. However, most of the tape devices have quite comprehensive write verification mechanisms. So, you should be good, as long as you verify source backup data via SureBackup, before it gets archived to tapes. Thanks.

icetrain
Influencer
Posts: 15
Liked: 2 times
Joined: Sep 05, 2017 9:02 am
Full Name: ECE
Contact:

Re: Veeam 7 tape and verify

Post by icetrain »

This means there are no definite plans to support this software wise?

veremin
Product Manager
Posts: 18786
Liked: 1893 times
Joined: Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin
Contact:

Re: Veeam 7 tape and verify

Post by veremin »

Not at the moment.

icetrain
Influencer
Posts: 15
Liked: 2 times
Joined: Sep 05, 2017 9:02 am
Full Name: ECE
Contact:

Re: Veeam 7 tape and verify

Post by icetrain » 1 person likes this post

Maybe I have missed something, but I still don't understand why you would skip adding such a feature.

There is always the possibility of a bit error, and even NRE - even on tape. How checking the source backup would prevent those NRE happening on tape is out of my mind right now.

veremin
Product Manager
Posts: 18786
Liked: 1893 times
Joined: Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin
Contact:

Re: Veeam 7 tape and verify

Post by veremin »

Since we don't have infinite development and QA resources, we always have to skip this or that particular functionality. Less demanded features are prone to such destiny. Thanks.

pshute
Expert
Posts: 254
Liked: 14 times
Joined: Nov 23, 2015 10:56 pm
Full Name: Peter Shute
Contact:

Re: Veeam 7 tape and verify

Post by pshute » 1 person likes this post

Are there still no plans to implement this? The first tape drive I ever bought, back in 1987, was advertised as having a built in read after write feature, but the backups were still often unreadable, probably because the backup software was getting something wrong. Built in tape drive verification only makes sure it's backed up what it's been told to backup. It doesn't know if it's correct or not.

In those days we were backing up directly from filesystem to tape, so the verification had to be done overnight before people started work in the morning, changing files and causing them fail the software verification. These days we backup to a single file on disk, and the backup file is available all the next day for comparison with the file on the tape. It's surely not a difficult feature to implement. It's just reading from tape while comparing to a file on disk. It doesn't need to do a full restore first, it could compare a few GB at a time and discard the verified data as it goes to save disk space.

Post Reply

Who is online

Users browsing this forum: No registered users and 6 guests