Discussions related to exporting backups to tape and backing up directly to tape.
mazur1000
Influencer
Posts: 23
Liked: 1 time
Joined: Feb 20, 2014 6:54 am
Full Name: alex mazur
Contact:

restore large files from tape

Post by mazur1000 »

veeam 7 patch 3
when restoring large files (3-10 TB) from tape (ibm 3200 lto 6) recovered file size turns out not to be the size of the original (differs by about 100-200 KB).
Recovered files gets corrupted.
And the size of the recovered file becomes invalid in the last stage recovery
Before installing patch 3 was a mistake: Script execution timed out
veremin
Product Manager
Posts: 20270
Liked: 2252 times
Joined: Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin
Contact:

Re: restore large files from tape

Post by veremin »

Hi, Alex,

What type of restore you're using? Also, does the given backup file reside on one tape or on many of them?

Thanks.
mazur1000
Influencer
Posts: 23
Liked: 1 time
Joined: Feb 20, 2014 6:54 am
Full Name: alex mazur
Contact:

Re: restore large files from tape

Post by mazur1000 »

I use RESTORE FILE FROM TAPE
Tried to recover. Vbk 6TB file and a file created using acronis 3 TB.
In the former case, three backup tapes held in the second - 2 tape.
veremin
Product Manager
Posts: 20270
Liked: 2252 times
Joined: Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin
Contact:

Re: restore large files from tape

Post by veremin »

So, both files were shown as corrupted, right?

Anyway, it seems that deep log investigation is required here, so, please, open a ticket with our support team and let them investigate it directly.

Also, once opened, kindly, post your ticket number here for the convenience of future readers.
mazur1000
Influencer
Posts: 23
Liked: 1 time
Joined: Feb 20, 2014 6:54 am
Full Name: alex mazur
Contact:

Re: restore large files from tape

Post by mazur1000 »

I installed Patch # 3a (build 7.0.0.839). After installing the problem persists ( I completely removed veeam, set , set Patch # 3a, set up the new configuration. Copied to tape file and restored it ) File occupies a volume of more than one tape . After installing the patch build 7.0.0.839 restored Fals began to occupy more than 200GB on the original. Prior to installing the patch 3a (version 7.0.0.833 ), the difference between the original and the restored file was 100KB .
After installing Patch # 3a situation with the restoration has become virtually the same as in version 7.0.0.771. Only on 7.0.0.771 when restoring error was Script execution timed out, and now it's not, but the restored file is broken .
mazur1000
Influencer
Posts: 23
Liked: 1 time
Joined: Feb 20, 2014 6:54 am
Full Name: alex mazur
Contact:

Re: restore large files from tape

Post by mazur1000 »

The newer Patch #3a build additionally addresses the following issues: ◾ Spanning a large file across multiple tapes may be performed incorrectly with certain tape drives that do not implement sending the end of media event. As a result, the restored file may have some of its parts corrupted.
This problem is not solved
veremin
Product Manager
Posts: 20270
Liked: 2252 times
Joined: Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin
Contact:

Re: restore large files from tape

Post by veremin »

Have you had a chance to contact the support team with this? This sort of issues typically requires logs investigation and can't be addressed properly via forum correspondence.

Thanks.
Escapo IT
Influencer
Posts: 11
Liked: 4 times
Joined: May 29, 2012 6:05 pm
Contact:

Re: restore large files from tape

Post by Escapo IT »

We *MIGHT* be impacted by this aswell; As a result of seeing this thread I am trying to conduct a test.

I have an IBM TS3200 library with an LTO6 drive (LTO5 cartridges though) with a rather large ~1TB VBK file spanning 2 tapes. I am currently restoring this as a file to a local filesystem on the primary Veeam server. Any tips on how to verify the integrity of this VBK file, short of trying to restore all of the 50 or so machines it contains ?

As an aside: The restore from tape seems quite a bit slower (~150 GB/hour) then the copy to tape (~450 GB/hour), but hey :)
veremin
Product Manager
Posts: 20270
Liked: 2252 times
Joined: Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin
Contact:

Re: restore large files from tape

Post by veremin »

Based on OP post, he seemed to get corruption error immediately when he tried to perform restore procedure. Also, are currently on the latest product patch? Do you see any difference in size of original and restored backups? Thanks.
foggy
Veeam Software
Posts: 21069
Liked: 2115 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: restore large files from tape

Post by foggy »

Escapo IT wrote:Any tips on how to verify the integrity of this VBK file, short of trying to restore all of the 50 or so machines it contains ?
You can utilize the backup validator command-line tool to perform this task.
Escapo IT
Influencer
Posts: 11
Liked: 4 times
Joined: May 29, 2012 6:05 pm
Contact:

Re: restore large files from tape

Post by Escapo IT »

The restore from tape to local storage as a VBK file went fine. We are using 'Reverse Incremental' and the job has run since duplicating to tape so cannot compare file size. Software version 833.

Upon importing the VBK file (before using the backup validator tool) the following error appears;

Code: Select all

---------------------------
Backup & Replication
---------------------------
Dir failed, storageFileName 'V:\RESTORE\GeneralPurpose2014-03-01T203103.vbk'

All instances of the storage metadata are corrupted.


---------------------------
OK   
---------------------------
So it would appear there is a real issue here !
foggy
Veeam Software
Posts: 21069
Liked: 2115 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: restore large files from tape

Post by foggy »

I suggest trying backup validator and contact support for investigation.
Escapo IT
Influencer
Posts: 11
Liked: 4 times
Joined: May 29, 2012 6:05 pm
Contact:

Re: restore large files from tape

Post by Escapo IT »

Can you use the validator on a VBK file without importing it first ? Cause the import fails, ...

According to the OP patch 3a does not solve the issue for him. Do you suggest I contact support now (running 833) or upgrade to 839 first, retest & then contact support should the issue not be solved ?
foggy
Veeam Software
Posts: 21069
Liked: 2115 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: restore large files from tape

Post by foggy »

Escapo IT wrote:Can you use the validator on a VBK file without importing it first ? Cause the import fails, ...
Yes, just point it to a backup file on disk, no need to import it first.
Escapo IT wrote:According to the OP patch 3a does not solve the issue for him. Do you suggest I contact support now (running 833) or upgrade to 839 first, retest & then contact support should the issue not be solved ?
I would try and upgrade first as the fix introduced in the patch could address your particular occasion of the issue (as far as I understand, the issue is specific to some certain tape drives).
mazur1000
Influencer
Posts: 23
Liked: 1 time
Joined: Feb 20, 2014 6:54 am
Full Name: alex mazur
Contact:

Re: restore large files from tape

Post by mazur1000 »

copy to tape file folder (backup created acronis - full and several incremental). Full occupies a volume 3000GB, incremental 4 x 50GB.
restore via FILE-TAPE-select a file full backup capacity 3000 GB.
In the end, after we get the file recovery volume 3200 (3000 +200)

files before backup:
F-Backup_2014_02_16_18_30_14_044D.TIB 2,95 ТB -full
F-Backup_2014_02_16_18_30_14_044D5.TIB 38,0 GB
F-Backup_2014_02_16_18_30_14_044D9.TIB 49,1GB
F-Backup_2014_02_16_18_30_14_044D11.TIB 57,2GB
F-Backup_2014_02_16_18_30_14_044D12.TIB 7,35 GB

After restore from tape full backup i had file:

F-Backup_2014_02_16_18_30_14_044D.TIB 3,10 ТБ

that differs from the sum of all reserved files at 200kb.

Pictures in this forum could not invest, sorry
foggy
Veeam Software
Posts: 21069
Liked: 2115 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: restore large files from tape

Post by foggy »

Alex, have you had a chance to open a case with support team?
Escapo IT
Influencer
Posts: 11
Liked: 4 times
Joined: May 29, 2012 6:05 pm
Contact:

Re: restore large files from tape

Post by Escapo IT » 4 people like this post

Had a chance to retest this scenario and happy to report this seems fixed for me with 839.

Took a new backup (backup-to-tape), split over two LTO5 tapes, restore ~1TB vbk file to local filesystem. Compared filesize, they are identical. Was able to import the VBK file into Veeam, was able to "double-click" the VBK file to see the contents within Veeam, ran backup validator which showed no errors. Load of my mind !!

To the others reading this thread; The patch notes and the comments from Veeam here on the forums seem to indicate this is environment specific, in fact the whole tape ecosystem seems to be a minefield of slight variations depending on drive/library/manufacturer, so I would urge you to test this for yourself, even after applying patch 3a. A backup is worthless without the ability to restore it !!

We have certainly been guilty of being lazy in that regard but this has opened my eyes and I will push for better policies making regular checks mandatory, perhaps using something like SureBackup as a first step.
veremin
Product Manager
Posts: 20270
Liked: 2252 times
Joined: Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin
Contact:

Re: restore large files from tape

Post by veremin »

Yes, as far as I know, the said issue was related to specific combination of devices, etc. So, even with the latest patch, it's worth double checking backup recoverability.

Also, it's recommended to verify the source backup data, so that, the potential corruption isn't copied to secondary location by backup copy or backup to tape job. The best way to do in automatic fashion is to use SureBackup functionality.

Anyway, thanks for coming back and confirming that the latest patch solves the issue.
mazur1000
Influencer
Posts: 23
Liked: 1 time
Joined: Feb 20, 2014 6:54 am
Full Name: alex mazur
Contact:

Re: restore large files from tape

Post by mazur1000 »

Yes. The latest patch solves the problem with restoring from tape backup is virtual machines created using veeam. If using veeam to backup and then restore individual files, the problem persists (ibm ts3200 lto 6).
When backups of the same file on the same library using symantec no problems
foggy
Veeam Software
Posts: 21069
Liked: 2115 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: restore large files from tape

Post by foggy »

As it is stated above, probably this is just another issue around the same topic, so please kindly continue working with support to address it.
Gostev
Chief Product Officer
Posts: 31460
Liked: 6648 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: restore large files from tape

Post by Gostev » 1 person likes this post

Escapo IT wrote:Had a chance to retest this scenario and happy to report this seems fixed for me with 839.

Took a new backup (backup-to-tape), split over two LTO5 tapes, restore ~1TB vbk file to local filesystem. Compared filesize, they are identical. Was able to import the VBK file into Veeam, was able to "double-click" the VBK file to see the contents within Veeam, ran backup validator which showed no errors. Load of my mind !!

To the others reading this thread; The patch notes and the comments from Veeam here on the forums seem to indicate this is environment specific, in fact the whole tape ecosystem seems to be a minefield of slight variations depending on drive/library/manufacturer, so I would urge you to test this for yourself, even after applying patch 3a. A backup is worthless without the ability to restore it !!

We have certainly been guilty of being lazy in that regard but this has opened my eyes and I will push for better policies making regular checks mandatory, perhaps using something like SureBackup as a first step.
Thanks for coming back to confirm the issue is resolved. And you are right, over 80% of tape-related issues we've faced so far were device-specific issues instead of our bugs. Nevertheless, we are forced to put workarounds directly in our code, because it provides our customers with the solution about 10x times faster than what it takes a typical hardware vendor to fix the drivers code (unless the issue sits in hardware, which makes it even longer).
foggy
Veeam Software
Posts: 21069
Liked: 2115 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: restore large files from tape

Post by foggy »

Alex, it would be much appreciated if you could provide your case ID here, our QA guys are very is interested in it.
mazur1000
Influencer
Posts: 23
Liked: 1 time
Joined: Feb 20, 2014 6:54 am
Full Name: alex mazur
Contact:

Re: restore large files from tape

Post by mazur1000 »

well, I'll create today
mazur1000
Influencer
Posts: 23
Liked: 1 time
Joined: Feb 20, 2014 6:54 am
Full Name: alex mazur
Contact:

Re: restore large files from tape

Post by mazur1000 » 1 person likes this post

case 00529403
Gostev
Chief Product Officer
Posts: 31460
Liked: 6648 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: restore large files from tape

Post by Gostev »

mazur1000 wrote:Yes. The latest patch solves the problem with restoring from tape backup is virtual machines created using veeam. If using veeam to backup and then restore individual files, the problem persists (ibm ts3200 lto 6)
This is very strange. We happen to have the exact same tape library in our QC lab. We did multiple tests with large files in the past few days, and could not confirm the issue. Thanks for opening a support case, may be the debug logs will shed some light...
mazur1000
Influencer
Posts: 23
Liked: 1 time
Joined: Feb 20, 2014 6:54 am
Full Name: alex mazur
Contact:

Re: restore large files from tape

Post by mazur1000 »

Can there be such a problem because of the built-in encryption library. Prior to installation patch3 I tried without encryption, the problem was the same
foggy
Veeam Software
Posts: 21069
Liked: 2115 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: restore large files from tape

Post by foggy »

Further investigation is likely to give answers to this and other questions, so please continue working with our engineers.
Dima P.
Product Manager
Posts: 14396
Liked: 1568 times
Joined: Feb 04, 2013 2:07 pm
Full Name: Dmitry Popov
Location: Prague
Contact:

Re: restore large files from tape

Post by Dima P. »

Alex,

Could you, please, clarify - you enabled/disabled tape library built in encryption after installing patch 3? If it is on, are you using it for library or on single drive only? Enabling/disabling hardware encryption could dramatically impact performance on backup/restore operations from tape libraries. (as well as hardware compression).

Thank you.
mazur1000
Influencer
Posts: 23
Liked: 1 time
Joined: Feb 20, 2014 6:54 am
Full Name: alex mazur
Contact:

Re: restore large files from tape

Post by mazur1000 »

No, I tried to backup-recovery without encryption before installing patch3. Encryption is used for all drives at the library level. Speed ​​difference between encryption and no encryption I have not seen
Dima P.
Product Manager
Posts: 14396
Liked: 1568 times
Joined: Feb 04, 2013 2:07 pm
Full Name: Dmitry Popov
Location: Prague
Contact:

Re: restore large files from tape

Post by Dima P. »

Alex,
Got it! I’ve already send this information to our support department. Thank you for the heads up!
Post Reply

Who is online

Users browsing this forum: Google [Bot] and 13 guests