restore large files from tape

[mazur1000]

veeam 7 patch 3
when restoring large files (3-10 TB) from tape (ibm 3200 lto 6) recovered file size turns out not to be the size of the original (differs by about 100-200 KB).
Recovered files gets corrupted.
And the size of the recovered file becomes invalid in the last stage recovery
Before installing patch 3 was a mistake: Script execution timed out

[veremin]

Hi, Alex,

What type of restore you're using? Also, does the given backup file reside on one tape or on many of them?

Thanks.

[mazur1000]

I use RESTORE FILE FROM TAPE
Tried to recover. Vbk 6TB file and a file created using acronis 3 TB.
In the former case, three backup tapes held in the second - 2 tape.

[veremin]

So, both files were shown as corrupted, right?

Anyway, it seems that deep log investigation is required here, so, please, open a ticket with our support team and let them investigate it directly.

Also, once opened, kindly, post your ticket number here for the convenience of future readers.

[mazur1000]

I installed Patch # 3a (build 7.0.0.839). After installing the problem persists ( I completely removed veeam, set , set Patch # 3a, set up the new configuration. Copied to tape file and restored it ) File occupies a volume of more than one tape . After installing the patch build 7.0.0.839 restored Fals began to occupy more than 200GB on the original. Prior to installing the patch 3a (version 7.0.0.833 ), the difference between the original and the restored file was 100KB .
After installing Patch # 3a situation with the restoration has become virtually the same as in version 7.0.0.771. Only on 7.0.0.771 when restoring error was Script execution timed out, and now it's not, but the restored file is broken .

[mazur1000]

The newer Patch #3a build additionally addresses the following issues: Spanning a large file across multiple tapes may be performed incorrectly with certain tape drives that do not implement sending the end of media event. As a result, the restored file may have some of its parts corrupted.
This problem is not solved

[veremin]

Have you had a chance to contact the support team with this? This sort of issues typically requires logs investigation and can't be addressed properly via forum correspondence.

Thanks.

[Escapo IT]

We *MIGHT* be impacted by this aswell; As a result of seeing this thread I am trying to conduct a test.

I have an IBM TS3200 library with an LTO6 drive (LTO5 cartridges though) with a rather large ~1TB VBK file spanning 2 tapes. I am currently restoring this as a file to a local filesystem on the primary Veeam server. Any tips on how to verify the integrity of this VBK file, short of trying to restore all of the 50 or so machines it contains ?

As an aside: The restore from tape seems quite a bit slower (~150 GB/hour) then the copy to tape (~450 GB/hour), but hey

[veremin]

Based on OP post, he seemed to get corruption error immediately when he tried to perform restore procedure. Also, are currently on the latest product patch? Do you see any difference in size of original and restored backups? Thanks.

[foggy]

Any tips on how to verify the integrity of this VBK file, short of trying to restore all of the 50 or so machines it contains ?

You can utilize the backup validator command-line tool to perform this task.

[Escapo IT]

The restore from tape to local storage as a VBK file went fine. We are using 'Reverse Incremental' and the job has run since duplicating to tape so cannot compare file size. Software version 833.

Upon importing the VBK file (before using the backup validator tool) the following error appears;

Code: Select all

---------------------------
Backup & Replication
---------------------------
Dir failed, storageFileName 'V:\RESTORE\GeneralPurpose2014-03-01T203103.vbk'

All instances of the storage metadata are corrupted.


---------------------------
OK   
---------------------------

So it would appear there is a real issue here !

[foggy]

I suggest trying backup validator and contact support for investigation.

[Escapo IT]

Can you use the validator on a VBK file without importing it first ? Cause the import fails, ...

According to the OP patch 3a does not solve the issue for him. Do you suggest I contact support now (running 833) or upgrade to 839 first, retest & then contact support should the issue not be solved ?

[foggy]

Can you use the validator on a VBK file without importing it first ? Cause the import fails, ...

Yes, just point it to a backup file on disk, no need to import it first.

According to the OP patch 3a does not solve the issue for him. Do you suggest I contact support now (running 833) or upgrade to 839 first, retest & then contact support should the issue not be solved ?

I would try and upgrade first as the fix introduced in the patch could address your particular occasion of the issue (as far as I understand, the issue is specific to some certain tape drives).

[mazur1000]

copy to tape file folder (backup created acronis - full and several incremental). Full occupies a volume 3000GB, incremental 4 x 50GB.
restore via FILE-TAPE-select a file full backup capacity 3000 GB.
In the end, after we get the file recovery volume 3200 (3000 +200)

files before backup:
F-Backup_2014_02_16_18_30_14_044D.TIB 2,95 ТB -full
F-Backup_2014_02_16_18_30_14_044D5.TIB 38,0 GB
F-Backup_2014_02_16_18_30_14_044D9.TIB 49,1GB
F-Backup_2014_02_16_18_30_14_044D11.TIB 57,2GB
F-Backup_2014_02_16_18_30_14_044D12.TIB 7,35 GB

After restore from tape full backup i had file:

F-Backup_2014_02_16_18_30_14_044D.TIB 3,10 ТБ

that differs from the sum of all reserved files at 200kb.

Pictures in this forum could not invest, sorry

[foggy]

Alex, have you had a chance to open a case with support team?

[Escapo IT]

Had a chance to retest this scenario and happy to report this seems fixed for me with 839.

Took a new backup (backup-to-tape), split over two LTO5 tapes, restore ~1TB vbk file to local filesystem. Compared filesize, they are identical. Was able to import the VBK file into Veeam, was able to "double-click" the VBK file to see the contents within Veeam, ran backup validator which showed no errors. Load of my mind !!

To the others reading this thread; The patch notes and the comments from Veeam here on the forums seem to indicate this is environment specific, in fact the whole tape ecosystem seems to be a minefield of slight variations depending on drive/library/manufacturer, so I would urge you to test this for yourself, even after applying patch 3a. A backup is worthless without the ability to restore it !!

We have certainly been guilty of being lazy in that regard but this has opened my eyes and I will push for better policies making regular checks mandatory, perhaps using something like SureBackup as a first step.

[veremin]

Yes, as far as I know, the said issue was related to specific combination of devices, etc. So, even with the latest patch, it's worth double checking backup recoverability.

Also, it's recommended to verify the source backup data, so that, the potential corruption isn't copied to secondary location by backup copy or backup to tape job. The best way to do in automatic fashion is to use SureBackup functionality.

Anyway, thanks for coming back and confirming that the latest patch solves the issue.

[mazur1000]

Yes. The latest patch solves the problem with restoring from tape backup is virtual machines created using veeam. If using veeam to backup and then restore individual files, the problem persists (ibm ts3200 lto 6).
When backups of the same file on the same library using symantec no problems

[foggy]

As it is stated above, probably this is just another issue around the same topic, so please kindly continue working with support to address it.

[Gostev]

Had a chance to retest this scenario and happy to report this seems fixed for me with 839.

Took a new backup (backup-to-tape), split over two LTO5 tapes, restore ~1TB vbk file to local filesystem. Compared filesize, they are identical. Was able to import the VBK file into Veeam, was able to "double-click" the VBK file to see the contents within Veeam, ran backup validator which showed no errors. Load of my mind !!

To the others reading this thread; The patch notes and the comments from Veeam here on the forums seem to indicate this is environment specific, in fact the whole tape ecosystem seems to be a minefield of slight variations depending on drive/library/manufacturer, so I would urge you to test this for yourself, even after applying patch 3a. A backup is worthless without the ability to restore it !!

We have certainly been guilty of being lazy in that regard but this has opened my eyes and I will push for better policies making regular checks mandatory, perhaps using something like SureBackup as a first step.

Thanks for coming back to confirm the issue is resolved. And you are right, over 80% of tape-related issues we've faced so far were device-specific issues instead of our bugs. Nevertheless, we are forced to put workarounds directly in our code, because it provides our customers with the solution about 10x times faster than what it takes a typical hardware vendor to fix the drivers code (unless the issue sits in hardware, which makes it even longer).

[foggy]

Alex, it would be much appreciated if you could provide your case ID here, our QA guys are very is interested in it.

[mazur1000]

well, I'll create today

[mazur1000]

case 00529403

[Gostev]

Yes. The latest patch solves the problem with restoring from tape backup is virtual machines created using veeam. If using veeam to backup and then restore individual files, the problem persists (ibm ts3200 lto 6)

This is very strange. We happen to have the exact same tape library in our QC lab. We did multiple tests with large files in the past few days, and could not confirm the issue. Thanks for opening a support case, may be the debug logs will shed some light...

[mazur1000]

Can there be such a problem because of the built-in encryption library. Prior to installation patch3 I tried without encryption, the problem was the same

[foggy]

Further investigation is likely to give answers to this and other questions, so please continue working with our engineers.

[Dima P.]

Alex,

Could you, please, clarify - you enabled/disabled tape library built in encryption after installing patch 3? If it is on, are you using it for library or on single drive only? Enabling/disabling hardware encryption could dramatically impact performance on backup/restore operations from tape libraries. (as well as hardware compression).

Thank you.

[mazur1000]

No, I tried to backup-recovery without encryption before installing patch3. Encryption is used for all drives at the library level. Speed ​​difference between encryption and no encryption I have not seen

[Dima P.]

Alex,
Got it! I’ve already send this information to our support department. Thank you for the heads up!