Veeam backup copy job corrupted files cannot backup to tape?

[albertwt]

Hi,

Can someone here please shed some light to my issue that I've been facing since I upgrade the Veeam Backup to version 8.0 ?
previously when the VBR was still on version 7 newly installed by the consultant it was working smoothly with Symantec but not recently after upgraded to Version 8.0

This has also been logged to case 00911739

PROBLEM:
Writing the Veeam backup files to LTO tape drives using Symantec Backup Exec 15 from my Offsite backup repository server (PRODBU01) where the VBR v8.0 and Backup Exec 15 is running.

OBJECT:
The files to be written to tape are backup copy job copied from the Data Center site continuously.

QUESTIONS:
Can anyone please suggest me how to verify or know which .VIB or .VBK file that is corrupted ?
What's the default Veeam behaviour of the corrupted Veeam Backup copy job ?

Error logs:
Because from the Veeam Job history I cannot see any failed backup job or failed Backup Copy job, but when the files is written to tape drive using Symantec Backup Exec 15, it complaints as follow:

Code: Select all

Backup - \\PRODBU01.domain.com\G:Directory Backups\BCOPY_DR_Oracle Servers 1\ was not found, or could not be accessed.
		None of the files or subdirectories contained within will be backed up.

		V-79-57344-33967 - The directory or file was not found, or could not be accessed.
		V-79-57344-65033 - Directory not found. Cannot backup directory \Backups\BCOPY_DR_SQL Servers 1 and its subdirectories.
		V-79-57344-65033 - Directory not found. Cannot backup directory \Backups\BCOPY_DR_SQL Servers 3 and its subdirectories.

		BackupV-79-57344-34108 - An unexpected error occurred when cleaning up snapshot volumes. Confirm that all snapped volumes are correctly resynchronized with the original volumes.
		Backup- \\PRODBU01.domain.com\G:WARNING: "\\PRODBU01.domain.com\G:\Backups\BCOPY_DR_Oracle Servers 1\BCOPY_DR_Oracle Servers 12015-04-27T070000.vbk" is a corrupt file. This file cannot verify.
		Unable to open the item \\PRODBU01.domain.com\G:\Backups\BCOPY_DR_Oracle Servers 1\BCOPY_DR_Oracle Servers 12015-04-28T070000.vib - skipped.
		Unable to open the item \\PRODBU01.domain.com\G:\Backups\BCOPY_DR_Oracle Servers 1\BCOPY_DR_Oracle Servers 12015-04-29T070000.vib - skipped.
		Unable to open the item \\PRODBU01.domain.com\G:\Backups\BCOPY_DR_Oracle Servers 1\BCOPY_DR_Oracle Servers 12015-04-30T070000.vib - skipped.
		Unable to open the item \\PRODBU01.domain.com\G:\Backups\BCOPY_DR_Oracle Servers 1\BCOPY_DR_Oracle Servers 12015-05-01T070000.vib - skipped.
		Unable to open the item \\PRODBU01.domain.com\G:\Backups\BCOPY_DR_Oracle Servers 1\BCOPY_DR_Oracle Servers 12015-05-02T070000.vib - skipped.
		Unable to open the item \\PRODBU01.domain.com\G:\Backups\BCOPY_DR_Oracle Servers 1\BCOPY_DR_Oracle Servers 12015-05-03T070000.vib - skipped.
		Unable to open the item \\PRODBU01.domain.com\G:\Backups\BCOPY_DR_Oracle Servers 1\BCOPY_DR_Oracle Servers 12015-05-04T070000.vib - skipped.

Thanks.

[Shestakov]

Hello Albert,
Backup copy job performs Health Check for Copied Backups.

When a new restore point is copied from the source backup repository, VBR calculates checksums for data blocks in the restore point and saves them with the backup data on the target backup repository.
During the health check, VBR calculates checksums for data blocks in the backup files stored on the target backup repository and compares them to the checksums that were previously stored to the backup files.
If a health check fails, VBR displays a warning in the job session report.

By default health check works once a month, but you can make it work as often as needed.

Do you leverage Surebackup capabilities as well? Thanks.

[albertwt]

Hi Nikita,

Does the corrupted files gets deleted by Veeam ?
if it does contains any corruption, I should get it by the email report / alert, but in this case I didn't get any.

Yes I do have leverage SUreBackup for test lab purpose.

[Shestakov]

Does the corrupted files gets deleted by Veeam ?

During the next synchronization interval, VBR transports valid data blocks from the source backup repository and stores them to the newly copied restore point on the target backup repository.

if it does contains any corruption, I should get it by the email report / alert, but in this case I didn't get any.

Yes, as I wrote VBR displays a warning in the job session report. You can receive a corresponding email notification if notifications are enabled.

Actually I wonder how Backup Exec can report the file corruption. In my understanding they need to compare source(result of Veeam backup copy job) and target(result of backup to tape) files to detect the corruption. Looks like the problem doesn`t apply to Veeam.

By the way, why no to use VBR backup-to-tape jobs as well? Thanks

[albertwt]

Well, my suspect is that the Symantec Backup Exec tries to get hold of the file by using VSS snapshot but somehow Veeam is constantly using it due to the continuous backup copy job. is my statement correct ?

[Shestakov]

Not really, Veeam backup copy job doesn`t use VSS, because it just copies existing restore points.
What version of VBR are you at? Thanks!

[albertwt]

I'm using VBR 8.0 Patch 1

[Shestakov]

Thanks, so there should be no issues since you use surebackup and backup copy health check. And as I said Backup Exec could not recognize if Veeam backup files are corrupted.
So I`d suggest to contact Symantec support or/and try Veeam backup to tape job.

[albertwt]

So based on Symantec support team explanation, the Final error: 0xe000fe36 - Corrupt data encountered means that The backup software cannot access the .VIB file during the Backup Exec window.
Is it because the Backup Copy job that is locking the file after the Veeam backup job finished ?

There was nothing changed in terms of abckup window for both Veeam and Backup exec, just the software was upgraded since 2013 to the latest version each and the backup to tape was always working until I upgraded to Veeam Backup v8.0 few months back.

Does Veeam Backup v8.0 has new behaviour when the backup copy finished or during the backup copy going ?

[foggy]

What files BE is set up to copy to tape and how specifically it defines whether the file is available for copying? The behavior could have similar nature to this issue.

[albertwt]

yes, that's correct. So in this case since I upgraded to VBR v8.0 my Symantec Backup Exec cannot successfully write it to tape.
Is there any idea what can be done here ? Symantec support mention that it is Veeam behaviour causing issue so it is not Symantec Backup Exec software issue.

[foggy]

What do our support engineers say about this behavior?

[albertwt]

I'm still waiting on that because the support engineer doesn't seems to know about this type of behaviour.
Any KB article or official statement on this ?

Because I need to write the files to the tape but it seems that it is impossible since VBR v8.0 like the case in http://forums.veeam.com/post147669.html#p147669 ?

[foggy]

The behavior discussed in that case slightly differs (only the most recent increment is not copied). So this should be investigated closer.

[albertwt]

Hi Foggy, Based on the case in this page: http://forums.veeam.com/veeam-backup-re ... 26374.html so is there any way to write it to tape ?

what's your suggestion or work around in this matter ?

[Shestakov]

Hello Albert,
What about using backup job, rather than backup copy as a source for the backup to tape job?
Thanks!

[albertwt]

Hi Nikita,

The primary backup repository is in the data center, while the Tape Drive is attached to the secondary repository server in HQ.
So it is not possible to do.

[foggy]

Since the behavior you're seeing does not look expected, I recommend continue investigating its reasons with support.

[Clayton.Lee]

Albert
Did you get a resolution to this? We are seeing exactly the same symptoms backing up Veeam copied images to tape with Backup Exec.

[foggy]

Clayton, I cannot see the resolution in the support case provided in the original post of this thread. So while you're waiting for Albert's reply, I recommend contacting support for your own investigation. Thanks.

[albertwt]

Albert
Did you get a resolution to this? We are seeing exactly the same symptoms backing up Veeam copied images to tape with Backup Exec.

Hi Clayton,

After long discussion and consideration with the management team, unfortuantely we have to ditch / get rid of the Symantec Backup Exec so I end up with Disk2Disk backup strategy.
Hence this issue was never resolved.

[Clayton.Lee]

Wow - that is not a great outcome. I will let you know if we go any better - we have raised a support case with Veeam in parallel with one with have had open with Symantec for some time.

[albertwt]

Yeah I know, but that was the incident last year.

Symantec support blamed that the files are opened or changed by Veeam service hence the backup is always corrupted.
While Veeam support also mentioned that the file is sometimes opened by Veeam Backup Copy job hence it is hard to know if the Backup Exec is currently backing up the large VBK files.

[veremin]

So I end up with Disk2Disk backup strategy.

Just our of curiosity - haven't you tried D2D2T strategy, using VB&R? Or did you find some caveats while working with tapes? Thanks.