Comprehensive data protection for all workloads
Post Reply
albertwt
Veeam Legend
Posts: 879
Liked: 46 times
Joined: Nov 05, 2009 12:24 pm
Location: Sydney, NSW
Contact:

Veeam backup copy job corrupted files cannot backup to tape?

Post by albertwt »

Hi,

Can someone here please shed some light to my issue that I've been facing since I upgrade the Veeam Backup to version 8.0 ?
previously when the VBR was still on version 7 newly installed by the consultant it was working smoothly with Symantec but not recently after upgraded to Version 8.0

This has also been logged to case 00911739

PROBLEM:
Writing the Veeam backup files to LTO tape drives using Symantec Backup Exec 15 from my Offsite backup repository server (PRODBU01) where the VBR v8.0 and Backup Exec 15 is running.

OBJECT:
The files to be written to tape are backup copy job copied from the Data Center site continuously.

QUESTIONS:
Can anyone please suggest me how to verify or know which .VIB or .VBK file that is corrupted ?
What's the default Veeam behaviour of the corrupted Veeam Backup copy job ?

Error logs:
Because from the Veeam Job history I cannot see any failed backup job or failed Backup Copy job, but when the files is written to tape drive using Symantec Backup Exec 15, it complaints as follow:

Code: Select all

Backup - \\PRODBU01.domain.com\G:Directory Backups\BCOPY_DR_Oracle Servers 1\ was not found, or could not be accessed.
		None of the files or subdirectories contained within will be backed up.

		V-79-57344-33967 - The directory or file was not found, or could not be accessed.
		V-79-57344-65033 - Directory not found. Cannot backup directory \Backups\BCOPY_DR_SQL Servers 1 and its subdirectories.
		V-79-57344-65033 - Directory not found. Cannot backup directory \Backups\BCOPY_DR_SQL Servers 3 and its subdirectories.

		BackupV-79-57344-34108 - An unexpected error occurred when cleaning up snapshot volumes. Confirm that all snapped volumes are correctly resynchronized with the original volumes.
		Backup- \\PRODBU01.domain.com\G:WARNING: "\\PRODBU01.domain.com\G:\Backups\BCOPY_DR_Oracle Servers 1\BCOPY_DR_Oracle Servers 12015-04-27T070000.vbk" is a corrupt file. This file cannot verify.
		Unable to open the item \\PRODBU01.domain.com\G:\Backups\BCOPY_DR_Oracle Servers 1\BCOPY_DR_Oracle Servers 12015-04-28T070000.vib - skipped.
		Unable to open the item \\PRODBU01.domain.com\G:\Backups\BCOPY_DR_Oracle Servers 1\BCOPY_DR_Oracle Servers 12015-04-29T070000.vib - skipped.
		Unable to open the item \\PRODBU01.domain.com\G:\Backups\BCOPY_DR_Oracle Servers 1\BCOPY_DR_Oracle Servers 12015-04-30T070000.vib - skipped.
		Unable to open the item \\PRODBU01.domain.com\G:\Backups\BCOPY_DR_Oracle Servers 1\BCOPY_DR_Oracle Servers 12015-05-01T070000.vib - skipped.
		Unable to open the item \\PRODBU01.domain.com\G:\Backups\BCOPY_DR_Oracle Servers 1\BCOPY_DR_Oracle Servers 12015-05-02T070000.vib - skipped.
		Unable to open the item \\PRODBU01.domain.com\G:\Backups\BCOPY_DR_Oracle Servers 1\BCOPY_DR_Oracle Servers 12015-05-03T070000.vib - skipped.
		Unable to open the item \\PRODBU01.domain.com\G:\Backups\BCOPY_DR_Oracle Servers 1\BCOPY_DR_Oracle Servers 12015-05-04T070000.vib - skipped.

Thanks.
--
/* Veeam software enthusiast user & supporter ! */
Shestakov
Veteran
Posts: 7328
Liked: 781 times
Joined: May 21, 2014 11:03 am
Full Name: Nikita Shestakov
Location: Prague
Contact:

Re: Veeam backup copy job corrupted files cannot backup to t

Post by Shestakov » 1 person likes this post

Hello Albert,
Backup copy job performs Health Check for Copied Backups.

When a new restore point is copied from the source backup repository, VBR calculates checksums for data blocks in the restore point and saves them with the backup data on the target backup repository.
During the health check, VBR calculates checksums for data blocks in the backup files stored on the target backup repository and compares them to the checksums that were previously stored to the backup files.
If a health check fails, VBR displays a warning in the job session report.

By default health check works once a month, but you can make it work as often as needed.

Do you leverage Surebackup capabilities as well? Thanks.
albertwt
Veeam Legend
Posts: 879
Liked: 46 times
Joined: Nov 05, 2009 12:24 pm
Location: Sydney, NSW
Contact:

Re: Veeam backup copy job corrupted files cannot backup to t

Post by albertwt »

Hi Nikita,

Does the corrupted files gets deleted by Veeam ?
if it does contains any corruption, I should get it by the email report / alert, but in this case I didn't get any.

Yes I do have leverage SUreBackup for test lab purpose.
--
/* Veeam software enthusiast user & supporter ! */
Shestakov
Veteran
Posts: 7328
Liked: 781 times
Joined: May 21, 2014 11:03 am
Full Name: Nikita Shestakov
Location: Prague
Contact:

Re: Veeam backup copy job corrupted files cannot backup to t

Post by Shestakov » 1 person likes this post

albertwt wrote:Does the corrupted files gets deleted by Veeam ?
During the next synchronization interval, VBR transports valid data blocks from the source backup repository and stores them to the newly copied restore point on the target backup repository.
albertwt wrote:if it does contains any corruption, I should get it by the email report / alert, but in this case I didn't get any.
Yes, as I wrote VBR displays a warning in the job session report. You can receive a corresponding email notification if notifications are enabled.

Actually I wonder how Backup Exec can report the file corruption. In my understanding they need to compare source(result of Veeam backup copy job) and target(result of backup to tape) files to detect the corruption. Looks like the problem doesn`t apply to Veeam.

By the way, why no to use VBR backup-to-tape jobs as well? Thanks
albertwt
Veeam Legend
Posts: 879
Liked: 46 times
Joined: Nov 05, 2009 12:24 pm
Location: Sydney, NSW
Contact:

Re: Veeam backup copy job corrupted files cannot backup to t

Post by albertwt »

Well, my suspect is that the Symantec Backup Exec tries to get hold of the file by using VSS snapshot but somehow Veeam is constantly using it due to the continuous backup copy job. is my statement correct ?
--
/* Veeam software enthusiast user & supporter ! */
Shestakov
Veteran
Posts: 7328
Liked: 781 times
Joined: May 21, 2014 11:03 am
Full Name: Nikita Shestakov
Location: Prague
Contact:

Re: Veeam backup copy job corrupted files cannot backup to t

Post by Shestakov » 1 person likes this post

Not really, Veeam backup copy job doesn`t use VSS, because it just copies existing restore points.
What version of VBR are you at? Thanks!
albertwt
Veeam Legend
Posts: 879
Liked: 46 times
Joined: Nov 05, 2009 12:24 pm
Location: Sydney, NSW
Contact:

Re: Veeam backup copy job corrupted files cannot backup to t

Post by albertwt »

I'm using VBR 8.0 Patch 1
--
/* Veeam software enthusiast user & supporter ! */
Shestakov
Veteran
Posts: 7328
Liked: 781 times
Joined: May 21, 2014 11:03 am
Full Name: Nikita Shestakov
Location: Prague
Contact:

Re: Veeam backup copy job corrupted files cannot backup to t

Post by Shestakov » 1 person likes this post

Thanks, so there should be no issues since you use surebackup and backup copy health check. And as I said Backup Exec could not recognize if Veeam backup files are corrupted.
So I`d suggest to contact Symantec support or/and try Veeam backup to tape job.
albertwt
Veeam Legend
Posts: 879
Liked: 46 times
Joined: Nov 05, 2009 12:24 pm
Location: Sydney, NSW
Contact:

Re: Veeam backup copy job corrupted files cannot backup to t

Post by albertwt »

So based on Symantec support team explanation, the Final error: 0xe000fe36 - Corrupt data encountered means that The backup software cannot access the .VIB file during the Backup Exec window.
Is it because the Backup Copy job that is locking the file after the Veeam backup job finished ?

There was nothing changed in terms of abckup window for both Veeam and Backup exec, just the software was upgraded since 2013 to the latest version each and the backup to tape was always working until I upgraded to Veeam Backup v8.0 few months back.

Does Veeam Backup v8.0 has new behaviour when the backup copy finished or during the backup copy going ?
--
/* Veeam software enthusiast user & supporter ! */
foggy
Veeam Software
Posts: 21069
Liked: 2115 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Veeam backup copy job corrupted files cannot backup to t

Post by foggy »

What files BE is set up to copy to tape and how specifically it defines whether the file is available for copying? The behavior could have similar nature to this issue.
albertwt
Veeam Legend
Posts: 879
Liked: 46 times
Joined: Nov 05, 2009 12:24 pm
Location: Sydney, NSW
Contact:

Re: Veeam backup copy job corrupted files cannot backup to t

Post by albertwt »

yes, that's correct. So in this case since I upgraded to VBR v8.0 my Symantec Backup Exec cannot successfully write it to tape.
Is there any idea what can be done here ? Symantec support mention that it is Veeam behaviour causing issue so it is not Symantec Backup Exec software issue.
--
/* Veeam software enthusiast user & supporter ! */
foggy
Veeam Software
Posts: 21069
Liked: 2115 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Veeam backup copy job corrupted files cannot backup to t

Post by foggy »

What do our support engineers say about this behavior?
albertwt
Veeam Legend
Posts: 879
Liked: 46 times
Joined: Nov 05, 2009 12:24 pm
Location: Sydney, NSW
Contact:

Re: Veeam backup copy job corrupted files cannot backup to t

Post by albertwt »

I'm still waiting on that because the support engineer doesn't seems to know about this type of behaviour.
Any KB article or official statement on this ?

Because I need to write the files to the tape but it seems that it is impossible since VBR v8.0 like the case in http://forums.veeam.com/post147669.html#p147669 ?
--
/* Veeam software enthusiast user & supporter ! */
foggy
Veeam Software
Posts: 21069
Liked: 2115 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Veeam backup copy job corrupted files cannot backup to t

Post by foggy » 1 person likes this post

The behavior discussed in that case slightly differs (only the most recent increment is not copied). So this should be investigated closer.
albertwt
Veeam Legend
Posts: 879
Liked: 46 times
Joined: Nov 05, 2009 12:24 pm
Location: Sydney, NSW
Contact:

Re: Veeam backup copy job corrupted files cannot backup to t

Post by albertwt »

Hi Foggy, Based on the case in this page: http://forums.veeam.com/veeam-backup-re ... 26374.html so is there any way to write it to tape ?

what's your suggestion or work around in this matter ?
--
/* Veeam software enthusiast user & supporter ! */
Shestakov
Veteran
Posts: 7328
Liked: 781 times
Joined: May 21, 2014 11:03 am
Full Name: Nikita Shestakov
Location: Prague
Contact:

Re: Veeam backup copy job corrupted files cannot backup to t

Post by Shestakov »

Hello Albert,
What about using backup job, rather than backup copy as a source for the backup to tape job?
Thanks!
albertwt
Veeam Legend
Posts: 879
Liked: 46 times
Joined: Nov 05, 2009 12:24 pm
Location: Sydney, NSW
Contact:

Re: Veeam backup copy job corrupted files cannot backup to t

Post by albertwt »

Hi Nikita,

The primary backup repository is in the data center, while the Tape Drive is attached to the secondary repository server in HQ.
So it is not possible to do.
--
/* Veeam software enthusiast user & supporter ! */
foggy
Veeam Software
Posts: 21069
Liked: 2115 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Veeam backup copy job corrupted files cannot backup to t

Post by foggy »

Since the behavior you're seeing does not look expected, I recommend continue investigating its reasons with support.
Clayton.Lee
Lurker
Posts: 2
Liked: 1 time
Joined: Feb 24, 2016 12:28 am
Full Name: Clayton Lee
Contact:

Re: Veeam backup copy job corrupted files cannot backup to t

Post by Clayton.Lee »

Albert
Did you get a resolution to this? We are seeing exactly the same symptoms backing up Veeam copied images to tape with Backup Exec.
foggy
Veeam Software
Posts: 21069
Liked: 2115 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Veeam backup copy job corrupted files cannot backup to t

Post by foggy »

Clayton, I cannot see the resolution in the support case provided in the original post of this thread. So while you're waiting for Albert's reply, I recommend contacting support for your own investigation. Thanks.
albertwt
Veeam Legend
Posts: 879
Liked: 46 times
Joined: Nov 05, 2009 12:24 pm
Location: Sydney, NSW
Contact:

Re: Veeam backup copy job corrupted files cannot backup to t

Post by albertwt » 1 person likes this post

Clayton.Lee wrote:Albert
Did you get a resolution to this? We are seeing exactly the same symptoms backing up Veeam copied images to tape with Backup Exec.
Hi Clayton,

After long discussion and consideration with the management team, unfortuantely we have to ditch / get rid of the Symantec Backup Exec :? so I end up with Disk2Disk backup strategy.
Hence this issue was never resolved.
--
/* Veeam software enthusiast user & supporter ! */
Clayton.Lee
Lurker
Posts: 2
Liked: 1 time
Joined: Feb 24, 2016 12:28 am
Full Name: Clayton Lee
Contact:

Re: Veeam backup copy job corrupted files cannot backup to t

Post by Clayton.Lee » 1 person likes this post

Wow - that is not a great outcome. I will let you know if we go any better - we have raised a support case with Veeam in parallel with one with have had open with Symantec for some time.
albertwt
Veeam Legend
Posts: 879
Liked: 46 times
Joined: Nov 05, 2009 12:24 pm
Location: Sydney, NSW
Contact:

Re: Veeam backup copy job corrupted files cannot backup to t

Post by albertwt »

:roll: Yeah I know, but that was the incident last year.

Symantec support blamed that the files are opened or changed by Veeam service hence the backup is always corrupted.
While Veeam support also mentioned that the file is sometimes opened by Veeam Backup Copy job hence it is hard to know if the Backup Exec is currently backing up the large VBK files.
--
/* Veeam software enthusiast user & supporter ! */
veremin
Product Manager
Posts: 20270
Liked: 2252 times
Joined: Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin
Contact:

Re: Veeam backup copy job corrupted files cannot backup to t

Post by veremin »

So I end up with Disk2Disk backup strategy.
Just our of curiosity - haven't you tried D2D2T strategy, using VB&R? Or did you find some caveats while working with tapes? Thanks.
Post Reply

Who is online

Users browsing this forum: legil.miguel and 191 guests