Possible Backup Corruption

[wa15]

Hello all,

So I have a ticket open on this (00572207) but also reaching out to the community:

I have two backup copy jobs that seem to be corrupt. Both copy jobs are taking local backups and bringing them to the main site over our WAN connection.

Backup Copy 1 is giving the following error message:
"5/29/2014 11:37:21 AM :: Processing 'VM' Error: RLE decompression error: [1054790] bytes decoded to [1081674] instead of [2315261956].
--tr:Failed to decompress data block.
--tr:Failed to read block [142413] from restore point
--tr:The specified chunk of the file cannot be read, specified read offset '149317222400', data size '33554432'.
--tr:Failed to read data from disk 'trfiler-flat.vmdk' in source backup 'E:\Veeam Backups\Troy Local Backup\Troy Local Backup2014-05-27T180108.vib', read offset '149317222400'.
--tr:Incrementa"

Interestingly, the local backup incrementals are completing fine, but Active Fulls and Synthetic Fulls are giving the following error messages, respectively:

Processing 'VM' Error: Client error: ChannelError: ConnectionReset
Error: The RPC server is unavailable RPC function call failed. Function name: [DoRpc]. Target machine: Proxy.

CompileFIB failed Client error: ChannelError: ConnectionReset
Error: Client error: ChannelError: ConnectionReset

Backup Copy 2 is giving them following error message:
6/2/2014 8:38:09 PM :: Patch failed
6/2/2014 8:39:11 PM :: Failed to transform full backup. Terminated at 6/2/2014 8:40:31 PM
6/2/2014 8:39:30 PM :: Failed to create restore point 3/31/2014 8:00:00 PM

While the local backups including the synthetic fulls are completing fine.

Any ideas on what may be causing these two errors? Backup Copy 1, I am assuming I will just have to blow away the local copy and start from scratch as the chain appears to be corrupt? Is there a way to delete individual restore points as of when the corruption started?

Thanks for your help!

[foggy]

Farshid, I would start investigating the regular backup job failures as most likely these cause backup copy to fail (or yes, just start both from scratch and see how it goes). Individual restore points cannot be deleted as this would break the entire backup chain.

[Vitaliy S.]

Hi Farshid,

I would probably try to focus on the original failures of the primary backup jobs. Do you see these errors happening constantly on full job runs? What kind of target storage are you using?

Thanks!

[wa15]

Thank you both for your help.

I do see the errors happening constantly on full job runs for backup copy #1, but for backup copy #2 the full's are completing fine. My suspicion is that for backup copy 2, the issue is on the target and that the source/local backups are okay since it's failing on transform. Is that correct? I suppose I would just blow the backup copy files away for the #2 and start from scratch? I might have older backup copy files on tape, is it worth getting those and using those as a seed?

I am running a Synology NAS connected via in guest iSCSI on the local backups. And an SAS connected MD1000 for the target where the backup copy jobs are going to.

[foggy]

My suspicion is that for backup copy 2, the issue is on the target and that the source/local backups are okay since it's failing on transform. Is that correct?

It looks like this. However I would double-check with support on possible reasons of the error.

I suppose I would just blow the backup copy files away for the #2 and start from scratch? I might have older backup copy files on tape, is it worth getting those and using those as a seed?

You could, indeed, seed the backup copy job to avoid excessive WAN traffic, however keep in mind that backup copy job can be mapped to a single full backup file only.

[wa15]

Thank you, I will take a look at that.

For backup copy #1 which apparently has corrupt backup files, is there any way I can keep one or two FULL backups around prior to when the corruption started occurring? I don't feel comfortable removing all the local backups. We are doing synthetic fulls along with traditional incrementals.

[foggy]

I wonder if you can tell when the corruption started to occur... Of course, you can keep backups prior to that, just copy them somewhere until you have the required number of fresh restore points available.

[wa15]

Looking at the job report, I can see that the corruption for backup copy #1 started occurring around May 15th. So would I just browse the backup files, delete files on and after May 15th, then copy everything older to another location, go under Backups in B&R and "remove from backups" any files and then just run an "Active Full"?

[wa15]

What I tried now is to just rename the folder the local backup repository (source for backup copy 1) points to, and creating a new folder with the same name so that the repository is basically "clean". Will see if the initial backup run which should be a full completes fine.

[Vitaliy S.]

Yes, second approach would be more recommended, as deleting backup files is really bad practice.