Comprehensive data protection for all workloads
Post Reply
leeg123
Enthusiast
Posts: 40
Liked: 2 times
Joined: Dec 13, 2010 12:50 am
Full Name: lee garet
Contact:

corrupted backup

Post by leeg123 »

Over the weekend, one of our veeam backup servers lost power in the middle a backup. Needles to say we corrupted the last full backup file in our reverese incremental chain.
The vbk file is still present and seems pretty large. We can import the entire chain, but if we test a restore, we get "some errors occured. See log files"
Is there any way to cleanup this vbk and back of of the failed backups? I'd hate to loose this history of backups? We have since started a new backup set and moved these files to a safe location for testing

thanks

lee
Gostev
Chief Product Officer
Posts: 31459
Liked: 6648 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: corrupted backup

Post by Gostev »

Hi, Lee. You can let the backup job run, and the job will take care of any incomplete restore points. Thanks!
leeg123
Enthusiast
Posts: 40
Liked: 2 times
Joined: Dec 13, 2010 12:50 am
Full Name: lee garet
Contact:

Re: corrupted backup

Post by leeg123 »

Thats just it

we got errors right away about repository being out of sync. Even remapping it back doesn't allow a job to run.
Last attempt saw "8/17/2015 5:38:36 PM :: Error: All instances of the storage metadata are corrupted.
Failed to backup text locally. Backup: [veeamfs:0:a99f0333-1fa7-4d1f-a9c2-4c3f12f1971c (8)\summary.xml@

Agent failed to process method {DataTransfer.BackupText}."


i've seen it be able to undo previous backup failures in the past with other servers. This particular time, it didn't happen.
Gostev
Chief Product Officer
Posts: 31459
Liked: 6648 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: corrupted backup

Post by Gostev »

leeg123 wrote:Error: All instances of the storage metadata are corrupted.
Ah, this error means it was certainly not a regular power outage. This error actually means that your backup storage device has severely corrupted backup files. There is redundancy for the metadata in question - basically, each metadata section is stored twice in the backup file. And having both instances corrupted means major storage-level corruption (not just some bad bit from URE).

Bottom line, this means your backups are completely unusable. Without this metadata, it is impossible to make any sense out of the backup files' content. So, you really have no other option except deleting them, and starting a new backup from the scratch... sorry to bring bad news.

Note: if you are using backup mode with periodic fulls, then be sure to also check restore points in older full backup chain separately before deleting them - depending on the data corruption scenario, those may still be good - unlike the latest one.
leeg123
Enthusiast
Posts: 40
Liked: 2 times
Joined: Dec 13, 2010 12:50 am
Full Name: lee garet
Contact:

Re: corrupted backup

Post by leeg123 »

Thank you

Yes, the storage repository seemed to lose power first. Ok, these backups are useless
I think periodic fulls is the best defense against something like this. Those backups would have been ok

thanks again
Gostev
Chief Product Officer
Posts: 31459
Liked: 6648 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: corrupted backup

Post by Gostev »

Yes, forever-incremental backup has its extra risks. This is also the reason why we require archived GFS restore points in Backup Copy jobs to be independent fulls, even though this causes some complaints due to the disk space requirements.

However, even periodic fulls may not always be the answer, as I saw situations when the entire content of the storage device got corrupted. Which is why the best solution is to always copy your backups to an independent isolated storage with Backup Copy jobs, or use tape.

Thanks!
rstone
Enthusiast
Posts: 45
Liked: 2 times
Joined: Jan 27, 2015 3:35 pm
Full Name: Rob Stone
Contact:

[MERGED] Currupt off site copies

Post by rstone »

Good afternoon

One of my buildings had a power cut and a veeam copy(ies) was running which stopped the copy. Now none of my veeam copies work and I can't do a restore from them. When the copies run they do one of the errors below, goto idle then errors then idle etc.

Code: Select all

01/10/2015 11:42:24 :: Error: Storage version [880558889] is not supported for read-only access.
01/10/2015 08:54:41 :: Error: All instances of the storage metadata are corrupted.
Agent failed to process method {Stg.OpenRead}.
So at the moment I cant do a copy and I can restore from the copies. I have four copy jobs with around 44 restore points each using 18 TB of data. Does this mean all this is useless and I have to start again? I hope not otherwise it might be the final straw with Veeam!

Any advice I would be grateful even if I can just get some of the backups back in the event of a disaster?

Thank you.............
rstone
Enthusiast
Posts: 45
Liked: 2 times
Joined: Jan 27, 2015 3:35 pm
Full Name: Rob Stone
Contact:

Re: Currupt off site copies

Post by rstone »

Sorry mean can't restore from the copies.
foggy
Veeam Software
Posts: 21069
Liked: 2115 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: corrupted backup

Post by foggy »

Rob, this looks like a storage-level corruption, not caused by backup copy jobs running during outage. Please see above for details.
Gostev
Chief Product Officer
Posts: 31459
Liked: 6648 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: corrupted backup

Post by Gostev »

Yes, this error indicates server storage hardware issue that severely corrupted backup files. Our backup format has redundancy for the metadata in question - basically, each metadata section is stored twice in the backup file. This protects metadata from occasional "bad bit" issue (each hard drive has URE parameter which grades for a chance of certain unrecoverable error, URE starts from 1 error per 100TB or so - unless you are using consumer hard drives).

However, in you case both instances of metadata are corrupted, and moreover - across 4 different backup files. Which can only mean that major storage-level corruption has happened on your backup storage device. In our experience, this usually happens either due to firmware bugs, or due to RAID controllers starting to misbehave and just write rubbish to disks. While these issues are more likely to happen on low end devices, we sometimes see this with enterprise-class hardware as well.

Bottom line, this means your backups are completely unusable. Without this metadata, it is impossible to make any sense out of the backup files' content, unfortunately. So yes, you need to start over again. However, for obvious reasons, it does not make sense to do so until you troubleshoot the issue with target storage for those Backup Copy jobs.

Thanks!
rstone
Enthusiast
Posts: 45
Liked: 2 times
Joined: Jan 27, 2015 3:35 pm
Full Name: Rob Stone
Contact:

Re: corrupted backup

Post by rstone »

I have found that my QNAP NAS had lost a drive which basically shouldn't matter as it has raid 5 and the share seems fine i.e. I quickly copied a TB backup to it last night. I find it very strange that it corrupts everything and not just part of the backup. Hey ho that's modern technology for you, sorry fed up technician here.........
foggy
Veeam Software
Posts: 21069
Liked: 2115 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: corrupted backup

Post by foggy »

A part of data blocks in the backup file could be OK, but without metadata Veeam B&R cannot order those data blocks correctly.
Gostev
Chief Product Officer
Posts: 31459
Liked: 6648 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: corrupted backup

Post by Gostev »

rstone wrote:I have found that my QNAP NAS had lost a drive which basically shouldn't matter as it has raid 5 and the share seems fine i.e. I quickly copied a TB backup to it last night. I find it very strange that it corrupts everything and not just part of the backup.
It's definitely not an uncommon situation, and it is much more likely to happen on those low end NAS... here is another example from today alone, also with QNAP NAS:
Moebius wrote:The backup primary repository (local disk, 38 TB) crashed due to a RAID problem and is being rebuilt.
Now chkdsk of the volume is showing several errors (corrupt attribute records, file record segments orphaned).
Danne84
Enthusiast
Posts: 56
Liked: never
Joined: May 12, 2014 2:05 pm
Contact:

[MERGED] : Metadata are corrupted

Post by Danne84 »

Today when i checked the backup i notice this error.
The job is a Copy job.

Failed to preprocess target Error: All instances of the storage metadata are corrupted. Agent failed to process method {Stg.OpenReadWrite}.

I have rescanned the repository and rebooted the veeam server and still the same error.
Any sugestions?
veremin
Product Manager
Posts: 20270
Liked: 2252 times
Joined: Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin
Contact:

Re: corrupted backup

Post by veremin »

This error most likely indicates that you've come across severe issues with the storage used as a target for backup copy job. More information can be found in the posts provided above. Thanks.
PTide
Product Manager
Posts: 6408
Liked: 724 times
Joined: May 19, 2015 1:46 pm
Contact:

Re: corrupted backup

Post by PTide »

For a brief overview of possible reasons and solutions please take a look at this article.
Post Reply

Who is online

Users browsing this forum: Google [Bot], ybarrap2003 and 311 guests