Backup copy job failing with *metadata are corrupted* error

Availability for the Always-On Enterprise

Backup copy job failing with *metadata are corrupted* error

Veeam Logoby paoloatuniurb » Thu Apr 20, 2017 2:15 pm

Hello.

Support case #02133769

I have a backup job failing on a few (4 over 111) vm.

20/04/2017 09:39:17 :: Copying restore point 19/04/2017 20:33:35 from backup repository Repository 16.24
20/04/2017 09:39:27 :: Error: All instances of the storage metadata are corrupted.
Failed to restore file from local backup. VFS link: [summary.xml]. Target file: [MemFs://frontend::CDataTransferCommandSet::RestoreText_{8656b8b9-1119-4026-807a-58c069ce2f7f}]. CHMOD mask: [666].
Agent failed to process method {DataTransfer.RestoreText}.


The *metadata are corrupted* error message has been, of course, already discussed but, figure out,
each of the 4 vm can be successfully restored from the same backup point that is claimed to be corrupted.

I disabled ad enabled again the backup copy job but cannot clear the error. Of course I started a new chain too ... nope.

Any one experienced this behaviour?

TyL on advance. Paolo.
paoloatuniurb
Novice
 
Posts: 6
Liked: never
Joined: Sat Mar 15, 2014 4:43 pm
Full Name: Paolo Cecchini

Re: Backup copy job failing with *metadata are corrupted* er

Veeam Logoby foggy » Thu Apr 20, 2017 2:35 pm

paoloatuniurb wrote:Of course I started a new chain too ... nope.

Do you mean the error persists after starting the backup chain anew?
foggy
Veeam Software
 
Posts: 14605
Liked: 1064 times
Joined: Mon Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson

Re: Backup copy job failing with *metadata are corrupted* er

Veeam Logoby paoloatuniurb » Thu Apr 20, 2017 8:18 pm

I suppose I'd to tell you all the long story.

I have a couple backup job, name them as an example A,B,C,D,E ... they're forward incremental with weekly active backup.

Then I have some offsite backup copy jobs. They collect *from infrastructure* vm so vm belongs to backup job A,B,C ...

I do have 4 failing vm's belonging to backup jobs A & B. If I restore them from A,B it succeed. I forced an active full on A,B, usually this clear the backup copy error, but this time nope. Of course I restarted the backup copy, too.

So I suppose YES, I mean the error persist after starting the backup chain anew ... or I mean, at least, the usual work-around, force a full, don't work this time.

:)

Cu. Paolo.
paoloatuniurb
Novice
 
Posts: 6
Liked: never
Joined: Sat Mar 15, 2014 4:43 pm
Full Name: Paolo Cecchini

Re: Backup copy job failing with *metadata are corrupted* er

Veeam Logoby Gostev » Thu Apr 20, 2017 8:24 pm

paoloatuniurb wrote:20/04/2017 09:39:27 :: Error: All instances of the storage metadata are corrupted.

This error indicates massive storage-level corruption. You should create the new Backup Copy job and point it to a different storage device.
Gostev
Veeam Software
 
Posts: 21365
Liked: 2337 times
Joined: Sun Jan 01, 2006 1:01 am
Full Name: Anton Gostev

Re: Backup copy job failing with *metadata are corrupted* er

Veeam Logoby paoloatuniurb » Thu Apr 20, 2017 8:49 pm

Yes. I feel like you, and this is the worse side of the problem. The device is (better said, was) a Fujitsu CS800 that has been in production as main backup storage for years and never failed. But now I scaled it down to second line and formatted with Windows server 2016 core and dedup feature.

So the hardware is darn good. The software is windows. It's failing? May I trust in w2016 and deduplication?

Ty. PaoloC
paoloatuniurb
Novice
 
Posts: 6
Liked: never
Joined: Sat Mar 15, 2014 4:43 pm
Full Name: Paolo Cecchini

Re: Backup copy job failing with *metadata are corrupted* er

Veeam Logoby paoloatuniurb » Fri Apr 21, 2017 9:22 am

paoloatuniurb wrote: But now I scaled it down to second line and formatted with Windows server 2016 core and dedup feature.


Well, I'm planning to restart from scratch and go for refs .. being a backup copy .. looks better.

Paolo
paoloatuniurb
Novice
 
Posts: 6
Liked: never
Joined: Sat Mar 15, 2014 4:43 pm
Full Name: Paolo Cecchini

Re: Backup copy job failing with *metadata are corrupted* er

Veeam Logoby foggy » Fri Apr 21, 2017 10:31 am

paoloatuniurb wrote:So I suppose YES, I mean the error persist after starting the backup chain anew ... or I mean, at least, the usual work-around, force a full, don't work this time.

Btw, to start a backup chain from scratch, you need to trigger manual active full on it, active full on original backup job doesn't affect the backup copy job chain.
foggy
Veeam Software
 
Posts: 14605
Liked: 1064 times
Joined: Mon Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson

Re: Backup copy job failing with *metadata are corrupted* er

Veeam Logoby paoloatuniurb » Fri Apr 21, 2017 1:59 pm

I meanwhile started the *architecture* from scratch :wink:

Delete job, delete repository, proxy, server, format with refs (was ntfs).
Add server et cetera, create new backup copy job.

CU. PaoloC
paoloatuniurb
Novice
 
Posts: 6
Liked: never
Joined: Sat Mar 15, 2014 4:43 pm
Full Name: Paolo Cecchini

Re: Backup copy job failing with *metadata are corrupted* er

Veeam Logoby Gostev » Fri Apr 21, 2017 2:37 pm

paoloatuniurb wrote:May I trust in w2016 and deduplication?

Not at this time. There is currently a known issue data corruption issue with Windows Server 2016 dedupe... we've recently seen the same corruption in our own lab and notified Microsoft about one. There seems to be a number of customers having this issue at this time.
Gostev
Veeam Software
 
Posts: 21365
Liked: 2337 times
Joined: Sun Jan 01, 2006 1:01 am
Full Name: Anton Gostev

Re: Backup copy job failing with *metadata are corrupted* er

Veeam Logoby mrbbs » Tue May 16, 2017 12:20 pm

Any updates on this issue?
A couple of weeks ago I read in your weekly newsletter this wasn't resolved by Microsoft yet.
Is it known if the patches from May have resolved the issue?

Kind regards,
Kevin
mrbbs
Novice
 
Posts: 7
Liked: never
Joined: Tue Sep 06, 2016 11:00 am
Full Name: mr bbs

Re: Backup copy job failing with *metadata are corrupted* er

Veeam Logoby mrbbs » Tue May 16, 2017 12:53 pm

mrbbs
Novice
 
Posts: 7
Liked: never
Joined: Tue Sep 06, 2016 11:00 am
Full Name: mr bbs


Return to Veeam Backup & Replication



Who is online

Users browsing this forum: No registered users and 39 guests