Backup copy job failing with *metadata are corrupted* error

[paoloatuniurb]

Hello.

Support case #02133769

I have a backup job failing on a few (4 over 111) vm.

20/04/2017 09:39:17 :: Copying restore point 19/04/2017 20:33:35 from backup repository Repository 16.24
20/04/2017 09:39:27 :: Error: All instances of the storage metadata are corrupted.
Failed to restore file from local backup. VFS link: [summary.xml]. Target file: [MemFs://frontend::CDataTransferCommandSet::RestoreText_{8656b8b9-1119-4026-807a-58c069ce2f7f}]. CHMOD mask: [666].
Agent failed to process method {DataTransfer.RestoreText}.


The *metadata are corrupted* error message has been, of course, already discussed but, figure out,
each of the 4 vm can be successfully restored from the same backup point that is claimed to be corrupted.

I disabled ad enabled again the backup copy job but cannot clear the error. Of course I started a new chain too ... nope.

Any one experienced this behaviour?

TyL on advance. Paolo.

[foggy]

Of course I started a new chain too ... nope.

Do you mean the error persists after starting the backup chain anew?

[paoloatuniurb]

I suppose I'd to tell you all the long story.

I have a couple backup job, name them as an example A,B,C,D,E ... they're forward incremental with weekly active backup.

Then I have some offsite backup copy jobs. They collect *from infrastructure* vm so vm belongs to backup job A,B,C ...

I do have 4 failing vm's belonging to backup jobs A & B. If I restore them from A,B it succeed. I forced an active full on A,B, usually this clear the backup copy error, but this time nope. Of course I restarted the backup copy, too.

So I suppose YES, I mean the error persist after starting the backup chain anew ... or I mean, at least, the usual work-around, force a full, don't work this time.



Cu. Paolo.

[Gostev]

20/04/2017 09:39:27 :: Error: All instances of the storage metadata are corrupted.

This error indicates massive storage-level corruption. You should create the new Backup Copy job and point it to a different storage device.

[paoloatuniurb]

Yes. I feel like you, and this is the worse side of the problem. The device is (better said, was) a Fujitsu CS800 that has been in production as main backup storage for years and never failed. But now I scaled it down to second line and formatted with Windows server 2016 core and dedup feature.

So the hardware is darn good. The software is windows. It's failing? May I trust in w2016 and deduplication?

Ty. PaoloC

[paoloatuniurb]

But now I scaled it down to second line and formatted with Windows server 2016 core and dedup feature.

Well, I'm planning to restart from scratch and go for refs .. being a backup copy .. looks better.

Paolo

[foggy]

So I suppose YES, I mean the error persist after starting the backup chain anew ... or I mean, at least, the usual work-around, force a full, don't work this time.

Btw, to start a backup chain from scratch, you need to trigger manual active full on it, active full on original backup job doesn't affect the backup copy job chain.

[paoloatuniurb]

I meanwhile started the *architecture* from scratch

Delete job, delete repository, proxy, server, format with refs (was ntfs).
Add server et cetera, create new backup copy job.

CU. PaoloC

[Gostev]

May I trust in w2016 and deduplication?

Not at this time. There is currently a known issue data corruption issue with Windows Server 2016 dedupe... we've recently seen the same corruption in our own lab and notified Microsoft about one. There seems to be a number of customers having this issue at this time.

[mrbbs]

Any updates on this issue?
A couple of weeks ago I read in your weekly newsletter this wasn't resolved by Microsoft yet.
Is it known if the patches from May have resolved the issue?

Kind regards,
Kevin

[mrbbs]

Another thread with this same issue.

veeam-backup-replication-f2/disk-errors ... 77-15.html