Corrupted Metadata

Availability for the Always-On Enterprise

Corrupted Metadata

Veeam Logoby Phaze » Wed May 31, 2017 11:01 am

Hello,

I'm having really bad problems with Veeam B&R. Iam using version 9.5 Update 2.
Let me explain our backup environment:
I have a HP server running Server 2012R2 as a Hyper-V Host. On that host I installed the Veeam Software. We are using backup to disk and also to tape. So the disk repository is an iSCSI volume on a Qnap TS253A.
That iSCSI volume is mapped to the Hyper-V server. So far so good, the backup ran fine for a while..all of a sudden I received the error: "All instances of the storage metadata are corrupted".
I opened a support case and had a technical supporter checking my environment..the entire backup data on the disk storage was unrepairable.
I have noticed that on the day the backup failed, the Qnap was restarted. I tried it again a few days later and restarted the NAS..the same error occured, all backups destroyed.

Now my questions is how to restart the Qnap without losing all my backups - thats a horrible scenario..even completed backup chains are corrupted.
Do I have to stop certain services or take the volume offline in the windows disk management?

Or even better, is there any way to stop that weekness? For example by disabling dedup or something? The Qnap was restarted while no job or anything was running, I cant see why it affects Veeam..we were using BackupExec since ever and wanted to change the product for all future installations - but that is an absolut KO behaviour.
The NAS might also shut down due to a powerloss - not all of our customers do have a ups.

Thanks for your help!
Phaze
Novice
 
Posts: 4
Liked: never
Joined: Wed May 31, 2017 10:44 am

Re: Corrupted Metadata

Veeam Logoby lando_uk » Wed May 31, 2017 3:28 pm 1 person likes this post

Its never a good idea to shutdown a SAN before shutting down hosts. If you cant shutdown the host, then at least disconnect the iscsi or unmount the volumes before restarting QNAP.

Although this sounds more of a QNAP issue, rather than a Veeam/Hyper-V issue. That thing is a consumer grade NAS, its not fit for business/enterprise workloads.
lando_uk
Expert
 
Posts: 236
Liked: 17 times
Joined: Thu Oct 17, 2013 10:02 am
Full Name: Mark

Re: Corrupted Metadata

Veeam Logoby Gostev » Wed May 31, 2017 3:38 pm

lando_uk wrote:That thing is a consumer grade NAS, its not fit for business/enterprise workloads.

That's right. I can only add that most of the corrupted backups we see in support, we see on consumer grade NAS. This is why we so highly recommend using regular servers with a bunch of drives in them instead.

That said, simply shutting down NAS should not corrupt backups, even with a customer grade NAS. When ALL backups on the storage are corrupted, it usually indicates bad storage-level corruption caused by things like malfunctioning RAID controller, bad memory, firmware bugs etc.

@Phaze one of many differences between Veeam and Backup Exec is that our backups are not plain images. We calculate and store hash for every block, and compare the restored data to its hash value to be able to detect storage-level corruption (that is, when what was stored on the storage is different from what was read back) at restore time. Now, this metadata containing hashes is stored twice in the backup file for redundancy, and also to enable us to update metadata in transactional manner - specifically to make sure we always have a "good" metadata bank even if the storage experiences shutdown during the metadata update. So, having all instances of storage metadata corrupted indicates major issues with the storage.
Gostev
Veeam Software
 
Posts: 21390
Liked: 2349 times
Joined: Sun Jan 01, 2006 1:01 am
Location: Baar, Switzerland

Re: Corrupted Metadata

Veeam Logoby Phaze » Thu Jun 01, 2017 8:03 am

Thank you for your answers.

Of course that Qnap is not really built for business purposes, but we are backing up about 500GB data which takes about an hour and a half..so we cant complain about performance issues.
As Gostev said, a shut down while no backup is running shouldnt cause a corruption of all backups. I also think it might be a problem with the NAS..maybe something with the new firmware (I updated after the first corruption and the problem was still the same).
I will go for a test scenario with a 2012R2 Hyper-V Server and a mounted iSCSI volume located on a similar Qnap with same Firmware..if I have the same result it's definitely no NAS problem.
I will also test what happens if I take the volume offline and disconnect the iSCSI volume..I think it wont change a thing - there is no diffenrence between a shutdown and the unmounting, for Veeam the volume is simply gone in either way.
Is there any service constantly accessing the B2D device within Veeam?
Maybe stopping dedup could help - that would explain why also the finished backup chains are corrupt. Is Veeam using one dedup-database for all backups?
Is there any recommondation on a storage device regarding model, filesystem an so on?
We really need to find the cause, as I said we ware planning to switch to Veeam from BackupExec - but for now we keep selling BackupExec.
Phaze
Novice
 
Posts: 4
Liked: never
Joined: Wed May 31, 2017 10:44 am

Re: Corrupted Metadata

Veeam Logoby foggy » Thu Jun 01, 2017 12:44 pm

Phaze wrote:Is there any service constantly accessing the B2D device within Veeam?

Constantly - no, periodically - yes (at least every 4 hours, by default).

Phaze wrote:Maybe stopping dedup could help - that would explain why also the finished backup chains are corrupt. Is Veeam using one dedup-database for all backups?

Deduplication is out of picture here.

Phaze wrote:Is there any recommondation on a storage device regarding model, filesystem an so on?

Here's a good thread regarding this.
foggy
Veeam Software
 
Posts: 14742
Liked: 1080 times
Joined: Mon Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson

Re: Corrupted Metadata

Veeam Logoby Phaze » Tue Jun 06, 2017 7:49 am

Update:

Iam currently building the same setup for testing purposes - I will give you some details later this week.
Phaze
Novice
 
Posts: 4
Liked: never
Joined: Wed May 31, 2017 10:44 am

[MERGED] All instances of the storage metadata are corrupted

Veeam Logoby albertwt » Fri Jun 09, 2017 2:36 pm

Veeam Support - Case # 02184179

Hi People,

Can anyone here please clarify if the low LUN free space can also cause the Veeam backup failure?

Because I got this error when backing up my large Exchange server VM:

Processing PRODMBX03-VM Error: All instances of the storage metadata are corrupted. Failed to download disk. Shared memory connection was closed. Failed to upload disk. Agent failed to process method {DataTransfer.SyncDisk}.


VM size is 2.4 TB
HPE LUN free disk space size is just 600 GB (Thin Provision).

I'm using Veeam Backup v9.0 Enterprise Plus that uses Storage Snapshot technology on my iSCSI SAN (HPE Lefthand P4530)
--
/* Veeam software enthusiast user & supporter ! */
albertwt
Expert
 
Posts: 607
Liked: 19 times
Joined: Thu Nov 05, 2009 12:24 pm
Location: Sydney, NSW

Re: Corrupted Metadata

Veeam Logoby foggy » Fri Jun 09, 2017 2:42 pm

This error is related to the target storage, not the production storage where VMs reside. Please see above.
foggy
Veeam Software
 
Posts: 14742
Liked: 1080 times
Joined: Mon Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson

Re: Corrupted Metadata

Veeam Logoby Phaze » Mon Jun 19, 2017 2:21 pm

So finally I have some results.
I wasn't able to reproduce the error with my identical test setup. Whatever I did, the backups were never corrupted.
So I think it's definitely the NAS, I will change it and see what happens.

Thank you so far.
Phaze
Novice
 
Posts: 4
Liked: never
Joined: Wed May 31, 2017 10:44 am

[MERGED] Veeam Storage Metadata Corrupted

Veeam Logoby ACL » Fri Jun 23, 2017 5:44 am

I am using Veeam Backup & Replication 8. My Veeam backup repository is Synology DS 1815+ with RAID 5 configuration. Since we had plug out 1 of the failed hard disk, all the backup job failed with error message below:

Error: All instances of the storage metadata are corrupted. Failed to download disk. End of file Failed to upload disk. Agent failed to process method {DataTransfer.SyncDisk}.

What is the possible cause of this error? Kindly advise

Thanks
ACL
Lurker
 
Posts: 1
Liked: never
Joined: Fri Jun 23, 2017 5:35 am
Full Name: ACL

Re: Corrupted Metadata

Veeam Logoby v.Eremin » Fri Jun 23, 2017 7:27 am

Appears that you came across storage level corruption. See posts above for more details. Thanks.
v.Eremin
Veeam Software
 
Posts: 13266
Liked: 968 times
Joined: Fri Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin

Re: Corrupted Metadata

Veeam Logoby sbsfaq » Mon Jul 17, 2017 2:16 pm 2 people like this post

Guys - I've been investigating corruption with QNAP RAID arrays for over 6 months now - the answer is that they have a bug in their firmware that was not fixed until April, and is NOT documented in the release notes. Read more about it here http://www.sbsfaq.com/?p=4277
sbsfaq
Novice
 
Posts: 4
Liked: 4 times
Joined: Mon Jul 17, 2017 1:36 pm
Full Name: Wayne Small


Return to Veeam Backup & Replication



Who is online

Users browsing this forum: Bing [Bot], Google [Bot], opg70 and 22 guests