Availability for the Always-On Enterprise
Post Reply
Phaze
Novice
Posts: 4
Liked: never
Joined: May 31, 2017 10:44 am
Contact:

Corrupted Metadata

Post by Phaze » May 31, 2017 11:01 am

Hello,

I'm having really bad problems with Veeam B&R. Iam using version 9.5 Update 2.
Let me explain our backup environment:
I have a HP server running Server 2012R2 as a Hyper-V Host. On that host I installed the Veeam Software. We are using backup to disk and also to tape. So the disk repository is an iSCSI volume on a Qnap TS253A.
That iSCSI volume is mapped to the Hyper-V server. So far so good, the backup ran fine for a while..all of a sudden I received the error: "All instances of the storage metadata are corrupted".
I opened a support case and had a technical supporter checking my environment..the entire backup data on the disk storage was unrepairable.
I have noticed that on the day the backup failed, the Qnap was restarted. I tried it again a few days later and restarted the NAS..the same error occured, all backups destroyed.

Now my questions is how to restart the Qnap without losing all my backups - thats a horrible scenario..even completed backup chains are corrupted.
Do I have to stop certain services or take the volume offline in the windows disk management?

Or even better, is there any way to stop that weekness? For example by disabling dedup or something? The Qnap was restarted while no job or anything was running, I cant see why it affects Veeam..we were using BackupExec since ever and wanted to change the product for all future installations - but that is an absolut KO behaviour.
The NAS might also shut down due to a powerloss - not all of our customers do have a ups.

Thanks for your help!

lando_uk
Expert
Posts: 277
Liked: 20 times
Joined: Oct 17, 2013 10:02 am
Full Name: Mark
Location: UK
Contact:

Re: Corrupted Metadata

Post by lando_uk » May 31, 2017 3:28 pm 1 person likes this post

Its never a good idea to shutdown a SAN before shutting down hosts. If you cant shutdown the host, then at least disconnect the iscsi or unmount the volumes before restarting QNAP.

Although this sounds more of a QNAP issue, rather than a Veeam/Hyper-V issue. That thing is a consumer grade NAS, its not fit for business/enterprise workloads.

Gostev
Veeam Software
Posts: 22808
Liked: 2801 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Corrupted Metadata

Post by Gostev » May 31, 2017 3:38 pm

lando_uk wrote:That thing is a consumer grade NAS, its not fit for business/enterprise workloads.
That's right. I can only add that most of the corrupted backups we see in support, we see on consumer grade NAS. This is why we so highly recommend using regular servers with a bunch of drives in them instead.

That said, simply shutting down NAS should not corrupt backups, even with a customer grade NAS. When ALL backups on the storage are corrupted, it usually indicates bad storage-level corruption caused by things like malfunctioning RAID controller, bad memory, firmware bugs etc.

@Phaze one of many differences between Veeam and Backup Exec is that our backups are not plain images. We calculate and store hash for every block, and compare the restored data to its hash value to be able to detect storage-level corruption (that is, when what was stored on the storage is different from what was read back) at restore time. Now, this metadata containing hashes is stored twice in the backup file for redundancy, and also to enable us to update metadata in transactional manner - specifically to make sure we always have a "good" metadata bank even if the storage experiences shutdown during the metadata update. So, having all instances of storage metadata corrupted indicates major issues with the storage.

Phaze
Novice
Posts: 4
Liked: never
Joined: May 31, 2017 10:44 am
Contact:

Re: Corrupted Metadata

Post by Phaze » Jun 01, 2017 8:03 am

Thank you for your answers.

Of course that Qnap is not really built for business purposes, but we are backing up about 500GB data which takes about an hour and a half..so we cant complain about performance issues.
As Gostev said, a shut down while no backup is running shouldnt cause a corruption of all backups. I also think it might be a problem with the NAS..maybe something with the new firmware (I updated after the first corruption and the problem was still the same).
I will go for a test scenario with a 2012R2 Hyper-V Server and a mounted iSCSI volume located on a similar Qnap with same Firmware..if I have the same result it's definitely no NAS problem.
I will also test what happens if I take the volume offline and disconnect the iSCSI volume..I think it wont change a thing - there is no diffenrence between a shutdown and the unmounting, for Veeam the volume is simply gone in either way.
Is there any service constantly accessing the B2D device within Veeam?
Maybe stopping dedup could help - that would explain why also the finished backup chains are corrupt. Is Veeam using one dedup-database for all backups?
Is there any recommondation on a storage device regarding model, filesystem an so on?
We really need to find the cause, as I said we ware planning to switch to Veeam from BackupExec - but for now we keep selling BackupExec.

foggy
Veeam Software
Posts: 16691
Liked: 1343 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Corrupted Metadata

Post by foggy » Jun 01, 2017 12:44 pm

Phaze wrote:Is there any service constantly accessing the B2D device within Veeam?
Constantly - no, periodically - yes (at least every 4 hours, by default).
Phaze wrote:Maybe stopping dedup could help - that would explain why also the finished backup chains are corrupt. Is Veeam using one dedup-database for all backups?
Deduplication is out of picture here.
Phaze wrote:Is there any recommondation on a storage device regarding model, filesystem an so on?
Here's a good thread regarding this.

Phaze
Novice
Posts: 4
Liked: never
Joined: May 31, 2017 10:44 am
Contact:

Re: Corrupted Metadata

Post by Phaze » Jun 06, 2017 7:49 am

Update:

Iam currently building the same setup for testing purposes - I will give you some details later this week.

albertwt
Expert
Posts: 626
Liked: 20 times
Joined: Nov 05, 2009 12:24 pm
Location: Sydney, NSW
Contact:

[MERGED] All instances of the storage metadata are corrupted

Post by albertwt » Jun 09, 2017 2:36 pm

Veeam Support - Case # 02184179

Hi People,

Can anyone here please clarify if the low LUN free space can also cause the Veeam backup failure?

Because I got this error when backing up my large Exchange server VM:
Processing PRODMBX03-VM Error: All instances of the storage metadata are corrupted. Failed to download disk. Shared memory connection was closed. Failed to upload disk. Agent failed to process method {DataTransfer.SyncDisk}.
VM size is 2.4 TB
HPE LUN free disk space size is just 600 GB (Thin Provision).

I'm using Veeam Backup v9.0 Enterprise Plus that uses Storage Snapshot technology on my iSCSI SAN (HPE Lefthand P4530)
--
/* Veeam software enthusiast user & supporter ! */

foggy
Veeam Software
Posts: 16691
Liked: 1343 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Corrupted Metadata

Post by foggy » Jun 09, 2017 2:42 pm

This error is related to the target storage, not the production storage where VMs reside. Please see above.

Phaze
Novice
Posts: 4
Liked: never
Joined: May 31, 2017 10:44 am
Contact:

Re: Corrupted Metadata

Post by Phaze » Jun 19, 2017 2:21 pm

So finally I have some results.
I wasn't able to reproduce the error with my identical test setup. Whatever I did, the backups were never corrupted.
So I think it's definitely the NAS, I will change it and see what happens.

Thank you so far.

ACL
Lurker
Posts: 1
Liked: never
Joined: Jun 23, 2017 5:35 am
Full Name: ACL
Contact:

[MERGED] Veeam Storage Metadata Corrupted

Post by ACL » Jun 23, 2017 5:44 am

I am using Veeam Backup & Replication 8. My Veeam backup repository is Synology DS 1815+ with RAID 5 configuration. Since we had plug out 1 of the failed hard disk, all the backup job failed with error message below:

Error: All instances of the storage metadata are corrupted. Failed to download disk. End of file Failed to upload disk. Agent failed to process method {DataTransfer.SyncDisk}.

What is the possible cause of this error? Kindly advise

Thanks

v.Eremin
Veeam Software
Posts: 15077
Liked: 1133 times
Joined: Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin
Contact:

Re: Corrupted Metadata

Post by v.Eremin » Jun 23, 2017 7:27 am

Appears that you came across storage level corruption. See posts above for more details. Thanks.

sbsfaq
Novice
Posts: 4
Liked: 4 times
Joined: Jul 17, 2017 1:36 pm
Full Name: Wayne Small
Contact:

Re: Corrupted Metadata

Post by sbsfaq » Jul 17, 2017 2:16 pm 2 people like this post

Guys - I've been investigating corruption with QNAP RAID arrays for over 6 months now - the answer is that they have a bug in their firmware that was not fixed until April, and is NOT documented in the release notes. Read more about it here http://www.sbsfaq.com/?p=4277

Post Reply

Who is online

Users browsing this forum: No registered users and 28 guests