Corrupted Metadata

[Phaze]

Hello,

I'm having really bad problems with Veeam B&R. Iam using version 9.5 Update 2.
Let me explain our backup environment:
I have a HP server running Server 2012R2 as a Hyper-V Host. On that host I installed the Veeam Software. We are using backup to disk and also to tape. So the disk repository is an iSCSI volume on a Qnap TS253A.
That iSCSI volume is mapped to the Hyper-V server. So far so good, the backup ran fine for a while..all of a sudden I received the error: "All instances of the storage metadata are corrupted".
I opened a support case and had a technical supporter checking my environment..the entire backup data on the disk storage was unrepairable.
I have noticed that on the day the backup failed, the Qnap was restarted. I tried it again a few days later and restarted the NAS..the same error occured, all backups destroyed.

Now my questions is how to restart the Qnap without losing all my backups - thats a horrible scenario..even completed backup chains are corrupted.
Do I have to stop certain services or take the volume offline in the windows disk management?

Or even better, is there any way to stop that weekness? For example by disabling dedup or something? The Qnap was restarted while no job or anything was running, I cant see why it affects Veeam..we were using BackupExec since ever and wanted to change the product for all future installations - but that is an absolut KO behaviour.
The NAS might also shut down due to a powerloss - not all of our customers do have a ups.

Thanks for your help!

[lando_uk]

Its never a good idea to shutdown a SAN before shutting down hosts. If you cant shutdown the host, then at least disconnect the iscsi or unmount the volumes before restarting QNAP.

Although this sounds more of a QNAP issue, rather than a Veeam/Hyper-V issue. That thing is a consumer grade NAS, its not fit for business/enterprise workloads.

[Gostev]

That thing is a consumer grade NAS, its not fit for business/enterprise workloads.

That's right. I can only add that most of the corrupted backups we see in support, we see on consumer grade NAS. This is why we so highly recommend using regular servers with a bunch of drives in them instead.

That said, simply shutting down NAS should not corrupt backups, even with a customer grade NAS. When ALL backups on the storage are corrupted, it usually indicates bad storage-level corruption caused by things like malfunctioning RAID controller, bad memory, firmware bugs etc.

@Phaze one of many differences between Veeam and Backup Exec is that our backups are not plain images. We calculate and store hash for every block, and compare the restored data to its hash value to be able to detect storage-level corruption (that is, when what was stored on the storage is different from what was read back) at restore time. Now, this metadata containing hashes is stored twice in the backup file for redundancy, and also to enable us to update metadata in transactional manner - specifically to make sure we always have a "good" metadata bank even if the storage experiences shutdown during the metadata update. So, having all instances of storage metadata corrupted indicates major issues with the storage.

[Phaze]

Thank you for your answers.

Of course that Qnap is not really built for business purposes, but we are backing up about 500GB data which takes about an hour and a half..so we cant complain about performance issues.
As Gostev said, a shut down while no backup is running shouldnt cause a corruption of all backups. I also think it might be a problem with the NAS..maybe something with the new firmware (I updated after the first corruption and the problem was still the same).
I will go for a test scenario with a 2012R2 Hyper-V Server and a mounted iSCSI volume located on a similar Qnap with same Firmware..if I have the same result it's definitely no NAS problem.
I will also test what happens if I take the volume offline and disconnect the iSCSI volume..I think it wont change a thing - there is no diffenrence between a shutdown and the unmounting, for Veeam the volume is simply gone in either way.
Is there any service constantly accessing the B2D device within Veeam?
Maybe stopping dedup could help - that would explain why also the finished backup chains are corrupt. Is Veeam using one dedup-database for all backups?
Is there any recommondation on a storage device regarding model, filesystem an so on?
We really need to find the cause, as I said we ware planning to switch to Veeam from BackupExec - but for now we keep selling BackupExec.

[foggy]

Is there any service constantly accessing the B2D device within Veeam?

Constantly - no, periodically - yes (at least every 4 hours, by default).

Maybe stopping dedup could help - that would explain why also the finished backup chains are corrupt. Is Veeam using one dedup-database for all backups?

Deduplication is out of picture here.

Is there any recommondation on a storage device regarding model, filesystem an so on?

Here's a good thread regarding this.

[Phaze]

Update:

Iam currently building the same setup for testing purposes - I will give you some details later this week.

[albertwt]

Veeam Support - Case # 02184179

Hi People,

Can anyone here please clarify if the low LUN free space can also cause the Veeam backup failure?

Because I got this error when backing up my large Exchange server VM:

Processing PRODMBX03-VM Error: All instances of the storage metadata are corrupted. Failed to download disk. Shared memory connection was closed. Failed to upload disk. Agent failed to process method {DataTransfer.SyncDisk}.

VM size is 2.4 TB
HPE LUN free disk space size is just 600 GB (Thin Provision).

I'm using Veeam Backup v9.0 Enterprise Plus that uses Storage Snapshot technology on my iSCSI SAN (HPE Lefthand P4530)

[foggy]

This error is related to the target storage, not the production storage where VMs reside. Please see above.

[Phaze]

So finally I have some results.
I wasn't able to reproduce the error with my identical test setup. Whatever I did, the backups were never corrupted.
So I think it's definitely the NAS, I will change it and see what happens.

Thank you so far.

[ACL]

I am using Veeam Backup & Replication 8. My Veeam backup repository is Synology DS 1815+ with RAID 5 configuration. Since we had plug out 1 of the failed hard disk, all the backup job failed with error message below:

Error: All instances of the storage metadata are corrupted. Failed to download disk. End of file Failed to upload disk. Agent failed to process method {DataTransfer.SyncDisk}.

What is the possible cause of this error? Kindly advise

Thanks

[veremin]

Appears that you came across storage level corruption. See posts above for more details. Thanks.

[sbsfaq]

Guys - I've been investigating corruption with QNAP RAID arrays for over 6 months now - the answer is that they have a bug in their firmware that was not fixed until April, and is NOT documented in the release notes. Read more about it here http://www.sbsfaq.com/?p=4277