Data corrupted after disk fail. Any suggestions?

[balocco]

Hi everybody,

my Veeam Backup Repository is a QNAP TS563, with 5 4TB disks, RAID5, connected via iSCSI.
Last night all the backup jobs failed because the RAID disk was in degreded mode (because of a single disk fail) and the log said:

Code: Select all

All instances of the storage metadata are corrupted.
Failed to download disk.
Reconnectable protocol device was closed.
Failed to upload disk.
Agent failed to process method {DataTransfer.SyncDisk}.

This is the second or third time I ran into this problem, and it happens everytime a disk fails. Now I need to re-start the backup chain but the biggest problem is that I won't be able to restore data from the old backups because of data corruption.
I know you've already talked about this issue in the forum, but what do you suggest me?
Is it a QNAP problem? Is there a way to avoid it?

Thank you for all your suggestions.

[foggy]

Fabio, how do you tell all backup data is actually corrupt? As far as I understand, if the drive fails, the RAID goes into degraded/read-only mode until you hot-swap the failed drive, then rebuilds and you can continue using data, isn't that right? So I assume the next job run should repair the backup chain (if failure effectively occurred during the job).

[balocco]

Hi Alexander,

unfortunately things don't go that way. As soon as raid disk goes to degraded mode (one disk is missing, but data are still accessible), Veeam starts to log that backups metadata are corrupted. Even after raid rebuilding with a new disk, metadata are still corrupted. All I can do is to restart a new backup chain, deleting the old backup from Veeam configuration.
I'm not assuming this is a Veeam issue, I think this is about QNAP. This happens everytime a disk fails and raid goes to degraded mode (and of course also raid rebuild).
Have you ever heard about this problem?

[foggy]

Well, in this case it does indeed mean that the chain cannot be used further - even if the backups themselves might be ok, there's no valid copy of metadata (out of the two we always keep for redundancy) to make sense of them. So yes, I'd pay attention at the QNAP side of things.

[Yannis]

Hi,
We encountered the same problem by 3 time with QNAP's NAS with repository in SMB file share.
Each time, when a disk failed, metadata are corrupted even after a successfull RAID rebuilt.
We doesn't find any solution about this and we hope to find an explaination.
Furthermore as a workaround, we don't sale and use any QNAP devices...
So, you are not alone
regards,
Yannis.

[fbaschin@asion-it.de]

HI,

I can confirm this problem. Since last 12 months we had also over 5 same issues.
Alle times a disk in raid5 on a qnap fails, replaced and raid restores without any errors
ALL files after that where okay, except the veeam repositoris.

[mmeurer]

Hi all,
we have the same problem with Synology's NAS, but what I don't understand it, we have two backup jobs
to the same backup repositorie (Synology NAS with RAID 5) one smb share. One backup chain works great with failed
disk and after rebuild the raid, the other backup chain metadata goes corrupt when disk failed.

Regards,
Marco

[sbsfaq]

Guys - I've been investigating corruption with QNAP RAID arrays for over 6 months now - the answer is that they have a bug in their firmware that was not fixed until April, and is NOT documented in the release notes. Read more about it here http://www.sbsfaq.com/?p=4277

[fbaschin@asion-it.de]

Yesterday I also got an email from QNAP with brand new information about our case. Take a look on the link.
https://www.qnap.com/en/technical-advis ... -201707-01

Its impossible, but the truth.
In the last years there was a major bug in the raid5/6 implementation of QNAP NAS systems.

[albertwt]

Yes,

I also having the same intermittent backup corruption issue with my Veeam backup copy job which uses QNAP TS 1200 series

Failed to preprocess target Error: Unable to save meta. See log for details.
Unable to create target storage, processing will be retried (retry 1 out of 6)
Failed to preprocess target

Windows CheckDisk reports some issue with the corrupted files hence some of them gets deleted when I run it with /F parameter.

I wonder if this is caused by the Windows NTFS iSCSI LUN that is too big 53 TB as one drive or this is due to this bug as reported by Anton Gostev https://www.crn.com.au/news/qnap-keeps- ... ata-468923

[sbsfaq]

Albertwt - the bug I reported to QNAP only applied if you had a disk fail in your RAID5/6 array. If there was no failure then your data is not corrupted by this issue.