Comprehensive data protection for all workloads
Post Reply
balocco
Influencer
Posts: 18
Liked: 1 time
Joined: Dec 17, 2013 3:16 pm
Full Name: Fabio Bozzolo
Contact:

Data corrupted after disk fail. Any suggestions?

Post by balocco » 1 person likes this post

Hi everybody,

my Veeam Backup Repository is a QNAP TS563, with 5 4TB disks, RAID5, connected via iSCSI.
Last night all the backup jobs failed because the RAID disk was in degreded mode (because of a single disk fail) and the log said:

Code: Select all

All instances of the storage metadata are corrupted.
Failed to download disk.
Reconnectable protocol device was closed.
Failed to upload disk.
Agent failed to process method {DataTransfer.SyncDisk}.
This is the second or third time I ran into this problem, and it happens everytime a disk fails. Now I need to re-start the backup chain but the biggest problem is that I won't be able to restore data from the old backups because of data corruption.
I know you've already talked about this issue in the forum, but what do you suggest me?
Is it a QNAP problem? Is there a way to avoid it?

Thank you for all your suggestions.
foggy
Veeam Software
Posts: 21069
Liked: 2115 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Data corrupted after disk fail. Any suggestions?

Post by foggy »

Fabio, how do you tell all backup data is actually corrupt? As far as I understand, if the drive fails, the RAID goes into degraded/read-only mode until you hot-swap the failed drive, then rebuilds and you can continue using data, isn't that right? So I assume the next job run should repair the backup chain (if failure effectively occurred during the job).
balocco
Influencer
Posts: 18
Liked: 1 time
Joined: Dec 17, 2013 3:16 pm
Full Name: Fabio Bozzolo
Contact:

Re: Data corrupted after disk fail. Any suggestions?

Post by balocco »

Hi Alexander,

unfortunately things don't go that way. As soon as raid disk goes to degraded mode (one disk is missing, but data are still accessible), Veeam starts to log that backups metadata are corrupted. Even after raid rebuilding with a new disk, metadata are still corrupted. All I can do is to restart a new backup chain, deleting the old backup from Veeam configuration.
I'm not assuming this is a Veeam issue, I think this is about QNAP. This happens everytime a disk fails and raid goes to degraded mode (and of course also raid rebuild).
Have you ever heard about this problem?
foggy
Veeam Software
Posts: 21069
Liked: 2115 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Data corrupted after disk fail. Any suggestions?

Post by foggy »

Well, in this case it does indeed mean that the chain cannot be used further - even if the backups themselves might be ok, there's no valid copy of metadata (out of the two we always keep for redundancy) to make sense of them. So yes, I'd pay attention at the QNAP side of things.
Yannis
Service Provider
Posts: 3
Liked: 2 times
Joined: Jun 13, 2016 7:51 am
Full Name: Yannis
Contact:

Re: Data corrupted after disk fail. Any suggestions?

Post by Yannis »

Hi,
We encountered the same problem by 3 time with QNAP's NAS with repository in SMB file share.
Each time, when a disk failed, metadata are corrupted even after a successfull RAID rebuilt.
We doesn't find any solution about this and we hope to find an explaination.
Furthermore as a workaround, we don't sale and use any QNAP devices...
So, you are not alone :-(
regards,
Yannis.
fbaschin@asion-it.de
Lurker
Posts: 2
Liked: never
Joined: Jul 05, 2017 12:19 pm
Full Name: Frank Baschin
Contact:

Re: Data corrupted after disk fail. Any suggestions?

Post by fbaschin@asion-it.de »

HI,

I can confirm this problem. Since last 12 months we had also over 5 same issues.
Alle times a disk in raid5 on a qnap fails, replaced and raid restores without any errors
ALL files after that where okay, except the veeam repositoris.
mmeurer
Lurker
Posts: 1
Liked: never
Joined: Jun 29, 2015 2:58 pm
Contact:

Re: Data corrupted after disk fail. Any suggestions?

Post by mmeurer »

Hi all,
we have the same problem with Synology's NAS, but what I don't understand it, we have two backup jobs
to the same backup repositorie (Synology NAS with RAID 5) one smb share. One backup chain works great with failed
disk and after rebuild the raid, the other backup chain metadata goes corrupt when disk failed.

Regards,
Marco
sbsfaq
Novice
Posts: 4
Liked: 4 times
Joined: Jul 17, 2017 1:36 pm
Full Name: Wayne Small
Contact:

Re: Data corrupted after disk fail. Any suggestions?

Post by sbsfaq » 2 people like this post

Guys - I've been investigating corruption with QNAP RAID arrays for over 6 months now - the answer is that they have a bug in their firmware that was not fixed until April, and is NOT documented in the release notes. Read more about it here http://www.sbsfaq.com/?p=4277
fbaschin@asion-it.de
Lurker
Posts: 2
Liked: never
Joined: Jul 05, 2017 12:19 pm
Full Name: Frank Baschin
Contact:

Re: Data corrupted after disk fail. Any suggestions?

Post by fbaschin@asion-it.de »

Yesterday I also got an email from QNAP with brand new information about our case. Take a look on the link.
https://www.qnap.com/en/technical-advis ... -201707-01

Its impossible, but the truth.
In the last years there was a major bug in the raid5/6 implementation of QNAP NAS systems.
albertwt
Veeam Legend
Posts: 879
Liked: 46 times
Joined: Nov 05, 2009 12:24 pm
Location: Sydney, NSW
Contact:

Re: Data corrupted after disk fail. Any suggestions?

Post by albertwt »

Yes,

I also having the same intermittent backup corruption issue with my Veeam backup copy job which uses QNAP TS 1200 series
Failed to preprocess target Error: Unable to save meta. See log for details.
Unable to create target storage, processing will be retried (retry 1 out of 6)
Failed to preprocess target
Windows CheckDisk reports some issue with the corrupted files hence some of them gets deleted when I run it with /F parameter.

I wonder if this is caused by the Windows NTFS iSCSI LUN that is too big 53 TB as one drive or this is due to this bug as reported by Anton Gostev https://www.crn.com.au/news/qnap-keeps- ... ata-468923
--
/* Veeam software enthusiast user & supporter ! */
sbsfaq
Novice
Posts: 4
Liked: 4 times
Joined: Jul 17, 2017 1:36 pm
Full Name: Wayne Small
Contact:

Re: Data corrupted after disk fail. Any suggestions?

Post by sbsfaq »

Albertwt - the bug I reported to QNAP only applied if you had a disk fail in your RAID5/6 array. If there was no failure then your data is not corrupted by this issue.
Post Reply

Who is online

Users browsing this forum: bct44, Bing [Bot], Google [Bot] and 182 guests