Data corrupted after disk fail. Any suggestions?

Availability for the Always-On Enterprise

Data corrupted after disk fail. Any suggestions?

Veeam Logoby balocco » Wed Nov 30, 2016 8:16 am 1 person likes this post

Hi everybody,

my Veeam Backup Repository is a QNAP TS563, with 5 4TB disks, RAID5, connected via iSCSI.
Last night all the backup jobs failed because the RAID disk was in degreded mode (because of a single disk fail) and the log said:

Code: Select all
All instances of the storage metadata are corrupted.
Failed to download disk.
Reconnectable protocol device was closed.
Failed to upload disk.
Agent failed to process method {DataTransfer.SyncDisk}.


This is the second or third time I ran into this problem, and it happens everytime a disk fails. Now I need to re-start the backup chain but the biggest problem is that I won't be able to restore data from the old backups because of data corruption.
I know you've already talked about this issue in the forum, but what do you suggest me?
Is it a QNAP problem? Is there a way to avoid it?

Thank you for all your suggestions.
balocco
Influencer
 
Posts: 16
Liked: 1 time
Joined: Tue Dec 17, 2013 3:16 pm
Full Name: Fabio Bozzolo

Re: Data corrupted after disk fail. Any suggestions?

Veeam Logoby foggy » Wed Nov 30, 2016 5:33 pm

Fabio, how do you tell all backup data is actually corrupt? As far as I understand, if the drive fails, the RAID goes into degraded/read-only mode until you hot-swap the failed drive, then rebuilds and you can continue using data, isn't that right? So I assume the next job run should repair the backup chain (if failure effectively occurred during the job).
foggy
Veeam Software
 
Posts: 14588
Liked: 1062 times
Joined: Mon Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson

Re: Data corrupted after disk fail. Any suggestions?

Veeam Logoby balocco » Wed Nov 30, 2016 6:16 pm

Hi Alexander,

unfortunately things don't go that way. As soon as raid disk goes to degraded mode (one disk is missing, but data are still accessible), Veeam starts to log that backups metadata are corrupted. Even after raid rebuilding with a new disk, metadata are still corrupted. All I can do is to restart a new backup chain, deleting the old backup from Veeam configuration.
I'm not assuming this is a Veeam issue, I think this is about QNAP. This happens everytime a disk fails and raid goes to degraded mode (and of course also raid rebuild).
Have you ever heard about this problem?
balocco
Influencer
 
Posts: 16
Liked: 1 time
Joined: Tue Dec 17, 2013 3:16 pm
Full Name: Fabio Bozzolo

Re: Data corrupted after disk fail. Any suggestions?

Veeam Logoby foggy » Thu Dec 01, 2016 11:49 am

Well, in this case it does indeed mean that the chain cannot be used further - even if the backups themselves might be ok, there's no valid copy of metadata (out of the two we always keep for redundancy) to make sense of them. So yes, I'd pay attention at the QNAP side of things.
foggy
Veeam Software
 
Posts: 14588
Liked: 1062 times
Joined: Mon Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson

Re: Data corrupted after disk fail. Any suggestions?

Veeam Logoby Yannis » Tue Jan 17, 2017 10:29 am

Hi,
We encountered the same problem by 3 time with QNAP's NAS with repository in SMB file share.
Each time, when a disk failed, metadata are corrupted even after a successfull RAID rebuilt.
We doesn't find any solution about this and we hope to find an explaination.
Furthermore as a workaround, we don't sale and use any QNAP devices...
So, you are not alone :-(
regards,
Yannis.
Yannis
Service Provider
 
Posts: 3
Liked: 2 times
Joined: Mon Jun 13, 2016 7:51 am
Full Name: Yannis

Re: Data corrupted after disk fail. Any suggestions?

Veeam Logoby fbaschin@asion-it.de » Wed Jul 05, 2017 12:22 pm

HI,

I can confirm this problem. Since last 12 months we had also over 5 same issues.
Alle times a disk in raid5 on a qnap fails, replaced and raid restores without any errors
ALL files after that where okay, except the veeam repositoris.
fbaschin@asion-it.de
Lurker
 
Posts: 1
Liked: never
Joined: Wed Jul 05, 2017 12:19 pm
Full Name: Frank Baschin

Re: Data corrupted after disk fail. Any suggestions?

Veeam Logoby mmeurer » Wed Jul 05, 2017 12:39 pm

Hi all,
we have the same problem with Synology's NAS, but what I don't understand it, we have two backup jobs
to the same backup repositorie (Synology NAS with RAID 5) one smb share. One backup chain works great with failed
disk and after rebuild the raid, the other backup chain metadata goes corrupt when disk failed.

Regards,
Marco
mmeurer
Lurker
 
Posts: 1
Liked: never
Joined: Mon Jun 29, 2015 2:58 pm

Re: Data corrupted after disk fail. Any suggestions?

Veeam Logoby sbsfaq » Mon Jul 17, 2017 2:14 pm 1 person likes this post

Guys - I've been investigating corruption with QNAP RAID arrays for over 6 months now - the answer is that they have a bug in their firmware that was not fixed until April, and is NOT documented in the release notes. Read more about it here http://www.sbsfaq.com/?p=4277
sbsfaq
Lurker
 
Posts: 2
Liked: 2 times
Joined: Mon Jul 17, 2017 1:36 pm
Full Name: Wayne Small


Return to Veeam Backup & Replication



Who is online

Users browsing this forum: jay.cheon and 21 guests