QNAP NAS Data Corruption & ReFS / Veeam Possible Connection?

[dpink]

Greetings. I just want to bring to attention a situation that we have been running into in our environment, and haven't had much luck in resolution. We have opened tickets on a few occasions, and after starting a new chain the issue will go away for a few weeks before re-occurring.

The errors we have seen from Veeam jobs have been along the lines of:

Failed to perform backup file verification Error: A data integrity checksum error occurred. Data in the file stream is corrupt.
Failed to read data from the file [###################.vbk].
Agent failed to process method {Signature.FullRecheckBackup}.

The ReFS Event logs read as:

Volume is formatted as ReFS but ReFS is unable to mount it; ReFS encountered status The volume repair was not successful..

And

The file system detected a checksum error and was not able to correct it. The name of the file or folder is "Container Table".

We utilize a QNAP 1280U for backup repository, and per this new QNAP Advisory Detail: https://www.qnap.com/en/technical-advis ... -201707-01 It looks like we might have been running into this issue without even knowing it.

We are using a ReFS volume on the QNAP and have been concerned when seeing these ReFS errors.

This is just a heads-up to check / update QNAP firmware, and we will be testing this out to see if ReFS verification issues continue to arise after the update to our QNAP.

[sbsfaq]

Hi David,

The specifics of that article only apply if you have a disk fail in your RAID pack. The additional information about the data scrubbing is a new feature they have added to help prevent issues in future not related to the disk failure issue that I've seen.

It certainly can't hurt as it's obvious now that QNAP (like other vendors) do not release ALL the information about fixes in their firmware in their release notes, so you might never know and it might well be fixed

Wayne

[dpink]

Unfortunately for us, we had previously tested the RAID rebuild times for a few different RAID configurations before fully deploying to production. Because of this, we believe we were getting affected by the silent corruption issue due to bad parity calculations stemming from those test re-builds.

But yes, the data scrubbing feature will hopefully help everyone going forward and I'll update this again in a few weeks with our result. It was like clockwork previously, with the ReFS errors appearing after 2 weeks.