Corrupted Files on Win2016 Deduplication

[LeoKurz]

Hi,

I'm running a Win2012R2 BU server with VEEAM BnR 9.5. This server runs copy jobs to a Windows 2016 repository server with a scale out repository with 3 extends, each 50TB, formatted with NTFS and Windows deduplication. Every time scrubbing runs, it reports uncorretable corruption (event ID 12800) on files >2TB. Are they really corrupt? All other files seem to be fine. I know, deduplication only works on parts of files >2TB, but they still should be all right. VEEAM reports no errors during copy job. Any ideas?
BTW, the repositry is meant to be an archive with changing VMs in the long run. That's why I'm not using ReFS with the new API. IMHO, deduplication will provide better space savings over the time.

__Leo

[Martin Damgaard]

I too have experienced this. It seriously doubt you can trust any of you archives that show scrubbing errors in the event log!
For me Veeam ran fine (because the data was still "hot", and not deduplicated yet. But later, when using the backup validator tool, i had errors in all my archives above 2 TB... Bye bye last years archive.

After a long time testing and giving feedback to MS dedupe team, they confirmed this is indeed a bug. (We have been discussing this with MS since before Server 2016 went RTM)
A couple of us have received a private test patch for this issue, that seems to fix the problem. I think they plan on releasing a preview release sometime this month.

Until this is fixed, i strongly urge everyone to NOT use Server 2016 with deduplication enabled for files above ~2TB ! ! !

[LeoKurz]

Veeam Backup Validator also confirmed corrupted files This indeed is very bad! I'll disable optimization jobs for now and see what M$ brings up.

__Leo

[jmmarton]

After a long time testing and giving feedback to MS dedupe team, they confirmed this is indeed a bug. (We have been discussing this with MS since before Server 2016 went RTM)
A couple of us have received a private test patch for this issue, that seems to fix the problem. I think they plan on releasing a preview release sometime this month.

This is good to know--thanks for sharing! Are you aware of any KB article Microsoft has posted about this?

Joe

[DonZoomik]

KB3216755 has the fixes and is public as the most recent build with one word about it in release notes. Preliminary testing shows no problems.
Martin and myself contacted MS sometime in August so this has been out there for a while.
About affected file sizes, I saw corruption already at 1-1,5TB.

[Gostev]

Please keep in mind that the current KB3216755 package is a pre-release preview, and this build is a subject to recall for quality considerations. The note I've got from Microsoft Connect when it became available last week was pretty clear about this:

This package contains a PRE-RELEASE PREVIEW of x86/x64 Windows 10 and Windows Server 2016 (Version 1607) 2017.01D (Cumulative) Servicing Update for TESTING PURPOSES ONLY, and cannot be redistributed.

If you file bugs based on your testing, please:
[bug submission instructions]

This preview release build is subject to recall for quality considerations.

I assume that if not recalled, this build will become February 2017 patch Tuesday update.

[DrCheese]

How issues like this get through MS QA is beyond me, especially as it was known about pre-release
https://blog.zoomik.pri.ee/uncategorize ... uge-files/

Thankfully I think we're ok, our largest file was 1.8TB & I've done a verification on our files - All have checked out ok. I've also had bad runin's before with Dedupe so had set the scrubbing/corruption checker to run fairly often.

I really like the space benefits of Dedupe (I get 80%!) - But I just can't rely on it anymore. What compression ratios will I get if I switch it all to REFS?

[grimson]

Please keep in mind that the current KB3216755 package is a pre-release preview, and this build is a subject to recall for quality considerations. The note I've got from Microsoft Connect when it became available last week was pretty clear about this:
I assume that if not recalled, this build will become February 2017 patch Tuesday update.

https://support.microsoft.com/en-us/hel ... -kb3216755
It is available for public download. No signs of preview status of test build. Only difference is that this update is not available trough Windows Update and can only be downloaded from the Windows Update catalogue.
Unfortunately very confusing if this posted release is in preview of not. My guess it's not?!
Strangely this issue is not mentioned in the 'Key changes' changelog.
Also included and not mentioned as a 'key change':
The update itself appears to fix a long-standing problem – the inability to create, rename or delete a folder on a shared network. The issue has been around ever since Windows 10 version 1607 was made available six months ago, and KB 3216755 finally addresses it.

[DonZoomik]

It depends on what you consider a key change but the new style of release notes really suck.

Addressed additional issues with ... deduplication ...

[Mike Resseler]

I agree. I still hope that they will change that at a certain point in time and actually say what it fixes.

The KB is indeed available for public download but as Arian said only comes through the catalogue. That is normally seen as a preview (they don't want it to be installed on every machine, only on those that really need it) and can change when the official one comes out (which I assume will be around the 7th of February)

[Delo123]

2 Issues this time, not only for large files but again i would like to ask for Veeam to support the option of splitting VBK's in smaller chunks

[DonZoomik]

Or alternatively - as we already have per-VM chains, how about more splitting to per-disk chains. For example - one file for metadata (configuration files etc...) and one per each VMDK/VHD/VHDX/...
If splitting at arbitary boundaries is too difficult, this could be much easier and save the day for some users...

[Delo123]

Per VM Chains is nice for performance and nice for scale out repositories but it doesn't help in whatever case in getting smaller file sizes. If the vm is big it's big...

[graham8]

If anyone sees this thread and thinks about switching to 2016+ReFS for their repos instead, make sure you format the ReFS Volume with 64KB clusters. This is not the default. See post226420.html#p226420 for more information.

[anglerzac]

I have been working through a support call with Veeam (#01993209) since we started using 2016 at our second site as a repository with Veeam 9.5 and per vm backup files - I also chose NTFS with the Server 2016 deduplication for the space savings.

We are randomly getting:

- "All instances of the storage metadata are corrupted" on machines.
- Backup file D:\VeeamBackups\<jobname>\<computername>.vm-39632D2016-11-27T180000.vbk is corrupted or unreadable.

Veeam support believed this to be due to network blips(wsus reboot) or the Sandisk SSD/DAS Cache we have on the server.

I immediately updated to 9.5 Update 1 having seen the release note mention dealing with network outages and the message seems to have changed to:

- Disk <computername>-flat.vmdk of VM <computername> is corrupted, possible reason: Storage I/O issue. Corrupted data is located in the following backup files: <computername>.vm-21034D2016-12-08T070000.vbk

Now having seen this I guess this could all be due to the underlying microsoft problem, I'll have to run the Veeam Backup Validator.

Anyhoo I was just posting in case others had been battling these "All instances of the storage metadata are corrupted" issues on Server 2016 deduplicated repositories.

[DonZoomik]

Will just posted in storage team blog so get patching!
https://blogs.technet.microsoft.com/fil ... kb3216755/

[Mike Resseler]

Don,

Thanks for bringing this to the attention of the community!

[SeandG]

Thanks Don! We too are experiencing this issue with 2016 and deduplication.

We have applied the MS kb3216755 yesterday, the MS KB notes didn't say anything about being preview so we decided to have a go as most of our backups were at risk. Lets see how that goes!

[gairys]

Does anyone know if this impacts volumes that were dedup'ed in 2012R2 and brought into 2016 via a cluster upgrade? We currently have 3 clusters that we run file services on, all formatted NTFS and running dedup on Server 2012R2. We just received our replacement hardware and had plans of doing a cluster upgrade to 2016 with new hardware and migrating the data off the NTFS volumes to newly created ReFS volumes on the cluster JBODs. After reading through this, I'm unclear as to whether this is only impacting dedup enabled NTFS volumes created by 2016, or does it also include volumes created by 2012 R2 where the attached server was upgraded to 2016.

[Martin Damgaard]

Hi Anglerzac

This is excactly how i started noticing the issue. I could confirm this by running a scrubbing job on the server.
That gave a match every time. When eventviewer listed a file as corrupt and unable to repair - Veeam Backup or Validator tool confirmed this.

I highly recommend running some manual scrubs on the volume or/and test ALL archives with the validator tool.
Otherwise your backups might be corrupt, without you noticing it!
(From my understanding Veeam health checks ONLY tests archives up til last full VBK. All earlier backups can only be checked by Validator tool).

[DonZoomik]

gairys, on-disk data structures will be silently upgraded to 2016 level during first optimization so yes, you will be affected.

[Delo123]

If we didn't know better one could think Microsoft created a no-way-back cryptolocker...

[anglerzac]

Hi Anglerzac, This is excactly how i started noticing the issue. I could confirm this by running a scrubbing job on the server.
That gave a match every time. When eventviewer listed a file as corrupt and unable to repair - Veeam Backup or Validator tool confirmed this.
I highly recommend running some manual scrubs on the volume or/and test ALL archives with the validator tool.
Otherwise your backups might be corrupt, without you noticing it!
(From my understanding Veeam health checks ONLY tests archives up til last full VBK. All earlier backups can only be checked by Validator tool).

I can't get the validator tool to work, If I point it at a CopyJob which I know has 50 machines in it for example it only checks 6 machines and says they are fine.
If I point it at a file it says failed to process storage, failed to access storage file, I have the drive mapped from veeam server and can see/copy the files fine.
I'll try the scrubbing task/command and wait to see how my Veeam support call progresses.

[mkretzer]

We installed KB3216755 because MS told us to do so and now our veeam backup service takes up all the memory and crashes after 12 hours! Waiting for Veeam support to call me now!

[FragglePete]

Does this 'bug' only come into effect if your have deduplication enabled on the volume where the Veeam Backup files reside?

Our Veeam backup server has it installed but not enabled on the volumes (obviously needed to restore files from a server that does have it enabled on the volumes). My largest backup file for on particular VM is now hitting 1.9Tb so need to work out if I need to take some sort of action.

Pete

[final]

@FragglePete: I think this is "just" a bug with Server 2016 dedup. Basically, if you feed it large data, it may corrupt them.

I've been running the preview patch since Decembet 5th and did not have any issues since. For me, Veeam 9.5U1 does not crash with the preview patch installed (but maybe they've packed some other things into the KB they've made publicly available).

I have a 20TB NTFS volume with 70TB of UserData on it (all Veeam Backups). The largest files are 2.4TB files (one VM, active full backup creates this file every week). Before the patch, dedup used to corrupt that particular VM every single time. With the patch, I haven't had a corruption in 10 weeks (that is 10 2.4TB files).

[TGacs]

Noob question here. We haven't yet deployed our 2016 servers and are new to Veeam.

Does this issue impact the source and target deduplication features of Veeam described here?: https://www.veeam.com/hyper-v-vmware-ba ... ssion.html

It seems from the discussion that the issue is with OS-level deduplication (which is separate from Veeam source/target deduplication) and only an issue on on backup destination volumes. Is that correct?

[veremin]

Does this issue impact the source and target deduplication features of Veeam described here?

No, it does not.

It seems from the discussion that the issue is with OS-level deduplication (which is separate from Veeam source/target deduplication) and only an issue on on backup destination volumes. Is that correct?

Correct.

[GregorSmrz]

Hi!

Just for clarification:
Is the problem of a repository formated with ReFS and deduplication enabled with a cluster size of 64k on a Veeam B&R 9.5 U1 server applied the Microsoft Hotfix KB3216755 fixed??
The discussion here give not really a final statement about that...


thx!

[Delo123]

You cannot enable Deduplication on a ReFS volume. Deduplication is only available for NTFS (where 64K are also preferred and formatting should use /L)