Availability for the Always-On Enterprise
LeoKurz
Veeam ProPartner
Posts: 25
Liked: 6 times
Joined: Mar 16, 2011 8:36 am
Full Name: Leonhard Kurz
Contact:

Corrupted Files on Win2016 Deduplication

Post by LeoKurz » Jan 23, 2017 4:38 pm 3 people like this post

Hi,

I'm running a Win2012R2 BU server with VEEAM BnR 9.5. This server runs copy jobs to a Windows 2016 repository server with a scale out repository with 3 extends, each 50TB, formatted with NTFS and Windows deduplication. Every time scrubbing runs, it reports uncorretable corruption (event ID 12800) on files >2TB. Are they really corrupt? All other files seem to be fine. I know, deduplication only works on parts of files >2TB, but they still should be all right. VEEAM reports no errors during copy job. Any ideas?
BTW, the repositry is meant to be an archive with changing VMs in the long run. That's why I'm not using ReFS with the new API. IMHO, deduplication will provide better space savings over the time.

__Leo

Martin Damgaard
Influencer
Posts: 16
Liked: 7 times
Joined: Aug 31, 2015 6:31 am
Full Name: Martin Damgaard
Contact:

Re: Corrupted Files on Win2016 Deduplication

Post by Martin Damgaard » Jan 24, 2017 12:52 pm 4 people like this post

I too have experienced this. It seriously doubt you can trust any of you archives that show scrubbing errors in the event log!
For me Veeam ran fine (because the data was still "hot", and not deduplicated yet. But later, when using the backup validator tool, i had errors in all my archives above 2 TB... Bye bye last years archive. :-(

After a long time testing and giving feedback to MS dedupe team, they confirmed this is indeed a bug. (We have been discussing this with MS since before Server 2016 went RTM)
A couple of us have received a private test patch for this issue, that seems to fix the problem. I think they plan on releasing a preview release sometime this month.

Until this is fixed, i strongly urge everyone to NOT use Server 2016 with deduplication enabled for files above ~2TB ! ! !

LeoKurz
Veeam ProPartner
Posts: 25
Liked: 6 times
Joined: Mar 16, 2011 8:36 am
Full Name: Leonhard Kurz
Contact:

Re: Corrupted Files on Win2016 Deduplication

Post by LeoKurz » Jan 24, 2017 1:14 pm 3 people like this post

Veeam Backup Validator also confirmed corrupted files :-( This indeed is very bad! I'll disable optimization jobs for now and see what M$ brings up.

__Leo

jmmarton
Veeam Software
Posts: 1207
Liked: 136 times
Joined: Nov 17, 2015 2:38 am
Full Name: Joe Marton
Location: Chicago, IL
Contact:

Re: Corrupted Files on Win2016 Deduplication

Post by jmmarton » Jan 25, 2017 2:37 pm

Martin Damgaard wrote:After a long time testing and giving feedback to MS dedupe team, they confirmed this is indeed a bug. (We have been discussing this with MS since before Server 2016 went RTM)
A couple of us have received a private test patch for this issue, that seems to fix the problem. I think they plan on releasing a preview release sometime this month.
This is good to know--thanks for sharing! Are you aware of any KB article Microsoft has posted about this?

Joe

DonZoomik
Enthusiast
Posts: 45
Liked: 15 times
Joined: Nov 25, 2016 1:56 pm
Contact:

Re: Corrupted Files on Win2016 Deduplication

Post by DonZoomik » Jan 29, 2017 10:54 pm 1 person likes this post

KB3216755 has the fixes and is public as the most recent build with one word about it in release notes. Preliminary testing shows no problems.
Martin and myself contacted MS sometime in August so this has been out there for a while.
About affected file sizes, I saw corruption already at 1-1,5TB.

Gostev
Veeam Software
Posts: 22770
Liked: 2798 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Corrupted Files on Win2016 Deduplication

Post by Gostev » Jan 30, 2017 2:17 am 1 person likes this post

Please keep in mind that the current KB3216755 package is a pre-release preview, and this build is a subject to recall for quality considerations. The note I've got from Microsoft Connect when it became available last week was pretty clear about this:
This package contains a PRE-RELEASE PREVIEW of x86/x64 Windows 10 and Windows Server 2016 (Version 1607) 2017.01D (Cumulative) Servicing Update for TESTING PURPOSES ONLY, and cannot be redistributed.

If you file bugs based on your testing, please:
[bug submission instructions]

This preview release build is subject to recall for quality considerations.
I assume that if not recalled, this build will become February 2017 patch Tuesday update.

DrCheese
Novice
Posts: 3
Liked: never
Joined: Jan 30, 2017 6:32 am
Contact:

Re: Corrupted Files on Win2016 Deduplication

Post by DrCheese » Jan 30, 2017 7:23 am

How issues like this get through MS QA is beyond me, especially as it was known about pre-release
https://blog.zoomik.pri.ee/uncategorize ... uge-files/

Thankfully I think we're ok, our largest file was 1.8TB & I've done a verification on our files - All have checked out ok. I've also had bad runin's before with Dedupe so had set the scrubbing/corruption checker to run fairly often.

I really like the space benefits of Dedupe (I get 80%!) - But I just can't rely on it anymore. What compression ratios will I get if I switch it all to REFS?

grimson
Novice
Posts: 6
Liked: never
Joined: Jul 28, 2011 8:20 am
Full Name: Arian van der Pijl
Contact:

Re: Corrupted Files on Win2016 Deduplication

Post by grimson » Jan 30, 2017 7:32 am

Gostev wrote:Please keep in mind that the current KB3216755 package is a pre-release preview, and this build is a subject to recall for quality considerations. The note I've got from Microsoft Connect when it became available last week was pretty clear about this:
I assume that if not recalled, this build will become February 2017 patch Tuesday update.
https://support.microsoft.com/en-us/hel ... -kb3216755
It is available for public download. No signs of preview status of test build. Only difference is that this update is not available trough Windows Update and can only be downloaded from the Windows Update catalogue.
Unfortunately very confusing if this posted release is in preview of not. My guess it's not?!
Strangely this issue is not mentioned in the 'Key changes' changelog.
Also included and not mentioned as a 'key change':
The update itself appears to fix a long-standing problem – the inability to create, rename or delete a folder on a shared network. The issue has been around ever since Windows 10 version 1607 was made available six months ago, and KB 3216755 finally addresses it.

DonZoomik
Enthusiast
Posts: 45
Liked: 15 times
Joined: Nov 25, 2016 1:56 pm
Contact:

Re: Corrupted Files on Win2016 Deduplication

Post by DonZoomik » Jan 30, 2017 7:43 am

It depends on what you consider a key change but the new style of release notes really suck.
Addressed additional issues with ... deduplication ...

Mike Resseler
Veeam Software
Posts: 4641
Liked: 497 times
Joined: Feb 08, 2013 3:08 pm
Full Name: Mike Resseler
Location: Belgium
Contact:

Re: Corrupted Files on Win2016 Deduplication

Post by Mike Resseler » Jan 30, 2017 7:47 am

I agree. I still hope that they will change that at a certain point in time and actually say what it fixes.

The KB is indeed available for public download but as Arian said only comes through the catalogue. That is normally seen as a preview (they don't want it to be installed on every machine, only on those that really need it) and can change when the official one comes out (which I assume will be around the 7th of February)

Delo123
Expert
Posts: 361
Liked: 108 times
Joined: Dec 28, 2012 5:20 pm
Full Name: Guido Meijers
Contact:

Re: Corrupted Files on Win2016 Deduplication

Post by Delo123 » Jan 30, 2017 8:42 am

2 Issues this time, not only for large files but again i would like to ask for Veeam to support the option of splitting VBK's in smaller chunks

DonZoomik
Enthusiast
Posts: 45
Liked: 15 times
Joined: Nov 25, 2016 1:56 pm
Contact:

Re: Corrupted Files on Win2016 Deduplication

Post by DonZoomik » Jan 30, 2017 9:04 am

Or alternatively - as we already have per-VM chains, how about more splitting to per-disk chains. For example - one file for metadata (configuration files etc...) and one per each VMDK/VHD/VHDX/...
If splitting at arbitary boundaries is too difficult, this could be much easier and save the day for some users...

Delo123
Expert
Posts: 361
Liked: 108 times
Joined: Dec 28, 2012 5:20 pm
Full Name: Guido Meijers
Contact:

Re: Corrupted Files on Win2016 Deduplication

Post by Delo123 » Jan 30, 2017 9:07 am

Per VM Chains is nice for performance and nice for scale out repositories but it doesn't help in whatever case in getting smaller file sizes. If the vm is big it's big...

graham8
Enthusiast
Posts: 59
Liked: 20 times
Joined: Dec 14, 2016 1:56 pm
Contact:

Re: Corrupted Files on Win2016 Deduplication

Post by graham8 » Jan 30, 2017 1:40 pm

If anyone sees this thread and thinks about switching to 2016+ReFS for their repos instead, make sure you format the ReFS Volume with 64KB clusters. This is not the default. See post226420.html#p226420 for more information.

anglerzac
Influencer
Posts: 22
Liked: never
Joined: Nov 14, 2014 11:03 am
Full Name: Zac
Contact:

Re: Corrupted Files on Win2016 Deduplication

Post by anglerzac » Jan 30, 2017 2:01 pm

I have been working through a support call with Veeam (#01993209) since we started using 2016 at our second site as a repository with Veeam 9.5 and per vm backup files - I also chose NTFS with the Server 2016 deduplication for the space savings.

We are randomly getting:

- "All instances of the storage metadata are corrupted" on machines.
- Backup file D:\VeeamBackups\<jobname>\<computername>.vm-39632D2016-11-27T180000.vbk is corrupted or unreadable.

Veeam support believed this to be due to network blips(wsus reboot) or the Sandisk SSD/DAS Cache we have on the server.

I immediately updated to 9.5 Update 1 having seen the release note mention dealing with network outages and the message seems to have changed to:

- Disk <computername>-flat.vmdk of VM <computername> is corrupted, possible reason: Storage I/O issue. Corrupted data is located in the following backup files: <computername>.vm-21034D2016-12-08T070000.vbk

Now having seen this I guess this could all be due to the underlying microsoft problem, I'll have to run the Veeam Backup Validator.

Anyhoo I was just posting in case others had been battling these "All instances of the storage metadata are corrupted" issues on Server 2016 deduplicated repositories.

Post Reply

Who is online

Users browsing this forum: Bing [Bot], david.tosoff, erohver and 15 guests