Corrupted Files on Win2016 Deduplication

Availability for the Always-On Enterprise

Corrupted Files on Win2016 Deduplication

Veeam Logoby LeoKurz » Mon Jan 23, 2017 4:38 pm 3 people like this post

Hi,

I'm running a Win2012R2 BU server with VEEAM BnR 9.5. This server runs copy jobs to a Windows 2016 repository server with a scale out repository with 3 extends, each 50TB, formatted with NTFS and Windows deduplication. Every time scrubbing runs, it reports uncorretable corruption (event ID 12800) on files >2TB. Are they really corrupt? All other files seem to be fine. I know, deduplication only works on parts of files >2TB, but they still should be all right. VEEAM reports no errors during copy job. Any ideas?
BTW, the repositry is meant to be an archive with changing VMs in the long run. That's why I'm not using ReFS with the new API. IMHO, deduplication will provide better space savings over the time.

__Leo
LeoKurz
Veeam ProPartner
 
Posts: 22
Liked: 6 times
Joined: Wed Mar 16, 2011 8:36 am
Full Name: Leonhard Kurz

Re: Corrupted Files on Win2016 Deduplication

Veeam Logoby Martin Damgaard » Tue Jan 24, 2017 12:52 pm 4 people like this post

I too have experienced this. It seriously doubt you can trust any of you archives that show scrubbing errors in the event log!
For me Veeam ran fine (because the data was still "hot", and not deduplicated yet. But later, when using the backup validator tool, i had errors in all my archives above 2 TB... Bye bye last years archive. :-(

After a long time testing and giving feedback to MS dedupe team, they confirmed this is indeed a bug. (We have been discussing this with MS since before Server 2016 went RTM)
A couple of us have received a private test patch for this issue, that seems to fix the problem. I think they plan on releasing a preview release sometime this month.

Until this is fixed, i strongly urge everyone to NOT use Server 2016 with deduplication enabled for files above ~2TB ! ! !
Martin Damgaard
Influencer
 
Posts: 14
Liked: 7 times
Joined: Mon Aug 31, 2015 6:31 am
Full Name: Martin Damgaard

Re: Corrupted Files on Win2016 Deduplication

Veeam Logoby LeoKurz » Tue Jan 24, 2017 1:14 pm 3 people like this post

Veeam Backup Validator also confirmed corrupted files :-( This indeed is very bad! I'll disable optimization jobs for now and see what M$ brings up.

__Leo
LeoKurz
Veeam ProPartner
 
Posts: 22
Liked: 6 times
Joined: Wed Mar 16, 2011 8:36 am
Full Name: Leonhard Kurz

Re: Corrupted Files on Win2016 Deduplication

Veeam Logoby jmmarton » Wed Jan 25, 2017 2:37 pm

Martin Damgaard wrote:After a long time testing and giving feedback to MS dedupe team, they confirmed this is indeed a bug. (We have been discussing this with MS since before Server 2016 went RTM)
A couple of us have received a private test patch for this issue, that seems to fix the problem. I think they plan on releasing a preview release sometime this month.


This is good to know--thanks for sharing! Are you aware of any KB article Microsoft has posted about this?

Joe
jmmarton
Veeam Software
 
Posts: 760
Liked: 86 times
Joined: Tue Nov 17, 2015 2:38 am
Location: Chicago, IL
Full Name: Joe Marton

Re: Corrupted Files on Win2016 Deduplication

Veeam Logoby DonZoomik » Sun Jan 29, 2017 10:54 pm 1 person likes this post

KB3216755 has the fixes and is public as the most recent build with one word about it in release notes. Preliminary testing shows no problems.
Martin and myself contacted MS sometime in August so this has been out there for a while.
About affected file sizes, I saw corruption already at 1-1,5TB.
DonZoomik
Influencer
 
Posts: 18
Liked: 7 times
Joined: Fri Nov 25, 2016 1:56 pm

Re: Corrupted Files on Win2016 Deduplication

Veeam Logoby Gostev » Mon Jan 30, 2017 2:17 am 1 person likes this post

Please keep in mind that the current KB3216755 package is a pre-release preview, and this build is a subject to recall for quality considerations. The note I've got from Microsoft Connect when it became available last week was pretty clear about this:

This package contains a PRE-RELEASE PREVIEW of x86/x64 Windows 10 and Windows Server 2016 (Version 1607) 2017.01D (Cumulative) Servicing Update for TESTING PURPOSES ONLY, and cannot be redistributed.

If you file bugs based on your testing, please:
[bug submission instructions]

This preview release build is subject to recall for quality considerations.

I assume that if not recalled, this build will become February 2017 patch Tuesday update.
Gostev
Veeam Software
 
Posts: 21254
Liked: 2318 times
Joined: Sun Jan 01, 2006 1:01 am
Full Name: Anton Gostev

Re: Corrupted Files on Win2016 Deduplication

Veeam Logoby DrCheese » Mon Jan 30, 2017 7:23 am

How issues like this get through MS QA is beyond me, especially as it was known about pre-release
https://blog.zoomik.pri.ee/uncategorize ... uge-files/

Thankfully I think we're ok, our largest file was 1.8TB & I've done a verification on our files - All have checked out ok. I've also had bad runin's before with Dedupe so had set the scrubbing/corruption checker to run fairly often.

I really like the space benefits of Dedupe (I get 80%!) - But I just can't rely on it anymore. What compression ratios will I get if I switch it all to REFS?
DrCheese
Novice
 
Posts: 3
Liked: never
Joined: Mon Jan 30, 2017 6:32 am

Re: Corrupted Files on Win2016 Deduplication

Veeam Logoby grimson » Mon Jan 30, 2017 7:32 am

Gostev wrote:Please keep in mind that the current KB3216755 package is a pre-release preview, and this build is a subject to recall for quality considerations. The note I've got from Microsoft Connect when it became available last week was pretty clear about this:
I assume that if not recalled, this build will become February 2017 patch Tuesday update.

https://support.microsoft.com/en-us/hel ... -kb3216755
It is available for public download. No signs of preview status of test build. Only difference is that this update is not available trough Windows Update and can only be downloaded from the Windows Update catalogue.
Unfortunately very confusing if this posted release is in preview of not. My guess it's not?!
Strangely this issue is not mentioned in the 'Key changes' changelog.
Also included and not mentioned as a 'key change':
The update itself appears to fix a long-standing problem – the inability to create, rename or delete a folder on a shared network. The issue has been around ever since Windows 10 version 1607 was made available six months ago, and KB 3216755 finally addresses it.
grimson
Novice
 
Posts: 5
Liked: never
Joined: Thu Jul 28, 2011 8:20 am
Full Name: Arian van der Pijl

Re: Corrupted Files on Win2016 Deduplication

Veeam Logoby DonZoomik » Mon Jan 30, 2017 7:43 am

It depends on what you consider a key change but the new style of release notes really suck.
Addressed additional issues with ... deduplication ...
DonZoomik
Influencer
 
Posts: 18
Liked: 7 times
Joined: Fri Nov 25, 2016 1:56 pm

Re: Corrupted Files on Win2016 Deduplication

Veeam Logoby Mike Resseler » Mon Jan 30, 2017 7:47 am

I agree. I still hope that they will change that at a certain point in time and actually say what it fixes.

The KB is indeed available for public download but as Arian said only comes through the catalogue. That is normally seen as a preview (they don't want it to be installed on every machine, only on those that really need it) and can change when the official one comes out (which I assume will be around the 7th of February)
Mike Resseler
Veeam Software
 
Posts: 3000
Liked: 354 times
Joined: Fri Feb 08, 2013 3:08 pm
Location: Belgium, the land of the fries, the beer, the chocolate and the diamonds...
Full Name: Mike Resseler

Re: Corrupted Files on Win2016 Deduplication

Veeam Logoby Delo123 » Mon Jan 30, 2017 8:42 am

2 Issues this time, not only for large files but again i would like to ask for Veeam to support the option of splitting VBK's in smaller chunks
Delo123
Expert
 
Posts: 337
Liked: 93 times
Joined: Fri Dec 28, 2012 5:20 pm
Full Name: Guido Meijers

Re: Corrupted Files on Win2016 Deduplication

Veeam Logoby DonZoomik » Mon Jan 30, 2017 9:04 am

Or alternatively - as we already have per-VM chains, how about more splitting to per-disk chains. For example - one file for metadata (configuration files etc...) and one per each VMDK/VHD/VHDX/...
If splitting at arbitary boundaries is too difficult, this could be much easier and save the day for some users...
DonZoomik
Influencer
 
Posts: 18
Liked: 7 times
Joined: Fri Nov 25, 2016 1:56 pm

Re: Corrupted Files on Win2016 Deduplication

Veeam Logoby Delo123 » Mon Jan 30, 2017 9:07 am

Per VM Chains is nice for performance and nice for scale out repositories but it doesn't help in whatever case in getting smaller file sizes. If the vm is big it's big...
Delo123
Expert
 
Posts: 337
Liked: 93 times
Joined: Fri Dec 28, 2012 5:20 pm
Full Name: Guido Meijers

Re: Corrupted Files on Win2016 Deduplication

Veeam Logoby graham8 » Mon Jan 30, 2017 1:40 pm

If anyone sees this thread and thinks about switching to 2016+ReFS for their repos instead, make sure you format the ReFS Volume with 64KB clusters. This is not the default. See https://forums.veeam.com/post226420.html#p226420 for more information.
graham8
Enthusiast
 
Posts: 59
Liked: 20 times
Joined: Wed Dec 14, 2016 1:56 pm

Re: Corrupted Files on Win2016 Deduplication

Veeam Logoby anglerzac » Mon Jan 30, 2017 2:01 pm

I have been working through a support call with Veeam (#01993209) since we started using 2016 at our second site as a repository with Veeam 9.5 and per vm backup files - I also chose NTFS with the Server 2016 deduplication for the space savings.

We are randomly getting:

- "All instances of the storage metadata are corrupted" on machines.
- Backup file D:\VeeamBackups\<jobname>\<computername>.vm-39632D2016-11-27T180000.vbk is corrupted or unreadable.

Veeam support believed this to be due to network blips(wsus reboot) or the Sandisk SSD/DAS Cache we have on the server.

I immediately updated to 9.5 Update 1 having seen the release note mention dealing with network outages and the message seems to have changed to:

- Disk <computername>-flat.vmdk of VM <computername> is corrupted, possible reason: Storage I/O issue. Corrupted data is located in the following backup files: <computername>.vm-21034D2016-12-08T070000.vbk

Now having seen this I guess this could all be due to the underlying microsoft problem, I'll have to run the Veeam Backup Validator.

Anyhoo I was just posting in case others had been battling these "All instances of the storage metadata are corrupted" issues on Server 2016 deduplicated repositories.
anglerzac
Influencer
 
Posts: 20
Liked: never
Joined: Fri Nov 14, 2014 11:03 am
Full Name: Zac

Next

Return to Veeam Backup & Replication



Who is online

Users browsing this forum: Bing [Bot], Nikolay.Stankevich, samifarhat, yastt and 57 guests