Comprehensive data protection for all workloads
LeoKurz
Veeam ProPartner
Posts: 28
Liked: 7 times
Joined: Mar 16, 2011 8:36 am
Full Name: Leonhard Kurz
Contact:

Corrupted Files on Win2016 Deduplication

Post by LeoKurz » 3 people like this post

Hi,

I'm running a Win2012R2 BU server with VEEAM BnR 9.5. This server runs copy jobs to a Windows 2016 repository server with a scale out repository with 3 extends, each 50TB, formatted with NTFS and Windows deduplication. Every time scrubbing runs, it reports uncorretable corruption (event ID 12800) on files >2TB. Are they really corrupt? All other files seem to be fine. I know, deduplication only works on parts of files >2TB, but they still should be all right. VEEAM reports no errors during copy job. Any ideas?
BTW, the repositry is meant to be an archive with changing VMs in the long run. That's why I'm not using ReFS with the new API. IMHO, deduplication will provide better space savings over the time.

__Leo
Martin Damgaard
Influencer
Posts: 19
Liked: 8 times
Joined: Aug 31, 2015 6:31 am
Full Name: Martin Damgaard
Contact:

Re: Corrupted Files on Win2016 Deduplication

Post by Martin Damgaard » 4 people like this post

I too have experienced this. It seriously doubt you can trust any of you archives that show scrubbing errors in the event log!
For me Veeam ran fine (because the data was still "hot", and not deduplicated yet. But later, when using the backup validator tool, i had errors in all my archives above 2 TB... Bye bye last years archive. :-(

After a long time testing and giving feedback to MS dedupe team, they confirmed this is indeed a bug. (We have been discussing this with MS since before Server 2016 went RTM)
A couple of us have received a private test patch for this issue, that seems to fix the problem. I think they plan on releasing a preview release sometime this month.

Until this is fixed, i strongly urge everyone to NOT use Server 2016 with deduplication enabled for files above ~2TB ! ! !
LeoKurz
Veeam ProPartner
Posts: 28
Liked: 7 times
Joined: Mar 16, 2011 8:36 am
Full Name: Leonhard Kurz
Contact:

Re: Corrupted Files on Win2016 Deduplication

Post by LeoKurz » 3 people like this post

Veeam Backup Validator also confirmed corrupted files :-( This indeed is very bad! I'll disable optimization jobs for now and see what M$ brings up.

__Leo
jmmarton
Veeam Software
Posts: 2092
Liked: 309 times
Joined: Nov 17, 2015 2:38 am
Full Name: Joe Marton
Location: Chicago, IL
Contact:

Re: Corrupted Files on Win2016 Deduplication

Post by jmmarton »

Martin Damgaard wrote:After a long time testing and giving feedback to MS dedupe team, they confirmed this is indeed a bug. (We have been discussing this with MS since before Server 2016 went RTM)
A couple of us have received a private test patch for this issue, that seems to fix the problem. I think they plan on releasing a preview release sometime this month.
This is good to know--thanks for sharing! Are you aware of any KB article Microsoft has posted about this?

Joe
DonZoomik
Service Provider
Posts: 368
Liked: 120 times
Joined: Nov 25, 2016 1:56 pm
Full Name: Mihkel Soomere
Contact:

Re: Corrupted Files on Win2016 Deduplication

Post by DonZoomik » 1 person likes this post

KB3216755 has the fixes and is public as the most recent build with one word about it in release notes. Preliminary testing shows no problems.
Martin and myself contacted MS sometime in August so this has been out there for a while.
About affected file sizes, I saw corruption already at 1-1,5TB.
Gostev
Chief Product Officer
Posts: 31457
Liked: 6648 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Corrupted Files on Win2016 Deduplication

Post by Gostev » 1 person likes this post

Please keep in mind that the current KB3216755 package is a pre-release preview, and this build is a subject to recall for quality considerations. The note I've got from Microsoft Connect when it became available last week was pretty clear about this:
This package contains a PRE-RELEASE PREVIEW of x86/x64 Windows 10 and Windows Server 2016 (Version 1607) 2017.01D (Cumulative) Servicing Update for TESTING PURPOSES ONLY, and cannot be redistributed.

If you file bugs based on your testing, please:
[bug submission instructions]

This preview release build is subject to recall for quality considerations.
I assume that if not recalled, this build will become February 2017 patch Tuesday update.
DrCheese
Novice
Posts: 3
Liked: never
Joined: Jan 30, 2017 6:32 am
Contact:

Re: Corrupted Files on Win2016 Deduplication

Post by DrCheese »

How issues like this get through MS QA is beyond me, especially as it was known about pre-release
https://blog.zoomik.pri.ee/uncategorize ... uge-files/

Thankfully I think we're ok, our largest file was 1.8TB & I've done a verification on our files - All have checked out ok. I've also had bad runin's before with Dedupe so had set the scrubbing/corruption checker to run fairly often.

I really like the space benefits of Dedupe (I get 80%!) - But I just can't rely on it anymore. What compression ratios will I get if I switch it all to REFS?
grimson
Novice
Posts: 9
Liked: 1 time
Joined: Jul 28, 2011 8:20 am
Full Name: Arian van der Pijl
Contact:

Re: Corrupted Files on Win2016 Deduplication

Post by grimson »

Gostev wrote:Please keep in mind that the current KB3216755 package is a pre-release preview, and this build is a subject to recall for quality considerations. The note I've got from Microsoft Connect when it became available last week was pretty clear about this:
I assume that if not recalled, this build will become February 2017 patch Tuesday update.
https://support.microsoft.com/en-us/hel ... -kb3216755
It is available for public download. No signs of preview status of test build. Only difference is that this update is not available trough Windows Update and can only be downloaded from the Windows Update catalogue.
Unfortunately very confusing if this posted release is in preview of not. My guess it's not?!
Strangely this issue is not mentioned in the 'Key changes' changelog.
Also included and not mentioned as a 'key change':
The update itself appears to fix a long-standing problem – the inability to create, rename or delete a folder on a shared network. The issue has been around ever since Windows 10 version 1607 was made available six months ago, and KB 3216755 finally addresses it.
DonZoomik
Service Provider
Posts: 368
Liked: 120 times
Joined: Nov 25, 2016 1:56 pm
Full Name: Mihkel Soomere
Contact:

Re: Corrupted Files on Win2016 Deduplication

Post by DonZoomik »

It depends on what you consider a key change but the new style of release notes really suck.
Addressed additional issues with ... deduplication ...
Mike Resseler
Product Manager
Posts: 8044
Liked: 1263 times
Joined: Feb 08, 2013 3:08 pm
Full Name: Mike Resseler
Location: Belgium
Contact:

Re: Corrupted Files on Win2016 Deduplication

Post by Mike Resseler »

I agree. I still hope that they will change that at a certain point in time and actually say what it fixes.

The KB is indeed available for public download but as Arian said only comes through the catalogue. That is normally seen as a preview (they don't want it to be installed on every machine, only on those that really need it) and can change when the official one comes out (which I assume will be around the 7th of February)
Delo123
Veteran
Posts: 361
Liked: 109 times
Joined: Dec 28, 2012 5:20 pm
Full Name: Guido Meijers
Contact:

Re: Corrupted Files on Win2016 Deduplication

Post by Delo123 »

2 Issues this time, not only for large files but again i would like to ask for Veeam to support the option of splitting VBK's in smaller chunks
DonZoomik
Service Provider
Posts: 368
Liked: 120 times
Joined: Nov 25, 2016 1:56 pm
Full Name: Mihkel Soomere
Contact:

Re: Corrupted Files on Win2016 Deduplication

Post by DonZoomik »

Or alternatively - as we already have per-VM chains, how about more splitting to per-disk chains. For example - one file for metadata (configuration files etc...) and one per each VMDK/VHD/VHDX/...
If splitting at arbitary boundaries is too difficult, this could be much easier and save the day for some users...
Delo123
Veteran
Posts: 361
Liked: 109 times
Joined: Dec 28, 2012 5:20 pm
Full Name: Guido Meijers
Contact:

Re: Corrupted Files on Win2016 Deduplication

Post by Delo123 »

Per VM Chains is nice for performance and nice for scale out repositories but it doesn't help in whatever case in getting smaller file sizes. If the vm is big it's big...
graham8
Enthusiast
Posts: 59
Liked: 20 times
Joined: Dec 14, 2016 1:56 pm
Contact:

Re: Corrupted Files on Win2016 Deduplication

Post by graham8 »

If anyone sees this thread and thinks about switching to 2016+ReFS for their repos instead, make sure you format the ReFS Volume with 64KB clusters. This is not the default. See post226420.html#p226420 for more information.
anglerzac
Influencer
Posts: 24
Liked: 1 time
Joined: Nov 14, 2014 11:03 am
Full Name: Zac
Contact:

Re: Corrupted Files on Win2016 Deduplication

Post by anglerzac »

I have been working through a support call with Veeam (#01993209) since we started using 2016 at our second site as a repository with Veeam 9.5 and per vm backup files - I also chose NTFS with the Server 2016 deduplication for the space savings.

We are randomly getting:

- "All instances of the storage metadata are corrupted" on machines.
- Backup file D:\VeeamBackups\<jobname>\<computername>.vm-39632D2016-11-27T180000.vbk is corrupted or unreadable.

Veeam support believed this to be due to network blips(wsus reboot) or the Sandisk SSD/DAS Cache we have on the server.

I immediately updated to 9.5 Update 1 having seen the release note mention dealing with network outages and the message seems to have changed to:

- Disk <computername>-flat.vmdk of VM <computername> is corrupted, possible reason: Storage I/O issue. Corrupted data is located in the following backup files: <computername>.vm-21034D2016-12-08T070000.vbk

Now having seen this I guess this could all be due to the underlying microsoft problem, I'll have to run the Veeam Backup Validator.

Anyhoo I was just posting in case others had been battling these "All instances of the storage metadata are corrupted" issues on Server 2016 deduplicated repositories.
DonZoomik
Service Provider
Posts: 368
Liked: 120 times
Joined: Nov 25, 2016 1:56 pm
Full Name: Mihkel Soomere
Contact:

Re: Corrupted Files on Win2016 Deduplication

Post by DonZoomik » 2 people like this post

Will just posted in storage team blog so get patching!
https://blogs.technet.microsoft.com/fil ... kb3216755/
Mike Resseler
Product Manager
Posts: 8044
Liked: 1263 times
Joined: Feb 08, 2013 3:08 pm
Full Name: Mike Resseler
Location: Belgium
Contact:

Re: Corrupted Files on Win2016 Deduplication

Post by Mike Resseler » 1 person likes this post

Don,

Thanks for bringing this to the attention of the community!
SeandG
Influencer
Posts: 15
Liked: 1 time
Joined: Jan 30, 2017 10:38 am
Contact:

Re: Corrupted Files on Win2016 Deduplication

Post by SeandG »

Thanks Don! We too are experiencing this issue with 2016 and deduplication.

We have applied the MS kb3216755 yesterday, the MS KB notes didn't say anything about being preview so we decided to have a go as most of our backups were at risk. Lets see how that goes!
gairys
Influencer
Posts: 12
Liked: 4 times
Joined: Nov 24, 2014 4:27 pm
Full Name: Gairy Spiers
Contact:

Re: Corrupted Files on Win2016 Deduplication

Post by gairys »

Does anyone know if this impacts volumes that were dedup'ed in 2012R2 and brought into 2016 via a cluster upgrade? We currently have 3 clusters that we run file services on, all formatted NTFS and running dedup on Server 2012R2. We just received our replacement hardware and had plans of doing a cluster upgrade to 2016 with new hardware and migrating the data off the NTFS volumes to newly created ReFS volumes on the cluster JBODs. After reading through this, I'm unclear as to whether this is only impacting dedup enabled NTFS volumes created by 2016, or does it also include volumes created by 2012 R2 where the attached server was upgraded to 2016.
Martin Damgaard
Influencer
Posts: 19
Liked: 8 times
Joined: Aug 31, 2015 6:31 am
Full Name: Martin Damgaard
Contact:

Re: Corrupted Files on Win2016 Deduplication

Post by Martin Damgaard »

Hi Anglerzac

This is excactly how i started noticing the issue. I could confirm this by running a scrubbing job on the server.
That gave a match every time. When eventviewer listed a file as corrupt and unable to repair - Veeam Backup or Validator tool confirmed this.

I highly recommend running some manual scrubs on the volume or/and test ALL archives with the validator tool.
Otherwise your backups might be corrupt, without you noticing it!
(From my understanding Veeam health checks ONLY tests archives up til last full VBK. All earlier backups can only be checked by Validator tool).
DonZoomik
Service Provider
Posts: 368
Liked: 120 times
Joined: Nov 25, 2016 1:56 pm
Full Name: Mihkel Soomere
Contact:

Re: Corrupted Files on Win2016 Deduplication

Post by DonZoomik »

gairys, on-disk data structures will be silently upgraded to 2016 level during first optimization so yes, you will be affected.
Delo123
Veteran
Posts: 361
Liked: 109 times
Joined: Dec 28, 2012 5:20 pm
Full Name: Guido Meijers
Contact:

Re: Corrupted Files on Win2016 Deduplication

Post by Delo123 » 1 person likes this post

If we didn't know better one could think Microsoft created a no-way-back cryptolocker... :(
anglerzac
Influencer
Posts: 24
Liked: 1 time
Joined: Nov 14, 2014 11:03 am
Full Name: Zac
Contact:

Re: Corrupted Files on Win2016 Deduplication

Post by anglerzac »

Martin Damgaard wrote:Hi Anglerzac, This is excactly how i started noticing the issue. I could confirm this by running a scrubbing job on the server.
That gave a match every time. When eventviewer listed a file as corrupt and unable to repair - Veeam Backup or Validator tool confirmed this.
I highly recommend running some manual scrubs on the volume or/and test ALL archives with the validator tool.
Otherwise your backups might be corrupt, without you noticing it!
(From my understanding Veeam health checks ONLY tests archives up til last full VBK. All earlier backups can only be checked by Validator tool).
I can't get the validator tool to work, If I point it at a CopyJob which I know has 50 machines in it for example it only checks 6 machines and says they are fine.
If I point it at a file it says failed to process storage, failed to access storage file, I have the drive mapped from veeam server and can see/copy the files fine.
I'll try the scrubbing task/command and wait to see how my Veeam support call progresses.
mkretzer
Veeam Legend
Posts: 1140
Liked: 387 times
Joined: Dec 17, 2015 7:17 am
Contact:

Re: Corrupted Files on Win2016 Deduplication

Post by mkretzer »

We installed KB3216755 because MS told us to do so and now our veeam backup service takes up all the memory and crashes after 12 hours! Waiting for Veeam support to call me now!
FragglePete
Novice
Posts: 3
Liked: 1 time
Joined: Dec 19, 2016 12:45 pm
Full Name: Pete Williams
Contact:

Re: Corrupted Files on Win2016 Deduplication

Post by FragglePete »

Does this 'bug' only come into effect if your have deduplication enabled on the volume where the Veeam Backup files reside?

Our Veeam backup server has it installed but not enabled on the volumes (obviously needed to restore files from a server that does have it enabled on the volumes). My largest backup file for on particular VM is now hitting 1.9Tb so need to work out if I need to take some sort of action.

Pete
final
Enthusiast
Posts: 33
Liked: 13 times
Joined: Aug 14, 2016 7:19 pm
Contact:

Re: Corrupted Files on Win2016 Deduplication

Post by final » 1 person likes this post

@FragglePete: I think this is "just" a bug with Server 2016 dedup. Basically, if you feed it large data, it may corrupt them.

I've been running the preview patch since Decembet 5th and did not have any issues since. For me, Veeam 9.5U1 does not crash with the preview patch installed (but maybe they've packed some other things into the KB they've made publicly available).

I have a 20TB NTFS volume with 70TB of UserData on it (all Veeam Backups). The largest files are 2.4TB files (one VM, active full backup creates this file every week). Before the patch, dedup used to corrupt that particular VM every single time. With the patch, I haven't had a corruption in 10 weeks (that is 10 2.4TB files).
TGacs
Enthusiast
Posts: 37
Liked: 8 times
Joined: Sep 27, 2016 6:59 pm
Contact:

Re: Corrupted Files on Win2016 Deduplication

Post by TGacs »

Noob question here. We haven't yet deployed our 2016 servers and are new to Veeam.

Does this issue impact the source and target deduplication features of Veeam described here?: https://www.veeam.com/hyper-v-vmware-ba ... ssion.html

It seems from the discussion that the issue is with OS-level deduplication (which is separate from Veeam source/target deduplication) and only an issue on on backup destination volumes. Is that correct?
veremin
Product Manager
Posts: 20270
Liked: 2252 times
Joined: Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin
Contact:

Re: Corrupted Files on Win2016 Deduplication

Post by veremin »

Does this issue impact the source and target deduplication features of Veeam described here?
No, it does not.
It seems from the discussion that the issue is with OS-level deduplication (which is separate from Veeam source/target deduplication) and only an issue on on backup destination volumes. Is that correct?
Correct.
GregorSmrz
Novice
Posts: 5
Liked: never
Joined: Nov 17, 2015 2:45 pm
Full Name: Gregor Smrz
Location: Vienna, Austria
Contact:

Re: Corrupted Files on Win2016 Deduplication

Post by GregorSmrz »

Hi!

Just for clarification:
Is the problem of a repository formated with ReFS and deduplication enabled with a cluster size of 64k on a Veeam B&R 9.5 U1 server applied the Microsoft Hotfix KB3216755 fixed??
The discussion here give not really a final statement about that...


thx!
Delo123
Veteran
Posts: 361
Liked: 109 times
Joined: Dec 28, 2012 5:20 pm
Full Name: Guido Meijers
Contact:

Re: Corrupted Files on Win2016 Deduplication

Post by Delo123 »

You cannot enable Deduplication on a ReFS volume. Deduplication is only available for NTFS (where 64K are also preferred and formatting should use /L)
Post Reply

Who is online

Users browsing this forum: No registered users and 220 guests