Corrupted Files on Win2016 Deduplication

Availability for the Always-On Enterprise

Re: Corrupted Files on Win2016 Deduplication

Veeam Logoby DonZoomik » Tue Jan 31, 2017 7:42 am 2 people like this post

Will just posted in storage team blog so get patching!
https://blogs.technet.microsoft.com/fil ... kb3216755/
DonZoomik
Influencer
 
Posts: 16
Liked: 7 times
Joined: Fri Nov 25, 2016 1:56 pm

Re: Corrupted Files on Win2016 Deduplication

Veeam Logoby Mike Resseler » Tue Jan 31, 2017 7:50 am 1 person likes this post

Don,

Thanks for bringing this to the attention of the community!
Mike Resseler
Veeam Software
 
Posts: 2780
Liked: 340 times
Joined: Fri Feb 08, 2013 3:08 pm
Location: Belgium, the land of the fries, the beer, the chocolate and the diamonds...
Full Name: Mike Resseler

Re: Corrupted Files on Win2016 Deduplication

Veeam Logoby SeandG » Tue Jan 31, 2017 9:30 am

Thanks Don! We too are experiencing this issue with 2016 and deduplication.

We have applied the MS kb3216755 yesterday, the MS KB notes didn't say anything about being preview so we decided to have a go as most of our backups were at risk. Lets see how that goes!
SeandG
Influencer
 
Posts: 13
Liked: 1 time
Joined: Mon Jan 30, 2017 10:38 am

Re: Corrupted Files on Win2016 Deduplication

Veeam Logoby gairys » Tue Jan 31, 2017 2:22 pm

Does anyone know if this impacts volumes that were dedup'ed in 2012R2 and brought into 2016 via a cluster upgrade? We currently have 3 clusters that we run file services on, all formatted NTFS and running dedup on Server 2012R2. We just received our replacement hardware and had plans of doing a cluster upgrade to 2016 with new hardware and migrating the data off the NTFS volumes to newly created ReFS volumes on the cluster JBODs. After reading through this, I'm unclear as to whether this is only impacting dedup enabled NTFS volumes created by 2016, or does it also include volumes created by 2012 R2 where the attached server was upgraded to 2016.
gairys
Novice
 
Posts: 8
Liked: 2 times
Joined: Mon Nov 24, 2014 4:27 pm
Full Name: Gairy S

Re: Corrupted Files on Win2016 Deduplication

Veeam Logoby Martin Damgaard » Tue Jan 31, 2017 2:30 pm

Hi Anglerzac

This is excactly how i started noticing the issue. I could confirm this by running a scrubbing job on the server.
That gave a match every time. When eventviewer listed a file as corrupt and unable to repair - Veeam Backup or Validator tool confirmed this.

I highly recommend running some manual scrubs on the volume or/and test ALL archives with the validator tool.
Otherwise your backups might be corrupt, without you noticing it!
(From my understanding Veeam health checks ONLY tests archives up til last full VBK. All earlier backups can only be checked by Validator tool).
Martin Damgaard
Influencer
 
Posts: 14
Liked: 7 times
Joined: Mon Aug 31, 2015 6:31 am
Full Name: Martin Damgaard

Re: Corrupted Files on Win2016 Deduplication

Veeam Logoby DonZoomik » Tue Jan 31, 2017 3:16 pm

gairys, on-disk data structures will be silently upgraded to 2016 level during first optimization so yes, you will be affected.
DonZoomik
Influencer
 
Posts: 16
Liked: 7 times
Joined: Fri Nov 25, 2016 1:56 pm

Re: Corrupted Files on Win2016 Deduplication

Veeam Logoby Delo123 » Wed Feb 01, 2017 11:48 am 1 person likes this post

If we didn't know better one could think Microsoft created a no-way-back cryptolocker... :(
Delo123
Expert
 
Posts: 330
Liked: 92 times
Joined: Fri Dec 28, 2012 5:20 pm
Full Name: Guido Meijers

Re: Corrupted Files on Win2016 Deduplication

Veeam Logoby anglerzac » Wed Feb 01, 2017 3:44 pm

Martin Damgaard wrote:Hi Anglerzac, This is excactly how i started noticing the issue. I could confirm this by running a scrubbing job on the server.
That gave a match every time. When eventviewer listed a file as corrupt and unable to repair - Veeam Backup or Validator tool confirmed this.
I highly recommend running some manual scrubs on the volume or/and test ALL archives with the validator tool.
Otherwise your backups might be corrupt, without you noticing it!
(From my understanding Veeam health checks ONLY tests archives up til last full VBK. All earlier backups can only be checked by Validator tool).

I can't get the validator tool to work, If I point it at a CopyJob which I know has 50 machines in it for example it only checks 6 machines and says they are fine.
If I point it at a file it says failed to process storage, failed to access storage file, I have the drive mapped from veeam server and can see/copy the files fine.
I'll try the scrubbing task/command and wait to see how my Veeam support call progresses.
anglerzac
Influencer
 
Posts: 20
Liked: never
Joined: Fri Nov 14, 2014 11:03 am
Full Name: Zac

Re: Corrupted Files on Win2016 Deduplication

Veeam Logoby mkretzer » Sat Feb 04, 2017 12:09 pm

We installed KB3216755 because MS told us to do so and now our veeam backup service takes up all the memory and crashes after 12 hours! Waiting for Veeam support to call me now!
mkretzer
Expert
 
Posts: 249
Liked: 60 times
Joined: Thu Dec 17, 2015 7:17 am

Re: Corrupted Files on Win2016 Deduplication

Veeam Logoby FragglePete » Mon Feb 06, 2017 8:48 am

Does this 'bug' only come into effect if your have deduplication enabled on the volume where the Veeam Backup files reside?

Our Veeam backup server has it installed but not enabled on the volumes (obviously needed to restore files from a server that does have it enabled on the volumes). My largest backup file for on particular VM is now hitting 1.9Tb so need to work out if I need to take some sort of action.

Pete
FragglePete
Novice
 
Posts: 3
Liked: 1 time
Joined: Mon Dec 19, 2016 12:45 pm
Full Name: Pete Williams

Re: Corrupted Files on Win2016 Deduplication

Veeam Logoby final » Mon Feb 06, 2017 11:18 am 1 person likes this post

@FragglePete: I think this is "just" a bug with Server 2016 dedup. Basically, if you feed it large data, it may corrupt them.

I've been running the preview patch since Decembet 5th and did not have any issues since. For me, Veeam 9.5U1 does not crash with the preview patch installed (but maybe they've packed some other things into the KB they've made publicly available).

I have a 20TB NTFS volume with 70TB of UserData on it (all Veeam Backups). The largest files are 2.4TB files (one VM, active full backup creates this file every week). Before the patch, dedup used to corrupt that particular VM every single time. With the patch, I haven't had a corruption in 10 weeks (that is 10 2.4TB files).
final
Influencer
 
Posts: 18
Liked: 4 times
Joined: Sun Aug 14, 2016 7:19 pm

Re: Corrupted Files on Win2016 Deduplication

Veeam Logoby TGacs » Mon Feb 06, 2017 11:38 pm

Noob question here. We haven't yet deployed our 2016 servers and are new to Veeam.

Does this issue impact the source and target deduplication features of Veeam described here?: https://www.veeam.com/hyper-v-vmware-ba ... ssion.html

It seems from the discussion that the issue is with OS-level deduplication (which is separate from Veeam source/target deduplication) and only an issue on on backup destination volumes. Is that correct?
TGacs
Novice
 
Posts: 8
Liked: never
Joined: Tue Sep 27, 2016 6:59 pm

Re: Corrupted Files on Win2016 Deduplication

Veeam Logoby v.Eremin » Tue Feb 07, 2017 8:22 am

Does this issue impact the source and target deduplication features of Veeam described here?

No, it does not.

It seems from the discussion that the issue is with OS-level deduplication (which is separate from Veeam source/target deduplication) and only an issue on on backup destination volumes. Is that correct?

Correct.
v.Eremin
Veeam Software
 
Posts: 12694
Liked: 920 times
Joined: Fri Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin

Re: Corrupted Files on Win2016 Deduplication

Veeam Logoby GregorSmrz » Sun Feb 19, 2017 2:23 pm

Hi!

Just for clarification:
Is the problem of a repository formated with ReFS and deduplication enabled with a cluster size of 64k on a Veeam B&R 9.5 U1 server applied the Microsoft Hotfix KB3216755 fixed??
The discussion here give not really a final statement about that...


thx!
GregorSmrz
Lurker
 
Posts: 1
Liked: never
Joined: Tue Nov 17, 2015 2:45 pm
Full Name: Gregor Smrz

Re: Corrupted Files on Win2016 Deduplication

Veeam Logoby Delo123 » Mon Feb 20, 2017 10:44 am

You cannot enable Deduplication on a ReFS volume. Deduplication is only available for NTFS (where 64K are also preferred and formatting should use /L)
Delo123
Expert
 
Posts: 330
Liked: 92 times
Joined: Fri Dec 28, 2012 5:20 pm
Full Name: Guido Meijers

PreviousNext

Return to Veeam Backup & Replication



Who is online

Users browsing this forum: Bing [Bot], Google [Bot], Google Feedfetcher, Samirang and 56 guests