Disk errors on Windows server 2016 with deduplication

Availability for the Always-On Enterprise

Re: Disk errors on Windows server 2016 with deduplication

Veeam Logoby somethingsomething » Thu Mar 30, 2017 4:14 pm

We're running Server 2016 w/NTFS Deduped and we've been bitten by "All instances of the metadata are corrupted" 3 times now.
I also ran a chkdsk and I see the "The Volume Bitmap is incorrect" error. But if I do a chkdsk /scan all I get is"Windows has scanned the file system and found no problems. / No further action is required."
KB3216755 did nothing to fix the problem.

I don't see anything in the event logs that appears to be dedup related.

@gostev Please feel free to share details with MS if I can help.

Case # 02104227
somethingsomething
Lurker
 
Posts: 2
Liked: never
Joined: Mon Feb 20, 2017 3:22 pm

Re: Disk errors on Windows server 2016 with deduplication

Veeam Logoby DonZoomik » Thu Mar 30, 2017 6:21 pm

somethingsomething wrote:"All instances of the metadata are corrupted"

It seems to go away by itself. After a few hours it works fine.
I've seen 2012R2 dedup lock VBM files that causes backup to fail, could be related...
DonZoomik
Influencer
 
Posts: 19
Liked: 8 times
Joined: Fri Nov 25, 2016 1:56 pm

Re: Disk errors on Windows server 2016 with deduplication

Veeam Logoby RGijsen » Fri Mar 31, 2017 1:19 pm

RGijsen
Enthusiast
 
Posts: 53
Liked: 4 times
Joined: Fri Oct 10, 2014 2:06 pm

Re: Disk errors on Windows server 2016 with deduplication

Veeam Logoby somethingsomething » Mon Apr 03, 2017 12:29 pm

DonZoomik wrote:
It seems to go away by itself. After a few hours it works fine.
I've seen 2012R2 dedup lock VBM files that causes backup to fail, could be related...


Before I applied the patch, I ran into the problem --- and it did not work fine later. Even on subsequent backups I still had the corruption.
It might have gone away after the third time I ran into the problem (post-hotfix), but I didn't want to meddle anymore.

I still have the scandisk error, so I'm done taking risks for now.

Interestingly enough, on another volume that is deduped but not being used by VEEAM, I've had no issues. Scandisk doesn't report any errors.

REFS, here I come. It's had some bugs but so far it's been more stable.
somethingsomething
Lurker
 
Posts: 2
Liked: never
Joined: Mon Feb 20, 2017 3:22 pm

Re: Disk errors on Windows server 2016 with deduplication

Veeam Logoby Delo123 » Thu Apr 06, 2017 7:44 am

We have seen these locks on 2012R2 only in the beginning when first experimenting with dedupe, in 100% of the cases (at our site) they were caused by Deduplication set to VDI Server instead of general purpose file server. in 2016 virtualized file server should be the option to choose.
Delo123
Expert
 
Posts: 345
Liked: 94 times
Joined: Fri Dec 28, 2012 5:20 pm
Full Name: Guido Meijers

Re: Disk errors on Windows server 2016 with deduplication

Veeam Logoby RGijsen » Thu Apr 06, 2017 8:02 am

We have our repository set to general purpose file server, yet we do face the metadata corruption issue. So far the only option that worked for me is to start a new active full. Waiting / rebooting never made the issue 'disappear'. I currently have a Veeam ticket on this, 02112363, but as expected it all points at Microsoft dedupe being the culprit here.

One thing I have to mention though, at the moment we are running out of space on our repository. That means that we dedup files 0 days of age. Also as we start the offsite copy as soon as a job is finished, the actual file would be in use. I've enabled OptimizeInUseFiles. I've just disabled that to see if that would make a difference. However, if I'm not mistaken we had that disabled initially. Will keep you informed.
RGijsen
Enthusiast
 
Posts: 53
Liked: 4 times
Joined: Fri Oct 10, 2014 2:06 pm

Re: Disk errors on Windows server 2016 with deduplication

Veeam Logoby RGijsen » Wed Apr 12, 2017 8:30 am

Update; we've ran for about one and a half week without Dedupe, and everything worked perfectly well. I enabled dedupe this weekend, and tonight already I again had the corrupt metadata issues again. So it really seems to be dedupe that is bugging here. Yet another MS problem. They really rushed out Server 2016 way too soon.
RGijsen
Enthusiast
 
Posts: 53
Liked: 4 times
Joined: Fri Oct 10, 2014 2:06 pm

Re: Disk errors on Windows server 2016 with deduplication

Veeam Logoby SeandG » Mon Jun 19, 2017 11:27 am

MS provided us with a private hotfix and since the implementation of the hotfix about 2 weeks ago we stopped having NTFS corruption. So far running CHKDSK on our volumes stopped reporting errors.

Cheers,
Sean
SeandG
Influencer
 
Posts: 15
Liked: 1 time
Joined: Mon Jan 30, 2017 10:38 am

Re: Disk errors on Windows server 2016 with deduplication

Veeam Logoby Mike Resseler » Tue Jun 20, 2017 3:51 am

Sean,

That sounds like good news! Any idea if MSFT will put that hotfix in the next rollup update? Or is it still under review?
Mike Resseler
Veeam Software
 
Posts: 3131
Liked: 360 times
Joined: Fri Feb 08, 2013 3:08 pm
Location: Belgium, the land of the fries, the beer, the chocolate and the diamonds...
Full Name: Mike Resseler

Re: Disk errors on Windows server 2016 with deduplication

Veeam Logoby SeandG » Mon Jun 26, 2017 8:15 am

Hi Mike,

This is the last message I have from Microsoft:
We have started the processing of generating a public fix of this change. We can let you know when it will be available once the update release date is determined

If I get more info, I will place it here

Thanks
SeandG
Influencer
 
Posts: 15
Liked: 1 time
Joined: Mon Jan 30, 2017 10:38 am

Previous

Return to Veeam Backup & Replication



Who is online

Users browsing this forum: foggy, v.Eremin and 62 guests