Disk errors on Windows server 2016 with deduplication

Availability for the Always-On Enterprise

Disk errors on Windows server 2016 with deduplication

Veeam Logoby SeandG » Tue Feb 14, 2017 11:16 am

Hi all,

We have recently upgraded our B&R servers to Windows Server 2016, we are using Windows deduplication and we have installed hotfix KB3216755 to deal with deduplication corruption issues.
We use our backup repositories only for backups, we do not save or write any other data to those drives.

We seem to be experiencing a high level of volume issues that have to be fixed with CHKDSK.
The errors we find seem to always be "The Volume Bitmap is incorrect".

On the 7th of Feb we checked all our repositories and found issues on these drives:
Server Drive
ABK01 F:
BBK01 F:
CBK01 F:
DBK02 I:
DBK02 H:

On the 8th we had no reported issues.

On the 14th we have these:
Server Drive
ABK01 F:
ABK01 G:
CBK01 F:
DBK01 F:

So we have the drive "F:" on 3 different servers reporting "The Volume Bitmap is incorrect" twice within a week and this has happened before, that is why we started testing and logging. I think drive letter F: is coincidental, that drive is Tier 2 storage on all the servers.
HP diagnostics and utilities are not reporting any issues with the servers.
The servers are fairly up to date as far as Windows updates and HP firmware and drivers are concerned. Windows was updated last week and HP firmware and drivers about a month ago.
Veeam B&R is version 9.5 with update 1.
We have not seen this behaviour on drives that are not deduplicated.

Is anyone else experiencing this?

Do you think changing the repositories to "Align backup file data blocks" could help? Should we do this anyway on our deduplicated volumes? Are any downsides?

I have logged a call on this and the call ID is: 02068714

Thanks,
Sean
SeandG
Influencer
 
Posts: 15
Liked: 1 time
Joined: Mon Jan 30, 2017 10:38 am

Re: Disk errors on Windows server 2016 with deduplication

Veeam Logoby MOBO » Tue Feb 14, 2017 12:02 pm

from Veeam Community Forums Digest for MOBO [Jan 23 - Jan 29, 2017]
THE WORD FROM GOSTEV
Major data corruption warning for those of you who have already jumped the much improved Windows Server 2016 deduplication for production use (the rest can take a deep breath). Last week, we have started to receive multiple reports on corruptions of backup files hosted on Windows Server 2016 NTFS volumes with Data Deduplication feature enabled. Luckily, the issue was easy to spot due to the system event log event (rarely the case by the way, as most storage-level corruptions go undetected - which is why it is extremely important to have storage-level corruption guard enabled in the advanced backup job settings at least when you are trying out new things).

I've already received the official confirmation from Microsoft that this is the know issue (ID 10165851) which is scheduled to be addressed in the next Windows Server 2016 servicing update. There are actually two separate issues, both leading to file corruption when using deduplication on very large files. One issue occurs when files grow to 2.2TB or larger, and another one causes loss of checksums for files with "smaller sizes" - this is the actual wording of the official note, so I have no idea how small. As such, I highly recommend assuming that all your existing backup can be damaged, and performing an active full backup to a repository backed by a volume without deduplication feature enabled. Needless to say, since those of you who are affected already have a Windows Server 2016 based repository, I highly recommend that you use ReFS.

https://forums.veeam.com/veeam-backup-replication-f2/corrupted-files-on-win2016-deduplication-t40406.html
MOBO
Influencer
 
Posts: 12
Liked: 2 times
Joined: Sat Jan 24, 2015 7:26 am
Full Name: Morten Boegeskov

Re: Disk errors on Windows server 2016 with deduplication

Veeam Logoby SeandG » Tue Feb 14, 2017 2:05 pm

@MOBO

Thanks, we went through all that pain, these symptoms are different though, but I guess they could be related. According to MS KB3216755 is preview, but it does fix the corruption issues.
As much as Veeam advise ReFS, there is no deduplication on ReFS and with the amount of times we have had to do Active Full backups our storage repositories would have been full a very long time ago.
We are left to wonder if 2016 is actually ready for production, I think not, but it is a bit too late for us.
SeandG
Influencer
 
Posts: 15
Liked: 1 time
Joined: Mon Jan 30, 2017 10:38 am

Re: Disk errors on Windows server 2016 with deduplication

Veeam Logoby SeandG » Wed Feb 15, 2017 9:14 am

Microsoft pointed me to this article:
https://support.microsoft.com/en-us/hel ... for-file-5

Does the Veeam backup copy use VSS? Could this be the cause of the problems we are having?

I guess it could that the preview KB3216755 doesn't cover all deduplication corruption issues...

Thanks
SeandG
Influencer
 
Posts: 15
Liked: 1 time
Joined: Mon Jan 30, 2017 10:38 am

Re: Disk errors on Windows server 2016 with deduplication

Veeam Logoby Mike Resseler » Wed Feb 15, 2017 9:18 am

I might be completely wrong here, but MSFT pointed you to this article? You are running 2016 right? That hotfix is only for older OS-es and is long time ago fixed and implemented in the current OS-es
Mike Resseler
Veeam Software
 
Posts: 3382
Liked: 384 times
Joined: Fri Feb 08, 2013 3:08 pm
Location: Belgium, the land of the fries, the beer, the chocolate and the diamonds...
Full Name: Mike Resseler

Re: Disk errors on Windows server 2016 with deduplication

Veeam Logoby SeandG » Wed Feb 15, 2017 12:23 pm

Yes MSFT pointed me to that article and I also noticed it was for older OS, I was just wondering if somehow this fault could have been re-introduced with Server 2016. Anyway I'm in touch with MS, hopefully something will come up.

It is just weird that we will fix the disk with CHKDSK and sometimes on the next it already has issues again. The only things that currently write to the disk are Veeam backups and, of course, the deduplication process.

Thanks
SeandG
Influencer
 
Posts: 15
Liked: 1 time
Joined: Mon Jan 30, 2017 10:38 am

Re: Disk errors on Windows server 2016 with deduplication

Veeam Logoby Gostev » Tue Feb 28, 2017 2:32 pm

"Volume Bitmap is incorrect" error message when using CHKDSK
If anyone else is seeing this issue, let me know as I need to put you in contact directly with the dedupe development team at Microsoft through the special channel - they are currently investigating and need more real-world environments to research this potential bug.

When posting, please note if you're OK with me sharing your forum registration email with the deduplication team at Microsoft - I won't share anyone's contact by default.
Gostev
Veeam Software
 
Posts: 21612
Liked: 2409 times
Joined: Sun Jan 01, 2006 1:01 am
Location: Baar, Switzerland

Re: Disk errors on Windows server 2016 with deduplication

Veeam Logoby VladV » Mon Mar 06, 2017 6:02 am

Hi Gostev,

On 19/12/2016 I experienced this issue and opened a case with Veeam. Case # 02013712. When running chkdsk on the affected drive without any switches, on stage 3 Examining security descriptors ... I got The Volume Bitmap is incorrect without any other errors.

The reason for opening the case was this error:
Code: Select all
19/12/2016 11:30:55 :: Processing XXXXX Error: All instances of the storage metadata are corrupted. Failed to restore file from local backup. VFS link: [summary.xml]. Target file: [MemFs://frontend::CDataTransferCommandSet::RestoreText_{fb87560a-f686-444a-add0-96e012ea3897}]. CHMOD mask: [721]. Agent failed to process method {DataTransfer.RestoreText


This volume is backed by a Nexenta ZFS appliance with end to end CRC check and Sync always option enabled so we excluded any issues with the storage itself. The dedup logs are clean also.

Let me know if you need anything else.

Thanks
VladV
Expert
 
Posts: 214
Liked: 24 times
Joined: Tue Apr 30, 2013 7:38 am
Full Name: Vlad Valeriu Velciu

Re: Disk errors on Windows server 2016 with deduplication

Veeam Logoby SeandG » Mon Mar 06, 2017 9:20 am

For us running CHKDSK /SCAN doesn't report the error, if we run it without switches or with /F we get "Volume Bitmap is incorrect" on a regular basis.
To detect this fault you may have to run CHKDSK without any switches.

Cheers
Sean
SeandG
Influencer
 
Posts: 15
Liked: 1 time
Joined: Mon Jan 30, 2017 10:38 am

Re: Disk errors on Windows server 2016 with deduplication

Veeam Logoby Gostev » Mon Mar 06, 2017 12:13 pm

@Vlad are you OK with me sharing your contact details with the deduplication team at Microsoft?
Gostev
Veeam Software
 
Posts: 21612
Liked: 2409 times
Joined: Sun Jan 01, 2006 1:01 am
Location: Baar, Switzerland

Re: Disk errors on Windows server 2016 with deduplication

Veeam Logoby VladV » Mon Mar 06, 2017 3:05 pm

Sure.
VladV
Expert
 
Posts: 214
Liked: 24 times
Joined: Tue Apr 30, 2013 7:38 am
Full Name: Vlad Valeriu Velciu

Re: Disk errors on Windows server 2016 with deduplication

Veeam Logoby AISD » Mon Mar 06, 2017 9:30 pm

We have this issue as well, I sent an email to dedupfeedback@microsoft.com earlier today. I opened a support request on 1/25 related to this issue. I closed it after installing KB3216755, which seemed to keep them calm for a while. However I’m seeing this occur again. I’m ready to go back to 2012 R2 so I can have greater reliability, but I will stay on 2016 if you need another person to do some testing with for any reason.

3/6/2017 4:42:22 AM :: Full backup file merge failed Error: Agent: Failed to process method {Transform.Patch}: Data error (cyclic redundancy check).
Failed to flush file buffers. File: [D:\Backups\File Servers Job\FILESHARE3.vm-96D2017-01-09T171528.vbk].
3/6/2017 5:22:27 AM :: Error: Agent: Failed to process method {Transform.Patch}: Data error (cyclic redundancy check).
Failed to flush file buffers. File: [D:\Backups\File Servers Job\FILESHARE3.vm-96D2017-01-09T171528.vbk].

Today approximately 12:00 pm

Event ID 12805
Data Deduplication service found 3 corruption(s) on volume D:\. 0 corruption(s) are fixed. 1 user file(s) are corrupted. 0 user file(s) are fixed. For the corrupted file list, see the Microsoft/Windows/Deduplication/Scrubbing events.

Event ID 12800
Data Deduplication service detected corruption in "D:\Backups\File Servers Job\FILESHARE3.vm-96D2017-01-10T171514.vbk". The corruption cannot be repaired.

Event ID 12802
Data Deduplication service detected a corrupted item (Bad checksum - 8, 0x190000, 0x0, 0x4B9, recall bitmap body) in Deduplication Chunk Store on volume D:\. See the event details for more information.
Data Deduplication service detected a corrupted item (Bad checksum - 7, 0x190000, 0x0, 0x4B9, recall bitmap body) in Deduplication Chunk Store on volume D:\. See the event details for more information.
Data Deduplication service detected a corrupted item (Bad checksum - Recall bitmap, 0x190000, 0x0, 0x4B9, recall bitmap body) in Deduplication Chunk Store on volume D:\. See the event details for more information.

Running chkdsk it give error:
“The Volume Bitmap is incorrect.”
AISD
Lurker
 
Posts: 1
Liked: never
Joined: Mon Feb 15, 2016 6:56 pm
Full Name: Nicholas Rutherford

Re: Disk errors on Windows server 2016 with deduplication

Veeam Logoby sdis68 » Sun Mar 12, 2017 7:46 am

Hi,

Same problem here, with the same event id. We have one job that produce a VBK greater than 2,2Tb, and after deduplication it corrupted.

I try to install KB3216755, I will see if it solve the problem.
sdis68
Novice
 
Posts: 3
Liked: never
Joined: Thu Jun 02, 2016 9:08 am
Full Name: Nicolas Riss

Re: Disk errors on Windows server 2016 with deduplication

Veeam Logoby SeandG » Mon Mar 13, 2017 9:21 am 1 person likes this post

Microsoft have acknowledged this to be a bug, they are not sure if it is a bug with deduplication or with NTFS, I will let you know what happens.
SeandG
Influencer
 
Posts: 15
Liked: 1 time
Joined: Mon Jan 30, 2017 10:38 am

Re: Disk errors on Windows server 2016 with deduplication

Veeam Logoby Mike Resseler » Tue Mar 14, 2017 8:00 am

@nicolas,

Please don't install KB3216755 if that server holds also a SQL installation. It could really mess up your environment :-(

@Sean: Keep us informed, that would be highly appreciated!
Mike Resseler
Veeam Software
 
Posts: 3382
Liked: 384 times
Joined: Fri Feb 08, 2013 3:08 pm
Location: Belgium, the land of the fries, the beer, the chocolate and the diamonds...
Full Name: Mike Resseler

Next

Return to Veeam Backup & Replication



Who is online

Users browsing this forum: Google [Bot], tdewin and 1 guest