Corrupted Files on Win2016 Deduplication

[jrippon]

I can't get the validator tool to work, If I point it at a CopyJob which I know has 50 machines in it for example it only checks 6 machines and says they are fine.
If I point it at a file it says failed to process storage, failed to access storage file, I have the drive mapped from veeam server and can see/copy the files fine.
I'll try the scrubbing task/command and wait to see how my Veeam support call progresses.

I am also trying to check backup files over the network with the validator tool and get the same error. It doesn't matter if I try a drive mapping or UNC path. Running as Administrator with full access to both servers.

I found some more details in a log file: C:\ProgramData\Veeam\Backup\Utils\Util.VmBackupValidate
Failed to open file [**********] in readonly mode. (Veeam.Backup.Common.CCppComponentException)
<01> Error in c++: Error code: 0x00000005
<01> Error in c++: Failed to retrieve key sets from encrypted storage.

The backups are not encrypted with Veeam. Bitlocker is on but that should be transparent.

I opened a ticket # 02094563. Hopefully Veeam can figure it out.

[JonSD]

I assume at this point everyone running Server 2016 and dedupe has installed KB3216755, or subsequent roll-ups (KB4013429, KB4015438).

What's everyone's feedback >1 month after patching?

[DonZoomik]

No corruption issues since then.

There are some other anomalies that may or may not affect Veeam:
* Very-very-very slow optimize job start and stop (like 8 hours to start and 2-3 hours to stop) - investigating with MS.
* Occasional message during backup that all metadata copies are corrupt (per random VM, with per-VM chains) - goes away after a few hours, no detectable corruption. May be related to running optimize job but not enough data to correlate.

[RGijsen]

We also still have issues with metadata corruption on our NTFS 64k /l formatted repository. We have no backup files > 2TB (although near, the biggest is 1.8TB). Since installing KB4010672 (which should include KB3216755) we've still had metadata corruption issues - but only on one specific job. I always trigger an active full to 'fix' that. We've installed all updates up to the recent 22th March KB4016635, and still had those issues. I logged a ticket yesterday, #02112363. We do run McAfee on the host though as per policy, which we were requested to uninstall to exclude that. After uninstalling WMI was completely broken, so I am currently reinstalling our Veeam host itself. As DoomZoomik we also face:

- sometimes very slow optimize jobs, which indeed takes an hour or so to stop. I know garbagecollection does a -full every 4 time I believe, maybe optimization does a full every so many times as well.
- All instances of metadata are corrupt several times a week, consolidating to one job only so far (ticket #02112363).

We are currently working with support, the first step are to rule out a few things. So we will run without McAfee and without dedup enabled for a week and see how that goes. Will let you know.

[HPersson]

Any updates on this? Also still get "corrupted or unreadable" files regulary.
Mostly in one of the jobs that is quite small (Active full ~205GB).

[Mike Resseler]

Hey,

did you install the rollups where MSFT provides fixes? If so, please log a case with support for further investigation. (Although I remain confident that the issue would be MSFT related and not Veeam related)

[RGijsen]

Update (corss-post from post237574.html#p237574)

we've ran for about one and a half week without Dedupe, and everything worked perfectly well. I enabled dedupe this weekend, and tonight already I again had the corrupt metadata issues again. So it really seems to be dedupe that is bugging here. Yet another MS problem. They really rushed out Server 2016 way too soon.

[Mike Resseler]

Hey @RGijsen,

I believe you already have a support ticket open with MSFT for the shared VHDX (VHD-set) issue. Do you also have one for this issue?

[RGijsen]

No, as we have a SPLA contract with MS, which never includes support, the only option I have is buying tickets per issue from MS Pro Support. They only support break - fix scenarios. Which means they only fix things that worked before. You know MS - everything is SO litteral and not helpful. This has never worked correctly for us on 2016 with dedupe, nor can I show them a working setup with dedupe enabled. That means it does not qualify for break-fix. I hate MS so much for their support (and al their bugs over the last 2 years) as we are a small company, we do NOT qualify for Premier support (we are too small) and there is no other way for us to raise tickets. Yet I am fully dependent of MS software. I don't mind paying for proper support but the option is just not there for us.
In the ticket about the shared VHDS issue the technician made a bit of a slip of the tongue when he told me 'yes, MS treats big companies much better than small ones'. At least he was honest about it. I've had this discussion with MS over and over again. I come from a top 500 company were I only had to raise a ticket and tell them ONE time what the issue was. Then MS would run like crazy to fix it for us. It's so much different when you're small.

So bottom line: no I did not raise a ticket with MS for this dedupe issue. Maybe someone with a Premier account can do so.

[JonSD]

For what it's worth I upgraded another one of our proxies this past week to 2016 and enabled dedupe against our two largest repositories (61TB each).

Downloaded and installed the latest roll up from Windows Update Catalog (kb4015438).

Similar with RGijsen, we had one job and only one job out of 47 experience corrupt metadata (4.7TB vbk).

I opened a case with support (02129932) and uploaded a log bundle. The engineer I worked with asked I wait to hear back from them regarding whether to disable 2016 Dedupe or not.

[DonZoomik]

* Very-very-very slow optimize job start and stop (like 8 hours to start and 2-3 hours to stop) - investigating with MS.

TL;DR: the more memory and cores assigned to job, the slower starting and stopping is.

But after last full scrubs I saw a few corruptions as well. Not as massive as during RTM but still... investigating with MS.

[JonSD]

TL;DR: the more memory and cores assigned to job, the slower starting and stopping is.

Thanks for the update... I'd hope we would be safe with 20 cores and 192GB of RAM in a dedicated physical proxy.

[RGijsen]

For anyone with these issues, can you confirm that when the issue is there, chkdsk on the datastore volume gives some bitmap error? That's a clue I got from Veeam support. Everytime I have that metadata issue I get an error with chkdsk. Fixing it with /f does not help with the metadata issue although it seems to be related. Unfortunately I cannot log this with MS as we only have prosupport, which does break-fix only.

[Gostev]

Confirming. After the corruption happens, chkdsk will start throwing "The Volume Bitmap is incorrect" error. We've actually observed one occurrence of this corruption ourselves recently... Microsoft is investigating.

[RGijsen]

I've had another occurance of this last night. Now something came to my mind looking back at my logs. Apart from maybe 2 or 3 times, ALL occurances of this issue are on tuesdays for us. And on one job only, which has an active full every sunday. Just trying to find out if there is a weird day-of-the-week-bug; would this be just coincidence or are there more people that have this issue on specific days?

[JonSD]

Do your scrubs run on Sunday RGijsen?

[RGijsen]

Actually I initiate the dedupe myself by running a script after the last backup job, to prevent scheduled dedupe jobs to run when backups are still being put on that volume. I run this script:

Start-DedupJob -Volume V: -Type Optimization -memory 70
if ((get-date).DayOfWeek -eq 'Thursday') {Start-DedupJob V: -Type GarbageCollection}
if ((get-date).DayOfWeek -eq 'Friday') {Start-DedupJob V: -Type GarbageCollection}
if ((get-date).DayOfWeek -eq 'Saturday') {Start-DedupJob V: -Type GarbageCollection}
if ((get-date).DayOfWeek -eq 'Sunday') {Start-DedupJob V: -Type GarbageCollection}

I run the garbagecollection on days when Active Fulls run in order to dedupe most of the full backup files again. We're quite short on storage at the moment, which is why I do that.

[Delo123]

Guys, just to be sure. You all have set Dedupe to General purpose file server or virtual backup server and not to VDI Server? VDI Servers already caused tons of issues in 2012...
For us in 2016 with all updates and Virtual Backup Server 0 issues until now with over 140TB of Data on 3 Volumes.

[RGijsen]

We use general purpose.

[Delo123]

Yeah, i just noticed in the other thread.
Did you enable OptimizeInUseFiles again? Because this is the actual difference to VDI if i'm not mistaken. At least with OptimizeInUseFiles we ran into a lot of issues years ago.

[RGijsen]

It's enabled now but I've had it disabled if I remember correctly to no avail. But I'll disable it right now, and see how that goes.

[edit]
no, I'll change the setting tomorrow. I'm curious if it strikes us again on Tuesday as seems to be more rule than exception in the past.

[Delo123]

Ah ok, as said. Optimize files in use failed on us every time the dedupe process was still running (big increments and fulls) as soon Veeam tried to access the files.
Anyway, it doesn't make sense to process in use files anyway (very scary!).

Ok, let's see what happens afer tomorrow then

[RGijsen]

Sure, scary, but I trigger the dedup myself after all jobs that write are done. However, during the copy jobs to offsite some files are still accessed for read ofcourse, which could very well be deduped at the same time in my oppinion. Still I've disabled it now. I've not had the error last night by the way breaking a streak of several weeks of error on tuesdays.

[Delo123]

But you had it enabled still yesterday right? So the reason for the job not failing is still unknown?
I know what you mean regarding reading only, but actually i do not see a single reason why one should want to actually dedupe in use files (regarding to backups) so yeah we disabled it on every repository just in case...

Ok, let's see how it goes for you now... How much data do you have on your Dedupe Volumes anyway? Volumes are around 62TB also in your case?

[Delo123]

RGijssen. How are things after disabling the open files thing? Or something new popped up?

[Ctek]

I confirm the same deduplication behavior described in this thread, with a fully patched W2016 server. All our VBK files are <500GB.



This is not ready for prime time yet IMHO.

[Delo123]

Ctek: Do you by any change have OptimizeInUseFile enabled or set Data deduplication type to "Virtual Desktop Infrastructure (VDI) server"?

[Ctek]

OptimizeInUseFile is enabled (Probably by default, I have not changed this setting on my own).

Data deduplication type is set to Virtualized backup server.

[Delo123]

My bad... We have set is for "general file server", this disables OptimitzeinUseFiles, but at least OptimizePartialfiles should be disabled (which is the VDI setting). Then again it might be safer to go for general file server, but just guessing

[RGijsen]

RGijssen. How are things after disabling the open files thing? Or something new popped up?

Last night I hit the issue again with OptimizeInUseFiles disabled. And again on Tuesday. Again on the same backup job (not even the biggest one).