Comprehensive data protection for all workloads
jrippon
Lurker
Posts: 2
Liked: never
Joined: Apr 16, 2013 4:09 pm
Full Name: Josh Rippon
Contact:

Re: Corrupted Files on Win2016 Deduplication

Post by jrippon »

anglerzac wrote: I can't get the validator tool to work, If I point it at a CopyJob which I know has 50 machines in it for example it only checks 6 machines and says they are fine.
If I point it at a file it says failed to process storage, failed to access storage file, I have the drive mapped from veeam server and can see/copy the files fine.
I'll try the scrubbing task/command and wait to see how my Veeam support call progresses.
I am also trying to check backup files over the network with the validator tool and get the same error. It doesn't matter if I try a drive mapping or UNC path. Running as Administrator with full access to both servers.

I found some more details in a log file: C:\ProgramData\Veeam\Backup\Utils\Util.VmBackupValidate
Failed to open file [**********] in readonly mode. (Veeam.Backup.Common.CCppComponentException)
<01> Error in c++: Error code: 0x00000005
<01> Error in c++: Failed to retrieve key sets from encrypted storage.

The backups are not encrypted with Veeam. Bitlocker is on but that should be transparent.

I opened a ticket # 02094563. Hopefully Veeam can figure it out.
JonSD
Novice
Posts: 6
Liked: never
Joined: Aug 05, 2014 2:11 pm
Full Name: Jon Bucud
Location: San Diego, CA
Contact:

Re: Corrupted Files on Win2016 Deduplication

Post by JonSD »

I assume at this point everyone running Server 2016 and dedupe has installed KB3216755, or subsequent roll-ups (KB4013429, KB4015438).

What's everyone's feedback >1 month after patching?
DonZoomik
Service Provider
Posts: 372
Liked: 120 times
Joined: Nov 25, 2016 1:56 pm
Full Name: Mihkel Soomere
Contact:

Re: Corrupted Files on Win2016 Deduplication

Post by DonZoomik »

No corruption issues since then.

There are some other anomalies that may or may not affect Veeam:
* Very-very-very slow optimize job start and stop (like 8 hours to start and 2-3 hours to stop) - investigating with MS.
* Occasional message during backup that all metadata copies are corrupt (per random VM, with per-VM chains) - goes away after a few hours, no detectable corruption. May be related to running optimize job but not enough data to correlate.
RGijsen
Expert
Posts: 127
Liked: 28 times
Joined: Oct 10, 2014 2:06 pm
Contact:

Re: Corrupted Files on Win2016 Deduplication

Post by RGijsen »

We also still have issues with metadata corruption on our NTFS 64k /l formatted repository. We have no backup files > 2TB (although near, the biggest is 1.8TB). Since installing KB4010672 (which should include KB3216755) we've still had metadata corruption issues - but only on one specific job. I always trigger an active full to 'fix' that. We've installed all updates up to the recent 22th March KB4016635, and still had those issues. I logged a ticket yesterday, #02112363. We do run McAfee on the host though as per policy, which we were requested to uninstall to exclude that. After uninstalling WMI was completely broken, so I am currently reinstalling our Veeam host itself. As DoomZoomik we also face:

- sometimes very slow optimize jobs, which indeed takes an hour or so to stop. I know garbagecollection does a -full every 4 time I believe, maybe optimization does a full every so many times as well.
- All instances of metadata are corrupt several times a week, consolidating to one job only so far (ticket #02112363).

We are currently working with support, the first step are to rule out a few things. So we will run without McAfee and without dedup enabled for a week and see how that goes. Will let you know.
HPersson
Novice
Posts: 5
Liked: never
Joined: Feb 15, 2013 8:05 am
Full Name: Håkan Persson
Contact:

Re: Corrupted Files on Win2016 Deduplication

Post by HPersson »

Any updates on this? Also still get "corrupted or unreadable" files regulary.
Mostly in one of the jobs that is quite small (Active full ~205GB).
Mike Resseler
Product Manager
Posts: 8190
Liked: 1320 times
Joined: Feb 08, 2013 3:08 pm
Full Name: Mike Resseler
Location: Belgium
Contact:

Re: Corrupted Files on Win2016 Deduplication

Post by Mike Resseler »

Hey,

did you install the rollups where MSFT provides fixes? If so, please log a case with support for further investigation. (Although I remain confident that the issue would be MSFT related and not Veeam related)
RGijsen
Expert
Posts: 127
Liked: 28 times
Joined: Oct 10, 2014 2:06 pm
Contact:

Re: Corrupted Files on Win2016 Deduplication

Post by RGijsen »

Update (corss-post from post237574.html#p237574)

we've ran for about one and a half week without Dedupe, and everything worked perfectly well. I enabled dedupe this weekend, and tonight already I again had the corrupt metadata issues again. So it really seems to be dedupe that is bugging here. Yet another MS problem. They really rushed out Server 2016 way too soon.
Mike Resseler
Product Manager
Posts: 8190
Liked: 1320 times
Joined: Feb 08, 2013 3:08 pm
Full Name: Mike Resseler
Location: Belgium
Contact:

Re: Corrupted Files on Win2016 Deduplication

Post by Mike Resseler »

Hey @RGijsen,

I believe you already have a support ticket open with MSFT for the shared VHDX (VHD-set) issue. Do you also have one for this issue?
RGijsen
Expert
Posts: 127
Liked: 28 times
Joined: Oct 10, 2014 2:06 pm
Contact:

Re: Corrupted Files on Win2016 Deduplication

Post by RGijsen »

No, as we have a SPLA contract with MS, which never includes support, the only option I have is buying tickets per issue from MS Pro Support. They only support break - fix scenarios. Which means they only fix things that worked before. You know MS - everything is SO litteral and not helpful. This has never worked correctly for us on 2016 with dedupe, nor can I show them a working setup with dedupe enabled. That means it does not qualify for break-fix. I hate MS so much for their support (and al their bugs over the last 2 years) as we are a small company, we do NOT qualify for Premier support (we are too small) and there is no other way for us to raise tickets. Yet I am fully dependent of MS software. I don't mind paying for proper support but the option is just not there for us.
In the ticket about the shared VHDS issue the technician made a bit of a slip of the tongue when he told me 'yes, MS treats big companies much better than small ones'. At least he was honest about it. I've had this discussion with MS over and over again. I come from a top 500 company were I only had to raise a ticket and tell them ONE time what the issue was. Then MS would run like crazy to fix it for us. It's so much different when you're small.

So bottom line: no I did not raise a ticket with MS for this dedupe issue. Maybe someone with a Premier account can do so.
JonSD
Novice
Posts: 6
Liked: never
Joined: Aug 05, 2014 2:11 pm
Full Name: Jon Bucud
Location: San Diego, CA
Contact:

Re: Corrupted Files on Win2016 Deduplication

Post by JonSD »

For what it's worth I upgraded another one of our proxies this past week to 2016 and enabled dedupe against our two largest repositories (61TB each).

Downloaded and installed the latest roll up from Windows Update Catalog (kb4015438).

Similar with RGijsen, we had one job and only one job out of 47 experience corrupt metadata (4.7TB vbk).

I opened a case with support (02129932) and uploaded a log bundle. The engineer I worked with asked I wait to hear back from them regarding whether to disable 2016 Dedupe or not.
DonZoomik
Service Provider
Posts: 372
Liked: 120 times
Joined: Nov 25, 2016 1:56 pm
Full Name: Mihkel Soomere
Contact:

Re: Corrupted Files on Win2016 Deduplication

Post by DonZoomik »

DonZoomik wrote:* Very-very-very slow optimize job start and stop (like 8 hours to start and 2-3 hours to stop) - investigating with MS.
TL;DR: the more memory and cores assigned to job, the slower starting and stopping is.

But after last full scrubs I saw a few corruptions as well. Not as massive as during RTM but still... investigating with MS.
JonSD
Novice
Posts: 6
Liked: never
Joined: Aug 05, 2014 2:11 pm
Full Name: Jon Bucud
Location: San Diego, CA
Contact:

Re: Corrupted Files on Win2016 Deduplication

Post by JonSD »

DonZoomik wrote:TL;DR: the more memory and cores assigned to job, the slower starting and stopping is.
Thanks for the update... I'd hope we would be safe with 20 cores and 192GB of RAM in a dedicated physical proxy.
RGijsen
Expert
Posts: 127
Liked: 28 times
Joined: Oct 10, 2014 2:06 pm
Contact:

Re: Corrupted Files on Win2016 Deduplication

Post by RGijsen »

For anyone with these issues, can you confirm that when the issue is there, chkdsk on the datastore volume gives some bitmap error? That's a clue I got from Veeam support. Everytime I have that metadata issue I get an error with chkdsk. Fixing it with /f does not help with the metadata issue although it seems to be related. Unfortunately I cannot log this with MS as we only have prosupport, which does break-fix only.
Gostev
Chief Product Officer
Posts: 31789
Liked: 7291 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Corrupted Files on Win2016 Deduplication

Post by Gostev »

Confirming. After the corruption happens, chkdsk will start throwing "The Volume Bitmap is incorrect" error. We've actually observed one occurrence of this corruption ourselves recently... Microsoft is investigating.
RGijsen
Expert
Posts: 127
Liked: 28 times
Joined: Oct 10, 2014 2:06 pm
Contact:

Re: Corrupted Files on Win2016 Deduplication

Post by RGijsen »

I've had another occurance of this last night. Now something came to my mind looking back at my logs. Apart from maybe 2 or 3 times, ALL occurances of this issue are on tuesdays for us. And on one job only, which has an active full every sunday. Just trying to find out if there is a weird day-of-the-week-bug; would this be just coincidence or are there more people that have this issue on specific days?
JonSD
Novice
Posts: 6
Liked: never
Joined: Aug 05, 2014 2:11 pm
Full Name: Jon Bucud
Location: San Diego, CA
Contact:

Re: Corrupted Files on Win2016 Deduplication

Post by JonSD »

Do your scrubs run on Sunday RGijsen?
RGijsen
Expert
Posts: 127
Liked: 28 times
Joined: Oct 10, 2014 2:06 pm
Contact:

Re: Corrupted Files on Win2016 Deduplication

Post by RGijsen »

Actually I initiate the dedupe myself by running a script after the last backup job, to prevent scheduled dedupe jobs to run when backups are still being put on that volume. I run this script:

Start-DedupJob -Volume V: -Type Optimization -memory 70
if ((get-date).DayOfWeek -eq 'Thursday') {Start-DedupJob V: -Type GarbageCollection}
if ((get-date).DayOfWeek -eq 'Friday') {Start-DedupJob V: -Type GarbageCollection}
if ((get-date).DayOfWeek -eq 'Saturday') {Start-DedupJob V: -Type GarbageCollection}
if ((get-date).DayOfWeek -eq 'Sunday') {Start-DedupJob V: -Type GarbageCollection}

I run the garbagecollection on days when Active Fulls run in order to dedupe most of the full backup files again. We're quite short on storage at the moment, which is why I do that.
Delo123
Veteran
Posts: 361
Liked: 109 times
Joined: Dec 28, 2012 5:20 pm
Full Name: Guido Meijers
Contact:

Re: Corrupted Files on Win2016 Deduplication

Post by Delo123 »

Guys, just to be sure. You all have set Dedupe to General purpose file server or virtual backup server and not to VDI Server? VDI Servers already caused tons of issues in 2012...
For us in 2016 with all updates and Virtual Backup Server 0 issues until now with over 140TB of Data on 3 Volumes.
RGijsen
Expert
Posts: 127
Liked: 28 times
Joined: Oct 10, 2014 2:06 pm
Contact:

Re: Corrupted Files on Win2016 Deduplication

Post by RGijsen »

We use general purpose.
Delo123
Veteran
Posts: 361
Liked: 109 times
Joined: Dec 28, 2012 5:20 pm
Full Name: Guido Meijers
Contact:

Re: Corrupted Files on Win2016 Deduplication

Post by Delo123 »

Yeah, i just noticed in the other thread.
Did you enable OptimizeInUseFiles again? Because this is the actual difference to VDI if i'm not mistaken. At least with OptimizeInUseFiles we ran into a lot of issues years ago.
RGijsen
Expert
Posts: 127
Liked: 28 times
Joined: Oct 10, 2014 2:06 pm
Contact:

Re: Corrupted Files on Win2016 Deduplication

Post by RGijsen »

It's enabled now but I've had it disabled if I remember correctly to no avail. But I'll disable it right now, and see how that goes.

[edit]
no, I'll change the setting tomorrow. I'm curious if it strikes us again on Tuesday as seems to be more rule than exception in the past.
Delo123
Veteran
Posts: 361
Liked: 109 times
Joined: Dec 28, 2012 5:20 pm
Full Name: Guido Meijers
Contact:

Re: Corrupted Files on Win2016 Deduplication

Post by Delo123 »

Ah ok, as said. Optimize files in use failed on us every time the dedupe process was still running (big increments and fulls) as soon Veeam tried to access the files.
Anyway, it doesn't make sense to process in use files anyway (very scary!).

Ok, let's see what happens afer tomorrow then
RGijsen
Expert
Posts: 127
Liked: 28 times
Joined: Oct 10, 2014 2:06 pm
Contact:

Re: Corrupted Files on Win2016 Deduplication

Post by RGijsen » 1 person likes this post

Sure, scary, but I trigger the dedup myself after all jobs that write are done. However, during the copy jobs to offsite some files are still accessed for read ofcourse, which could very well be deduped at the same time in my oppinion. Still I've disabled it now. I've not had the error last night by the way breaking a streak of several weeks of error on tuesdays.
Delo123
Veteran
Posts: 361
Liked: 109 times
Joined: Dec 28, 2012 5:20 pm
Full Name: Guido Meijers
Contact:

Re: Corrupted Files on Win2016 Deduplication

Post by Delo123 »

But you had it enabled still yesterday right? So the reason for the job not failing is still unknown? :(
I know what you mean regarding reading only, but actually i do not see a single reason why one should want to actually dedupe in use files (regarding to backups) so yeah we disabled it on every repository just in case...

Ok, let's see how it goes for you now... How much data do you have on your Dedupe Volumes anyway? Volumes are around 62TB also in your case?
Delo123
Veteran
Posts: 361
Liked: 109 times
Joined: Dec 28, 2012 5:20 pm
Full Name: Guido Meijers
Contact:

Re: Corrupted Files on Win2016 Deduplication

Post by Delo123 »

RGijssen. How are things after disabling the open files thing? Or something new popped up?
Ctek
Service Provider
Posts: 84
Liked: 13 times
Joined: Nov 11, 2015 3:50 pm
Location: Canada
Contact:

Re: Corrupted Files on Win2016 Deduplication

Post by Ctek »

I confirm the same deduplication behavior described in this thread, with a fully patched W2016 server. All our VBK files are <500GB.

:(

This is not ready for prime time yet IMHO.
VMCE
Delo123
Veteran
Posts: 361
Liked: 109 times
Joined: Dec 28, 2012 5:20 pm
Full Name: Guido Meijers
Contact:

Re: Corrupted Files on Win2016 Deduplication

Post by Delo123 »

Ctek: Do you by any change have OptimizeInUseFile enabled or set Data deduplication type to "Virtual Desktop Infrastructure (VDI) server"?
Ctek
Service Provider
Posts: 84
Liked: 13 times
Joined: Nov 11, 2015 3:50 pm
Location: Canada
Contact:

Re: Corrupted Files on Win2016 Deduplication

Post by Ctek »

OptimizeInUseFile is enabled (Probably by default, I have not changed this setting on my own).

Data deduplication type is set to Virtualized backup server.
VMCE
Delo123
Veteran
Posts: 361
Liked: 109 times
Joined: Dec 28, 2012 5:20 pm
Full Name: Guido Meijers
Contact:

Re: Corrupted Files on Win2016 Deduplication

Post by Delo123 »

My bad... We have set is for "general file server", this disables OptimitzeinUseFiles, but at least OptimizePartialfiles should be disabled (which is the VDI setting). Then again it might be safer to go for general file server, but just guessing :(
RGijsen
Expert
Posts: 127
Liked: 28 times
Joined: Oct 10, 2014 2:06 pm
Contact:

Re: Corrupted Files on Win2016 Deduplication

Post by RGijsen »

Delo123 wrote:RGijssen. How are things after disabling the open files thing? Or something new popped up?
Last night I hit the issue again with OptimizeInUseFiles disabled. And again on Tuesday. Again on the same backup job (not even the biggest one).
Post Reply

Who is online

Users browsing this forum: Bing [Bot], Semrush [Bot] and 32 guests