Corrupted Files on Win2016 Deduplication

Availability for the Always-On Enterprise

Re: Corrupted Files on Win2016 Deduplication

Veeam Logoby jrippon » Tue Mar 14, 2017 2:34 pm

anglerzac wrote:I can't get the validator tool to work, If I point it at a CopyJob which I know has 50 machines in it for example it only checks 6 machines and says they are fine.
If I point it at a file it says failed to process storage, failed to access storage file, I have the drive mapped from veeam server and can see/copy the files fine.
I'll try the scrubbing task/command and wait to see how my Veeam support call progresses.


I am also trying to check backup files over the network with the validator tool and get the same error. It doesn't matter if I try a drive mapping or UNC path. Running as Administrator with full access to both servers.

I found some more details in a log file: C:\ProgramData\Veeam\Backup\Utils\Util.VmBackupValidate
Failed to open file [**********] in readonly mode. (Veeam.Backup.Common.CCppComponentException)
<01> Error in c++: Error code: 0x00000005
<01> Error in c++: Failed to retrieve key sets from encrypted storage.

The backups are not encrypted with Veeam. Bitlocker is on but that should be transparent.

I opened a ticket # 02094563. Hopefully Veeam can figure it out.
jrippon
Lurker
 
Posts: 2
Liked: never
Joined: Tue Apr 16, 2013 4:09 pm
Full Name: Josh Rippon

Re: Corrupted Files on Win2016 Deduplication

Veeam Logoby JonSD » Tue Mar 28, 2017 12:25 pm

I assume at this point everyone running Server 2016 and dedupe has installed KB3216755, or subsequent roll-ups (KB4013429, KB4015438).

What's everyone's feedback >1 month after patching?
JonSD
Novice
 
Posts: 5
Liked: never
Joined: Tue Aug 05, 2014 2:11 pm
Location: San Diego, CA
Full Name: Jon Bucud

Re: Corrupted Files on Win2016 Deduplication

Veeam Logoby DonZoomik » Tue Mar 28, 2017 1:04 pm

No corruption issues since then.

There are some other anomalies that may or may not affect Veeam:
* Very-very-very slow optimize job start and stop (like 8 hours to start and 2-3 hours to stop) - investigating with MS.
* Occasional message during backup that all metadata copies are corrupt (per random VM, with per-VM chains) - goes away after a few hours, no detectable corruption. May be related to running optimize job but not enough data to correlate.
DonZoomik
Influencer
 
Posts: 16
Liked: 7 times
Joined: Fri Nov 25, 2016 1:56 pm

Re: Corrupted Files on Win2016 Deduplication

Veeam Logoby RGijsen » Wed Mar 29, 2017 10:42 am

We also still have issues with metadata corruption on our NTFS 64k /l formatted repository. We have no backup files > 2TB (although near, the biggest is 1.8TB). Since installing KB4010672 (which should include KB3216755) we've still had metadata corruption issues - but only on one specific job. I always trigger an active full to 'fix' that. We've installed all updates up to the recent 22th March KB4016635, and still had those issues. I logged a ticket yesterday, #02112363. We do run McAfee on the host though as per policy, which we were requested to uninstall to exclude that. After uninstalling WMI was completely broken, so I am currently reinstalling our Veeam host itself. As DoomZoomik we also face:

- sometimes very slow optimize jobs, which indeed takes an hour or so to stop. I know garbagecollection does a -full every 4 time I believe, maybe optimization does a full every so many times as well.
- All instances of metadata are corrupt several times a week, consolidating to one job only so far (ticket #02112363).

We are currently working with support, the first step are to rule out a few things. So we will run without McAfee and without dedup enabled for a week and see how that goes. Will let you know.
RGijsen
Enthusiast
 
Posts: 43
Liked: 2 times
Joined: Fri Oct 10, 2014 2:06 pm

Re: Corrupted Files on Win2016 Deduplication

Veeam Logoby HPersson » Tue Apr 11, 2017 9:32 am

Any updates on this? Also still get "corrupted or unreadable" files regulary.
Mostly in one of the jobs that is quite small (Active full ~205GB).
HPersson
Novice
 
Posts: 5
Liked: never
Joined: Fri Feb 15, 2013 8:05 am
Full Name: Håkan Persson

Re: Corrupted Files on Win2016 Deduplication

Veeam Logoby Mike Resseler » Wed Apr 12, 2017 7:49 am

Hey,

did you install the rollups where MSFT provides fixes? If so, please log a case with support for further investigation. (Although I remain confident that the issue would be MSFT related and not Veeam related)
Mike Resseler
Veeam Software
 
Posts: 2616
Liked: 314 times
Joined: Fri Feb 08, 2013 3:08 pm
Location: Belgium, the land of the fries, the beer, the chocolate and the diamonds...
Full Name: Mike Resseler

Re: Corrupted Files on Win2016 Deduplication

Veeam Logoby RGijsen » Wed Apr 12, 2017 8:32 am

Update (corss-post from post237574.html#p237574)

we've ran for about one and a half week without Dedupe, and everything worked perfectly well. I enabled dedupe this weekend, and tonight already I again had the corrupt metadata issues again. So it really seems to be dedupe that is bugging here. Yet another MS problem. They really rushed out Server 2016 way too soon.
RGijsen
Enthusiast
 
Posts: 43
Liked: 2 times
Joined: Fri Oct 10, 2014 2:06 pm

Re: Corrupted Files on Win2016 Deduplication

Veeam Logoby Mike Resseler » Wed Apr 12, 2017 9:28 am

Hey @RGijsen,

I believe you already have a support ticket open with MSFT for the shared VHDX (VHD-set) issue. Do you also have one for this issue?
Mike Resseler
Veeam Software
 
Posts: 2616
Liked: 314 times
Joined: Fri Feb 08, 2013 3:08 pm
Location: Belgium, the land of the fries, the beer, the chocolate and the diamonds...
Full Name: Mike Resseler

Re: Corrupted Files on Win2016 Deduplication

Veeam Logoby RGijsen » Wed Apr 12, 2017 9:48 am

No, as we have a SPLA contract with MS, which never includes support, the only option I have is buying tickets per issue from MS Pro Support. They only support break - fix scenarios. Which means they only fix things that worked before. You know MS - everything is SO litteral and not helpful. This has never worked correctly for us on 2016 with dedupe, nor can I show them a working setup with dedupe enabled. That means it does not qualify for break-fix. I hate MS so much for their support (and al their bugs over the last 2 years) as we are a small company, we do NOT qualify for Premier support (we are too small) and there is no other way for us to raise tickets. Yet I am fully dependent of MS software. I don't mind paying for proper support but the option is just not there for us.
In the ticket about the shared VHDS issue the technician made a bit of a slip of the tongue when he told me 'yes, MS treats big companies much better than small ones'. At least he was honest about it. I've had this discussion with MS over and over again. I come from a top 500 company were I only had to raise a ticket and tell them ONE time what the issue was. Then MS would run like crazy to fix it for us. It's so much different when you're small.

So bottom line: no I did not raise a ticket with MS for this dedupe issue. Maybe someone with a Premier account can do so.
RGijsen
Enthusiast
 
Posts: 43
Liked: 2 times
Joined: Fri Oct 10, 2014 2:06 pm

Re: Corrupted Files on Win2016 Deduplication

Veeam Logoby JonSD » Mon Apr 17, 2017 4:06 pm

For what it's worth I upgraded another one of our proxies this past week to 2016 and enabled dedupe against our two largest repositories (61TB each).

Downloaded and installed the latest roll up from Windows Update Catalog (kb4015438).

Similar with RGijsen, we had one job and only one job out of 47 experience corrupt metadata (4.7TB vbk).

I opened a case with support (02129932) and uploaded a log bundle. The engineer I worked with asked I wait to hear back from them regarding whether to disable 2016 Dedupe or not.
JonSD
Novice
 
Posts: 5
Liked: never
Joined: Tue Aug 05, 2014 2:11 pm
Location: San Diego, CA
Full Name: Jon Bucud

Re: Corrupted Files on Win2016 Deduplication

Veeam Logoby DonZoomik » Mon Apr 17, 2017 7:24 pm

DonZoomik wrote:* Very-very-very slow optimize job start and stop (like 8 hours to start and 2-3 hours to stop) - investigating with MS.
TL;DR: the more memory and cores assigned to job, the slower starting and stopping is.

But after last full scrubs I saw a few corruptions as well. Not as massive as during RTM but still... investigating with MS.
DonZoomik
Influencer
 
Posts: 16
Liked: 7 times
Joined: Fri Nov 25, 2016 1:56 pm

Re: Corrupted Files on Win2016 Deduplication

Veeam Logoby JonSD » Mon Apr 17, 2017 11:22 pm

DonZoomik wrote:TL;DR: the more memory and cores assigned to job, the slower starting and stopping is.

Thanks for the update... I'd hope we would be safe with 20 cores and 192GB of RAM in a dedicated physical proxy.
JonSD
Novice
 
Posts: 5
Liked: never
Joined: Tue Aug 05, 2014 2:11 pm
Location: San Diego, CA
Full Name: Jon Bucud

Re: Corrupted Files on Win2016 Deduplication

Veeam Logoby RGijsen » Tue Apr 18, 2017 10:09 am

For anyone with these issues, can you confirm that when the issue is there, chkdsk on the datastore volume gives some bitmap error? That's a clue I got from Veeam support. Everytime I have that metadata issue I get an error with chkdsk. Fixing it with /f does not help with the metadata issue although it seems to be related. Unfortunately I cannot log this with MS as we only have prosupport, which does break-fix only.
RGijsen
Enthusiast
 
Posts: 43
Liked: 2 times
Joined: Fri Oct 10, 2014 2:06 pm

Re: Corrupted Files on Win2016 Deduplication

Veeam Logoby Gostev » Tue Apr 18, 2017 10:54 pm

Confirming. After the corruption happens, chkdsk will start throwing "The Volume Bitmap is incorrect" error. We've actually observed one occurrence of this corruption ourselves recently... Microsoft is investigating.
Gostev
Veeam Software
 
Posts: 21047
Liked: 2266 times
Joined: Sun Jan 01, 2006 1:01 am
Full Name: Anton Gostev

Re: Corrupted Files on Win2016 Deduplication

Veeam Logoby RGijsen » Tue Apr 25, 2017 7:25 am

I've had another occurance of this last night. Now something came to my mind looking back at my logs. Apart from maybe 2 or 3 times, ALL occurances of this issue are on tuesdays for us. And on one job only, which has an active full every sunday. Just trying to find out if there is a weird day-of-the-week-bug; would this be just coincidence or are there more people that have this issue on specific days?
RGijsen
Enthusiast
 
Posts: 43
Liked: 2 times
Joined: Fri Oct 10, 2014 2:06 pm

PreviousNext

Return to Veeam Backup & Replication



Who is online

Users browsing this forum: Bing [Bot], driley, pdkilian and 36 guests