Comprehensive data protection for all workloads
Locked
wingphil
Novice
Posts: 7
Liked: never
Joined: Jun 11, 2018 8:51 am
Contact:

Re: REFS issues (server lockups, high CPU, high RAM)

Post by wingphil » Jun 12, 2018 11:30 am

Hi,

I also would like to know how to extract the refs.sys file from this patch. I've found the manifest for it at "Windows10.0-KB4093120-x64\Cab_1_for_KB4093120\amd64_microsoft-windows-refs_31bf3856ad364e35_10.0.14393.2097_none_1c46919c776b0217.manifest" - no idea how to get the actual file.

Thanks,

Phil

jslic
Novice
Posts: 3
Liked: 4 times
Joined: Jun 20, 2016 8:30 am
Full Name: Jesper Sorensen
Contact:

Re: REFS issues (server lockups, high CPU, high RAM)

Post by jslic » Jun 12, 2018 11:58 am 1 person likes this post

Had the same issue with trying to extract the refs.sys - 10.0.14393.2097 from the update.
Ended up with installing a new VM, applying the WU patch (https://www.catalog.update.microsoft.co ... =KB4093120) and then extracting the file from C:\windows\system32\drivers

Gostev
SVP, Product Management
Posts: 24473
Liked: 3413 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: REFS issues (server lockups, high CPU, high RAM)

Post by Gostev » Jun 12, 2018 12:00 pm

This is what I would do too. Perhaps there's an easier way, but I don't know one.

mkaec
Expert
Posts: 318
Liked: 72 times
Joined: Jul 16, 2015 1:31 pm
Full Name: Marc K
Contact:

Re: REFS issues (server lockups, high CPU, high RAM)

Post by mkaec » Jun 12, 2018 3:26 pm

I think the suggestion is to extract the ReFS.sys file from the patch and then manually replace it on the impacted system(s).

dellock6
Veeam Software
Posts: 5689
Liked: 1604 times
Joined: Jul 26, 2009 3:39 pm
Full Name: Luca Dell'Oca
Location: Varese, Italy
Contact:

Re: REFS issues (server lockups, high CPU, high RAM)

Post by dellock6 » Jun 12, 2018 4:07 pm

Indeed, also because if you simply try to re-apply the patch, Windows Update will tell you that it's already applied.
Luca Dell'Oca
Principal EMEA Cloud Architect @ Veeam Software

@dellock6
https://www.virtualtothecore.com/
vExpert 2011 -> 2019
Veeam VMCE #1

Gostev
SVP, Product Management
Posts: 24473
Liked: 3413 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: REFS issues (server lockups, high CPU, high RAM)

Post by Gostev » Jun 12, 2018 4:17 pm

We meant applying patch on clean Windows install ;)

DonZoomik
Expert
Posts: 111
Liked: 27 times
Joined: Nov 25, 2016 1:56 pm
Contact:

Re: REFS issues (server lockups, high CPU, high RAM)

Post by DonZoomik » Jun 12, 2018 9:10 pm

Todays patch includes new driver with version 10.0.14939.2312

wingphil
Novice
Posts: 7
Liked: never
Joined: Jun 11, 2018 8:51 am
Contact:

Re: REFS issues (server lockups, high CPU, high RAM)

Post by wingphil » Jun 13, 2018 9:57 am

Interesting, will install today's patch and report back. Can't be much worse than my current version (10.0.14939.2273)

wingphil
Novice
Posts: 7
Liked: never
Joined: Jun 11, 2018 8:51 am
Contact:

Re: REFS issues (server lockups, high CPU, high RAM)

Post by wingphil » Jun 14, 2018 9:17 am

Well anecdotally it's no faster, a full compact on one of my jobs took 9.5hrs with 2312, same as 2273. Memory usage might be a little better but that might be other factors.

nezzer
Service Provider
Posts: 17
Liked: 5 times
Joined: Apr 27, 2018 11:40 am
Full Name: Andreas Svensson
Contact:

Re: REFS issues (server lockups, high CPU, high RAM)

Post by nezzer » Jun 14, 2018 9:48 am

We removed the mentioned patch and we are now running at full speed again.
I'm not sure I like the workaround changing the REFS.SYS driver on a fully patched system.

I will wait and recommend others to wait for a working Windows patch from Microsoft (:

mvalpreda
Enthusiast
Posts: 48
Liked: 1 time
Joined: May 06, 2015 10:57 pm
Full Name: Mark Valpreda
Contact:

Re: REFS issues (server lockups, high CPU, high RAM)

Post by mvalpreda » Jun 14, 2018 7:25 pm

What is the version on refs.sys that Veeam got that worked? It obviously isn't in the June updates. I hate getting that far behind!

Gostev
SVP, Product Management
Posts: 24473
Liked: 3413 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: REFS issues (server lockups, high CPU, high RAM)

Post by Gostev » Jun 15, 2018 11:24 am

Ours was VERY private :D we even had to disable driver signature check in the OS to use one! I suppose by now you should be able to get a better version by opening a support case with Microsoft. The internal bug ID to link the support case to is 17762340.

NightBird
Service Provider
Posts: 175
Liked: 32 times
Joined: Apr 28, 2009 8:33 am
Location: Strasbourg, FRANCE
Contact:

Re: REFS issues (server lockups, high CPU, high RAM)

Post by NightBird » Jun 15, 2018 5:28 pm

Another silent ReFS.sys driver update with June cumulative... (10.0.14393.2312) what a pain...
Nothing in the release notes... April, May, June => 3 silent Refs.sys driver update !!! (2248 2273 2312)

Gostev
SVP, Product Management
Posts: 24473
Liked: 3413 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: REFS issues (server lockups, high CPU, high RAM)

Post by Gostev » Jun 15, 2018 10:39 pm

Being on the Microsoft MVP distribution list I can tell you even they are mad about this by now :D particularly because many are Microsoft partners selling S2D/ReFS based solutions.

FrancWest
Expert
Posts: 103
Liked: 7 times
Joined: Sep 17, 2017 3:20 am
Full Name: Franc
Contact:

Re: REFS issues (server lockups, high CPU, high RAM)

Post by FrancWest » Jun 16, 2018 10:21 pm

You can extract refs.sys from the patch file without first installing a clean windows 2016 and then installing the patch.

download the full update package from https://www.catalog.update.microsoft.co ... =KB4093120

open a command prompt and do:

expand -F:* <update>.msu <target_dir>

you end up with several .cab files

go to the target folder you extracted the .msu to and then do:

expand -f:refs.sys *.cab .

you end of with a folder called 'amd64_microsoft-windows-refs_31bf3856ad364e35_10.0.14393.2097_none_1c46919c776b0217' in the current folder.

Inside this folder is refs.sys with version 10.0.14393.2097

Franc.

paallergy
Enthusiast
Posts: 35
Liked: 5 times
Joined: May 18, 2010 7:10 pm
Full Name: Jonathan Schultz
Contact:

Re: REFS issues (server lockups, high CPU, high RAM)

Post by paallergy » Jun 17, 2018 4:54 am 2 people like this post

@Franc: Thank you for the instructions

I had to do one more expand, after the msu and before refs:
expand -F:*.cab *.cab <target_dir_2>

So:
1. download and go to download directory
2. expand -F:* <update>.msu <target_dir_1>
3. cd <target_dir_1>
4. expand -F:*.cab *.cab <target_dir_2> // this results in several cab files
5. cd <target_dir_2>
6. expand -f:refs.sys *.cab . (or expand -f:refs.sys *.cab <target_dir_3>)

frankive
Service Provider
Posts: 884
Liked: 104 times
Joined: May 14, 2013 8:35 pm
Full Name: Frank Iversen
Location: Norway
Contact:

[MERGED] Refs issue

Post by frankive » Jun 18, 2018 1:22 pm

Hi.
Anyone tested if the june CU for server 2016 fix the refs issue?

dellock6
Veeam Software
Posts: 5689
Liked: 1604 times
Joined: Jul 26, 2009 3:39 pm
Full Name: Luca Dell'Oca
Location: Varese, Italy
Contact:

Re: Refs issue

Post by dellock6 » Jun 19, 2018 1:57 pm

Not personally, but you can have better feedback by using the general discussion about ReFS in the Veeam Backup & Replication public forum, here: https://forums.veeam.com/veeam-backup-r ... 40629.html
Luca Dell'Oca
Principal EMEA Cloud Architect @ Veeam Software

@dellock6
https://www.virtualtothecore.com/
vExpert 2011 -> 2019
Veeam VMCE #1

HenrikS.
Novice
Posts: 6
Liked: 5 times
Joined: Jul 04, 2017 12:59 pm
Full Name: Henrik Schewe
Contact:

Re: REFS issues (server lockups, high CPU, high RAM)

Post by HenrikS. » Jun 20, 2018 6:52 am 1 person likes this post

We just stumbled upon this after patching our backup and repository servers. Fast clone operations went from around 1,5hours to 60!hours.
After manually reverting back to the older ReFS.sys driver, we're back to normal operation again.
Looking forward to the fixed driver from MS...

gm2783
Service Provider
Posts: 6
Liked: 2 times
Joined: Apr 10, 2017 12:42 pm
Full Name: Giuseppe Marchese
Location: Mägenwil
Contact:

Re: REFS issues (server lockups, high CPU, high RAM)

Post by gm2783 » Jun 22, 2018 9:11 am

Hi all.

Wich Version of the driver is the best choice at the moment?

Regards
Giuseppe Marchese

Gostev
SVP, Product Management
Posts: 24473
Liked: 3413 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: REFS issues (server lockups, high CPU, high RAM)

Post by Gostev » Jun 22, 2018 12:58 pm

Hi Giuseppe, see last post on the previous page.

jbader
Novice
Posts: 4
Liked: never
Joined: May 22, 2018 4:08 pm
Full Name: John Bader
Contact:

Re: REFS issues (server lockups, high CPU, high RAM)

Post by jbader » Jun 25, 2018 7:21 pm

Same issue here. Anyone find a solution?

jbader
Novice
Posts: 4
Liked: never
Joined: May 22, 2018 4:08 pm
Full Name: John Bader
Contact:

Re: REFS issues (server lockups, high CPU, high RAM)

Post by jbader » Jun 25, 2018 8:23 pm

2018-06 cumulative updated includes refs.sys version 10.0.14393.2312. This appears to be working better but still a bit slower than before the 2018-05 cumulative update.

Mgamerz
Expert
Posts: 119
Liked: 19 times
Joined: Sep 29, 2017 8:07 pm
Contact:

Re: REFS issues (server lockups, high CPU, high RAM)

Post by Mgamerz » Jun 25, 2018 9:38 pm

Have been reading this thread for a while, was not hit with any issues until like May or June update. Synthetic full for our ~14TB primary file server is taking literally days now and is completely stalling our backups. Used to only take about 40 minutes. Installed June update after noticing it with may update, nothing fixed. Will be manually reverting refs using instructions in old post.

Gostev
SVP, Product Management
Posts: 24473
Liked: 3413 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: REFS issues (server lockups, high CPU, high RAM)

Post by Gostev » Jun 25, 2018 10:32 pm 1 person likes this post

I have confirmed that the fix we tested is coming in July "Patch Tuesday" update.

KFM
Service Provider
Posts: 13
Liked: 1 time
Joined: May 14, 2013 1:46 am
Full Name: KFM
Contact:

Re: REFS issues (server lockups, high CPU, high RAM)

Post by KFM » Jun 26, 2018 6:25 am

Gostev wrote:I have confirmed that the fix we tested is coming in July "Patch Tuesday" update.
I certainly hope so! We're on 10.0.14393.2097 and I still have problems with high CPU causing to lock the server up. I can isolate this to outside of Veeam by simply deleting a large number of large (VBK) files in Windows File Explorer. The repository is passing down the UNMAPs to the underlying storage array (DisableDeleteNotify=0). An hour (or so) after the deletes the CPU on the repository servers goes to 100% and hangs the host. Reset is the only way to recover from it.

I'm assuming this is also what people are seeing? Just want to make sure we're on the same page with this refs problem else I might have to open a support case directly with Microsoft.

gm2783
Service Provider
Posts: 6
Liked: 2 times
Joined: Apr 10, 2017 12:42 pm
Full Name: Giuseppe Marchese
Location: Mägenwil
Contact:

Re: REFS issues (server lockups, high CPU, high RAM)

Post by gm2783 » Jun 26, 2018 6:47 am

Hi all

We need to replace the driver definitely - The performance is really poor.

What's the best way to do it?
Just replace the file and reboot?
Or is a Linux Boot DVD needed?

Thanks a lot.

Regards
Giuseppe

SBarrett847
Service Provider
Posts: 304
Liked: 40 times
Joined: Feb 02, 2016 5:02 pm
Full Name: Stephen Barrett
Contact:

Re: REFS issues (server lockups, high CPU, high RAM)

Post by SBarrett847 » Jun 26, 2018 7:03 am

KFM wrote:I'm assuming this is also what people are seeing? Just want to make sure we're on the same page with this refs problem else I might have to open a support case directly with Microsoft.
I see this hard lockup if my Repo VM hasn't been assigned enough memory - increase memory and the issue doesn't occur for me.

WinstonWolf
Expert
Posts: 190
Liked: 5 times
Joined: Jan 06, 2011 8:33 am
Contact:

Re: REFS issues (server lockups, high CPU, high RAM)

Post by WinstonWolf » Jun 26, 2018 11:44 am

Hello ,

I have the extracted Refs.sys File Version 10.0.14393.2097 .

Whats the best way to change the refs.sys File on the Server with the extracted one ?

Thanks
Michael

gm2783
Service Provider
Posts: 6
Liked: 2 times
Joined: Apr 10, 2017 12:42 pm
Full Name: Giuseppe Marchese
Location: Mägenwil
Contact:

Re: REFS issues (server lockups, high CPU, high RAM)

Post by gm2783 » Jun 26, 2018 11:58 am 1 person likes this post

Hi all

see below the commands to export the REFS driver:

mkdir C:\%Path%
expand -f:* C:\%Path%\windows10.0-kb4093120-x64_72c7d6ce20eb42c0df760cd13a917bbc1e57c0b7.msu C:\%Path%\Expand
expand -f:*.cab C:\%Path%\Expand\*.cab c:\%Path%\expand\2
expand -f:refs.sys c:\%Path%\expand\2\*.cab c:\%Path%\expand\2\
expand -f:refsv1.sys c:\%Path%\expand\2\*.cab c:\%Path%\expand\2\

Microsoft®Update Catalog:
https://www.catalog.update.microsoft.co ... =KB4093120

Locked

Who is online

Users browsing this forum: Bing [Bot], Google [Bot] and 21 guests