Comprehensive data protection for all workloads
Locked
wingphil
Novice
Posts: 7
Liked: never
Joined: Jun 11, 2018 8:51 am
Contact:

Re: REFS issues (server lockups, high CPU, high RAM)

Post by wingphil »

Hi,

I also would like to know how to extract the refs.sys file from this patch. I've found the manifest for it at "Windows10.0-KB4093120-x64\Cab_1_for_KB4093120\amd64_microsoft-windows-refs_31bf3856ad364e35_10.0.14393.2097_none_1c46919c776b0217.manifest" - no idea how to get the actual file.

Thanks,

Phil
jslic
Novice
Posts: 3
Liked: 4 times
Joined: Jun 20, 2016 8:30 am
Full Name: Jesper Sorensen
Contact:

Re: REFS issues (server lockups, high CPU, high RAM)

Post by jslic » 1 person likes this post

Had the same issue with trying to extract the refs.sys - 10.0.14393.2097 from the update.
Ended up with installing a new VM, applying the WU patch (https://www.catalog.update.microsoft.co ... =KB4093120) and then extracting the file from C:\windows\system32\drivers
Gostev
Chief Product Officer
Posts: 31792
Liked: 7295 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: REFS issues (server lockups, high CPU, high RAM)

Post by Gostev »

This is what I would do too. Perhaps there's an easier way, but I don't know one.
mkaec
Veteran
Posts: 465
Liked: 136 times
Joined: Jul 16, 2015 1:31 pm
Full Name: Marc K
Contact:

Re: REFS issues (server lockups, high CPU, high RAM)

Post by mkaec »

I think the suggestion is to extract the ReFS.sys file from the patch and then manually replace it on the impacted system(s).
dellock6
VeeaMVP
Posts: 6165
Liked: 1971 times
Joined: Jul 26, 2009 3:39 pm
Full Name: Luca Dell'Oca
Location: Varese, Italy
Contact:

Re: REFS issues (server lockups, high CPU, high RAM)

Post by dellock6 »

Indeed, also because if you simply try to re-apply the patch, Windows Update will tell you that it's already applied.
Luca Dell'Oca
Principal EMEA Cloud Architect @ Veeam Software

@dellock6
https://www.virtualtothecore.com/
vExpert 2011 -> 2022
Veeam VMCE #1
Gostev
Chief Product Officer
Posts: 31792
Liked: 7295 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: REFS issues (server lockups, high CPU, high RAM)

Post by Gostev »

We meant applying patch on clean Windows install ;)
DonZoomik
Service Provider
Posts: 372
Liked: 120 times
Joined: Nov 25, 2016 1:56 pm
Full Name: Mihkel Soomere
Contact:

Re: REFS issues (server lockups, high CPU, high RAM)

Post by DonZoomik »

Todays patch includes new driver with version 10.0.14939.2312
wingphil
Novice
Posts: 7
Liked: never
Joined: Jun 11, 2018 8:51 am
Contact:

Re: REFS issues (server lockups, high CPU, high RAM)

Post by wingphil »

Interesting, will install today's patch and report back. Can't be much worse than my current version (10.0.14939.2273)
wingphil
Novice
Posts: 7
Liked: never
Joined: Jun 11, 2018 8:51 am
Contact:

Re: REFS issues (server lockups, high CPU, high RAM)

Post by wingphil »

Well anecdotally it's no faster, a full compact on one of my jobs took 9.5hrs with 2312, same as 2273. Memory usage might be a little better but that might be other factors.
nezzer
Influencer
Posts: 22
Liked: 5 times
Joined: Apr 27, 2018 11:40 am
Full Name: Andreas Svensson
Contact:

Re: REFS issues (server lockups, high CPU, high RAM)

Post by nezzer »

We removed the mentioned patch and we are now running at full speed again.
I'm not sure I like the workaround changing the REFS.SYS driver on a fully patched system.

I will wait and recommend others to wait for a working Windows patch from Microsoft (:
mvalpreda
Enthusiast
Posts: 77
Liked: 3 times
Joined: May 06, 2015 10:57 pm
Full Name: Mark Valpreda
Contact:

Re: REFS issues (server lockups, high CPU, high RAM)

Post by mvalpreda »

What is the version on refs.sys that Veeam got that worked? It obviously isn't in the June updates. I hate getting that far behind!
Gostev
Chief Product Officer
Posts: 31792
Liked: 7295 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: REFS issues (server lockups, high CPU, high RAM)

Post by Gostev »

Ours was VERY private :D we even had to disable driver signature check in the OS to use one! I suppose by now you should be able to get a better version by opening a support case with Microsoft. The internal bug ID to link the support case to is 17762340.
NightBird
Expert
Posts: 245
Liked: 58 times
Joined: Apr 28, 2009 8:33 am
Location: Strasbourg, FRANCE
Contact:

Re: REFS issues (server lockups, high CPU, high RAM)

Post by NightBird »

Another silent ReFS.sys driver update with June cumulative... (10.0.14393.2312) what a pain...
Nothing in the release notes... April, May, June => 3 silent Refs.sys driver update !!! (2248 2273 2312)
Gostev
Chief Product Officer
Posts: 31792
Liked: 7295 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: REFS issues (server lockups, high CPU, high RAM)

Post by Gostev »

Being on the Microsoft MVP distribution list I can tell you even they are mad about this by now :D particularly because many are Microsoft partners selling S2D/ReFS based solutions.
FrancWest
Veteran
Posts: 525
Liked: 104 times
Joined: Sep 17, 2017 3:20 am
Full Name: Franc
Contact:

Re: REFS issues (server lockups, high CPU, high RAM)

Post by FrancWest »

You can extract refs.sys from the patch file without first installing a clean windows 2016 and then installing the patch.

download the full update package from https://www.catalog.update.microsoft.co ... =KB4093120

open a command prompt and do:

expand -F:* <update>.msu <target_dir>

you end up with several .cab files

go to the target folder you extracted the .msu to and then do:

expand -f:refs.sys *.cab .

you end of with a folder called 'amd64_microsoft-windows-refs_31bf3856ad364e35_10.0.14393.2097_none_1c46919c776b0217' in the current folder.

Inside this folder is refs.sys with version 10.0.14393.2097

Franc.
paallergy
Enthusiast
Posts: 51
Liked: 10 times
Joined: May 18, 2010 7:10 pm
Full Name: Jonathan Schultz
Contact:

Re: REFS issues (server lockups, high CPU, high RAM)

Post by paallergy » 2 people like this post

@Franc: Thank you for the instructions

I had to do one more expand, after the msu and before refs:
expand -F:*.cab *.cab <target_dir_2>

So:
1. download and go to download directory
2. expand -F:* <update>.msu <target_dir_1>
3. cd <target_dir_1>
4. expand -F:*.cab *.cab <target_dir_2> // this results in several cab files
5. cd <target_dir_2>
6. expand -f:refs.sys *.cab . (or expand -f:refs.sys *.cab <target_dir_3>)
frankive
Service Provider
Posts: 1092
Liked: 134 times
Joined: May 14, 2013 8:35 pm
Full Name: Frank Iversen
Location: Norway
Contact:

[MERGED] Refs issue

Post by frankive »

Hi.
Anyone tested if the june CU for server 2016 fix the refs issue?
dellock6
VeeaMVP
Posts: 6165
Liked: 1971 times
Joined: Jul 26, 2009 3:39 pm
Full Name: Luca Dell'Oca
Location: Varese, Italy
Contact:

Re: Refs issue

Post by dellock6 »

Not personally, but you can have better feedback by using the general discussion about ReFS in the Veeam Backup & Replication public forum, here: https://forums.veeam.com/veeam-backup-r ... 40629.html
Luca Dell'Oca
Principal EMEA Cloud Architect @ Veeam Software

@dellock6
https://www.virtualtothecore.com/
vExpert 2011 -> 2022
Veeam VMCE #1
HenrikS.
Influencer
Posts: 21
Liked: 8 times
Joined: Jul 04, 2017 12:59 pm
Full Name: Henrik Schewe
Contact:

Re: REFS issues (server lockups, high CPU, high RAM)

Post by HenrikS. » 1 person likes this post

We just stumbled upon this after patching our backup and repository servers. Fast clone operations went from around 1,5hours to 60!hours.
After manually reverting back to the older ReFS.sys driver, we're back to normal operation again.
Looking forward to the fixed driver from MS...
gm2783
Service Provider
Posts: 6
Liked: 2 times
Joined: Apr 10, 2017 12:42 pm
Full Name: Giuseppe Marchese
Location: Mägenwil
Contact:

Re: REFS issues (server lockups, high CPU, high RAM)

Post by gm2783 »

Hi all.

Wich Version of the driver is the best choice at the moment?

Regards
Giuseppe Marchese
Gostev
Chief Product Officer
Posts: 31792
Liked: 7295 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: REFS issues (server lockups, high CPU, high RAM)

Post by Gostev »

Hi Giuseppe, see last post on the previous page.
jbader
Novice
Posts: 7
Liked: 1 time
Joined: May 22, 2018 4:08 pm
Full Name: John Bader
Contact:

Re: REFS issues (server lockups, high CPU, high RAM)

Post by jbader »

Same issue here. Anyone find a solution?
jbader
Novice
Posts: 7
Liked: 1 time
Joined: May 22, 2018 4:08 pm
Full Name: John Bader
Contact:

Re: REFS issues (server lockups, high CPU, high RAM)

Post by jbader »

2018-06 cumulative updated includes refs.sys version 10.0.14393.2312. This appears to be working better but still a bit slower than before the 2018-05 cumulative update.
Mgamerz
Expert
Posts: 160
Liked: 28 times
Joined: Sep 29, 2017 8:07 pm
Contact:

Re: REFS issues (server lockups, high CPU, high RAM)

Post by Mgamerz »

Have been reading this thread for a while, was not hit with any issues until like May or June update. Synthetic full for our ~14TB primary file server is taking literally days now and is completely stalling our backups. Used to only take about 40 minutes. Installed June update after noticing it with may update, nothing fixed. Will be manually reverting refs using instructions in old post.
Gostev
Chief Product Officer
Posts: 31792
Liked: 7295 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: REFS issues (server lockups, high CPU, high RAM)

Post by Gostev » 1 person likes this post

I have confirmed that the fix we tested is coming in July "Patch Tuesday" update.
KFM
Service Provider
Posts: 13
Liked: 2 times
Joined: May 14, 2013 1:46 am
Full Name: KFM
Contact:

Re: REFS issues (server lockups, high CPU, high RAM)

Post by KFM »

Gostev wrote:I have confirmed that the fix we tested is coming in July "Patch Tuesday" update.
I certainly hope so! We're on 10.0.14393.2097 and I still have problems with high CPU causing to lock the server up. I can isolate this to outside of Veeam by simply deleting a large number of large (VBK) files in Windows File Explorer. The repository is passing down the UNMAPs to the underlying storage array (DisableDeleteNotify=0). An hour (or so) after the deletes the CPU on the repository servers goes to 100% and hangs the host. Reset is the only way to recover from it.

I'm assuming this is also what people are seeing? Just want to make sure we're on the same page with this refs problem else I might have to open a support case directly with Microsoft.
gm2783
Service Provider
Posts: 6
Liked: 2 times
Joined: Apr 10, 2017 12:42 pm
Full Name: Giuseppe Marchese
Location: Mägenwil
Contact:

Re: REFS issues (server lockups, high CPU, high RAM)

Post by gm2783 »

Hi all

We need to replace the driver definitely - The performance is really poor.

What's the best way to do it?
Just replace the file and reboot?
Or is a Linux Boot DVD needed?

Thanks a lot.

Regards
Giuseppe
SBarrett847
Service Provider
Posts: 315
Liked: 41 times
Joined: Feb 02, 2016 5:02 pm
Full Name: Stephen Barrett
Contact:

Re: REFS issues (server lockups, high CPU, high RAM)

Post by SBarrett847 »

KFM wrote:I'm assuming this is also what people are seeing? Just want to make sure we're on the same page with this refs problem else I might have to open a support case directly with Microsoft.
I see this hard lockup if my Repo VM hasn't been assigned enough memory - increase memory and the issue doesn't occur for me.
WinstonWolf
Veteran
Posts: 284
Liked: 11 times
Joined: Jan 06, 2011 8:33 am
Contact:

Re: REFS issues (server lockups, high CPU, high RAM)

Post by WinstonWolf »

Hello ,

I have the extracted Refs.sys File Version 10.0.14393.2097 .

Whats the best way to change the refs.sys File on the Server with the extracted one ?

Thanks
Michael
gm2783
Service Provider
Posts: 6
Liked: 2 times
Joined: Apr 10, 2017 12:42 pm
Full Name: Giuseppe Marchese
Location: Mägenwil
Contact:

Re: REFS issues (server lockups, high CPU, high RAM)

Post by gm2783 » 1 person likes this post

Hi all

see below the commands to export the REFS driver:

mkdir C:\%Path%
expand -f:* C:\%Path%\windows10.0-kb4093120-x64_72c7d6ce20eb42c0df760cd13a917bbc1e57c0b7.msu C:\%Path%\Expand
expand -f:*.cab C:\%Path%\Expand\*.cab c:\%Path%\expand\2
expand -f:refs.sys c:\%Path%\expand\2\*.cab c:\%Path%\expand\2\
expand -f:refsv1.sys c:\%Path%\expand\2\*.cab c:\%Path%\expand\2\

Microsoft®Update Catalog:
https://www.catalog.update.microsoft.co ... =KB4093120
Locked

Who is online

Users browsing this forum: Bing [Bot], Google [Bot], Semrush [Bot] and 98 guests