REFS issues (server lockups, high CPU, high RAM)

[wingphil]

Hi,

I also would like to know how to extract the refs.sys file from this patch. I've found the manifest for it at "Windows10.0-KB4093120-x64\Cab_1_for_KB4093120\amd64_microsoft-windows-refs_31bf3856ad364e35_10.0.14393.2097_none_1c46919c776b0217.manifest" - no idea how to get the actual file.

Thanks,

Phil

[jslic]

Had the same issue with trying to extract the refs.sys - 10.0.14393.2097 from the update.
Ended up with installing a new VM, applying the WU patch (https://www.catalog.update.microsoft.co ... =KB4093120) and then extracting the file from C:\windows\system32\drivers

[Gostev]

This is what I would do too. Perhaps there's an easier way, but I don't know one.

[mkaec]

I think the suggestion is to extract the ReFS.sys file from the patch and then manually replace it on the impacted system(s).

[dellock6]

Indeed, also because if you simply try to re-apply the patch, Windows Update will tell you that it's already applied.

[Gostev]

We meant applying patch on clean Windows install

[DonZoomik]

Todays patch includes new driver with version 10.0.14939.2312

[wingphil]

Interesting, will install today's patch and report back. Can't be much worse than my current version (10.0.14939.2273)

[wingphil]

Well anecdotally it's no faster, a full compact on one of my jobs took 9.5hrs with 2312, same as 2273. Memory usage might be a little better but that might be other factors.

[nezzer]

We removed the mentioned patch and we are now running at full speed again.
I'm not sure I like the workaround changing the REFS.SYS driver on a fully patched system.

I will wait and recommend others to wait for a working Windows patch from Microsoft (:

[mvalpreda]

What is the version on refs.sys that Veeam got that worked? It obviously isn't in the June updates. I hate getting that far behind!

[Gostev]

Ours was VERY private we even had to disable driver signature check in the OS to use one! I suppose by now you should be able to get a better version by opening a support case with Microsoft. The internal bug ID to link the support case to is 17762340.

[NightBird]

Another silent ReFS.sys driver update with June cumulative... (10.0.14393.2312) what a pain...
Nothing in the release notes... April, May, June => 3 silent Refs.sys driver update !!! (2248 2273 2312)

[Gostev]

Being on the Microsoft MVP distribution list I can tell you even they are mad about this by now particularly because many are Microsoft partners selling S2D/ReFS based solutions.

[FrancWest]

You can extract refs.sys from the patch file without first installing a clean windows 2016 and then installing the patch.

download the full update package from https://www.catalog.update.microsoft.co ... =KB4093120

open a command prompt and do:

expand -F:* <update>.msu <target_dir>

you end up with several .cab files

go to the target folder you extracted the .msu to and then do:

expand -f:refs.sys *.cab .

you end of with a folder called 'amd64_microsoft-windows-refs_31bf3856ad364e35_10.0.14393.2097_none_1c46919c776b0217' in the current folder.

Inside this folder is refs.sys with version 10.0.14393.2097

Franc.

[paallergy]

@Franc: Thank you for the instructions

I had to do one more expand, after the msu and before refs:
expand -F:*.cab *.cab <target_dir_2>

So:
1. download and go to download directory
2. expand -F:* <update>.msu <target_dir_1>
3. cd <target_dir_1>
4. expand -F:*.cab *.cab <target_dir_2> // this results in several cab files
5. cd <target_dir_2>
6. expand -f:refs.sys *.cab . (or expand -f:refs.sys *.cab <target_dir_3>)

[frankive]

Hi.
Anyone tested if the june CU for server 2016 fix the refs issue?

[dellock6]

Not personally, but you can have better feedback by using the general discussion about ReFS in the Veeam Backup & Replication public forum, here: https://forums.veeam.com/veeam-backup-r ... 40629.html

[HenrikS.]

We just stumbled upon this after patching our backup and repository servers. Fast clone operations went from around 1,5hours to 60!hours.
After manually reverting back to the older ReFS.sys driver, we're back to normal operation again.
Looking forward to the fixed driver from MS...

[gm2783]

Hi all.

Wich Version of the driver is the best choice at the moment?

Regards
Giuseppe Marchese

[Gostev]

Hi Giuseppe, see last post on the previous page.

[jbader]

Same issue here. Anyone find a solution?

[jbader]

2018-06 cumulative updated includes refs.sys version 10.0.14393.2312. This appears to be working better but still a bit slower than before the 2018-05 cumulative update.

[Mgamerz]

Have been reading this thread for a while, was not hit with any issues until like May or June update. Synthetic full for our ~14TB primary file server is taking literally days now and is completely stalling our backups. Used to only take about 40 minutes. Installed June update after noticing it with may update, nothing fixed. Will be manually reverting refs using instructions in old post.

[Gostev]

I have confirmed that the fix we tested is coming in July "Patch Tuesday" update.

[KFM]

I have confirmed that the fix we tested is coming in July "Patch Tuesday" update.

I certainly hope so! We're on 10.0.14393.2097 and I still have problems with high CPU causing to lock the server up. I can isolate this to outside of Veeam by simply deleting a large number of large (VBK) files in Windows File Explorer. The repository is passing down the UNMAPs to the underlying storage array (DisableDeleteNotify=0). An hour (or so) after the deletes the CPU on the repository servers goes to 100% and hangs the host. Reset is the only way to recover from it.

I'm assuming this is also what people are seeing? Just want to make sure we're on the same page with this refs problem else I might have to open a support case directly with Microsoft.

[gm2783]

Hi all

We need to replace the driver definitely - The performance is really poor.

What's the best way to do it?
Just replace the file and reboot?
Or is a Linux Boot DVD needed?

Thanks a lot.

Regards
Giuseppe

[SBarrett847]

I'm assuming this is also what people are seeing? Just want to make sure we're on the same page with this refs problem else I might have to open a support case directly with Microsoft.

I see this hard lockup if my Repo VM hasn't been assigned enough memory - increase memory and the issue doesn't occur for me.

[WinstonWolf]

Hello ,

I have the extracted Refs.sys File Version 10.0.14393.2097 .

Whats the best way to change the refs.sys File on the Server with the extracted one ?

Thanks
Michael

[gm2783]

Hi all

see below the commands to export the REFS driver:

mkdir C:\%Path%
expand -f:* C:\%Path%\windows10.0-kb4093120-x64_72c7d6ce20eb42c0df760cd13a917bbc1e57c0b7.msu C:\%Path%\Expand
expand -f:*.cab C:\%Path%\Expand\*.cab c:\%Path%\expand\2
expand -f:refs.sys c:\%Path%\expand\2\*.cab c:\%Path%\expand\2\
expand -f:refsv1.sys c:\%Path%\expand\2\*.cab c:\%Path%\expand\2\

Microsoft®Update Catalog:
https://www.catalog.update.microsoft.co ... =KB4093120