REFS issues (server lockups, high CPU, high RAM)

[Iain_Green]

Any news on testing of the new Driver @gostev?

Still holding off patching, until I know the safest REFS driver to use?

[Gostev]

Stress testing has just finished, everything looks golden with the ReFS driver version 2457 and beyond.
Looks like the new era of ReFS is finally here. I will write a more detailed report over the weekend.

[JaySt]

great news!

[ejenner]

One of my repositories crashed on 3/10/18.

I still think it's worth pursuing the possibility of incompatibility with the System Centre agent.

So what I've done today is to remove CCM from the crashing repository and put it back onto the stable repository. Then I can see if the currently stable repository starts to crash and the crashing one stops crashing.

I have mostly proved the Configuration Manager Client causes STOP errors on 2016 servers with ReFS formatted volumes.

I'll be giving it a few more weeks for final confirmation. I installed CCM client to my stable repository server on Friday and it crashed that evening after being stable for months.

So it may not be the main ReFS issue that this thread started with but the particular combination of ReFS, Windows 2016 Server and CCM client does cause fairly regular blue-screening / STOP errors. I've removed CCM client from both of my 2016 ReFS repositories.

CCM client is very common in corporate environments, it's Microsoft's own system management agent.

The problem does not happen on other Windows 2016 Servers which aren't running ReFS formatted drives for Veeam repositories.

CCM client version: 5.00.8692.1007
ReFS.sys version/s: 10.0.14393.2363 and 10.0.14393.2457
Windows version: 10.0.14393 / Windows 2016 Server
Server hardware: HP ProLiant DL380 Gen9, 64GB RAM, 50/100TB of ReFS formatted direct attached disk

[Iain_Green]

Stress testing has just finished, everything looks golden with the ReFS driver version 2457 and beyond.
Looks like the new era of ReFS is finally here. I will write a more detailed report over the weekend.

Having now read your brief on the testing, does this also now mean we are safe moving away from 1gb/1tb?
I have two 176TB repos with 64GB of RAM and would like to know if this setup is safe?

Is there a new standard you would advise to use?

[ejenner]

I have 64GB with 100TB of disk and the 2457 (latest) ReFS driver. It works fine... if CCM client isn't on it.

[burak]

I am new to Veeam repositories running on ReFS. I will soon create a new ReFS repository with 16-20TB, because of the advantages with fast block cloning.

However, this thread has 83 pages and it seems that there is a lot of trouble with ReFS and it seems that some people still have problems with it.

Can someone sum up the current state and give me a recommendation, if I should make it as ReFS repository with 4k or 64k cluster size?
There is also this 1GB/1TB memory recommendation. So, the server which hosts the ReFS volume should have that amount of RAM or the server which runs Veeam software? Current plan is to create a Hyper-V host with 1x VM running Veeam B&R and on the host itself is the ReFS drive.

[kubimike]

@burak, to sum this issue up from over a year yes .2515 is the one to run. I am backing up my machine with veeam agent then applying. I've never felt comfortable applying a patch until now. I've been running a test patch for almost a year waiting for a release like this.

[kubimike]

Stress testing has just finished, everything looks golden with the ReFS driver version 2457 and beyond.
Looks like the new era of ReFS is finally here. I will write a more detailed report over the weekend.

Will all the registry keys we set over time be ignored by the new driver ?

[Gostev]

When I last asked about it half a year ago, they recommended to remove all registry values used in conjunction with earlier drivers.

[Seve CH]

I have mostly proved the Configuration Manager Client causes STOP errors on 2016 servers with ReFS formatted volumes.

I'll be giving it a few more weeks for final confirmation. I installed CCM client to my stable repository server on Friday and it crashed that evening after being stable for months.

So it may not be the main ReFS issue that this thread started with but the particular combination of ReFS, Windows 2016 Server and CCM client does cause fairly regular blue-screening / STOP errors. I've removed CCM client from both of my 2016 ReFS repositories.

I have too started to get performance issues after reformating some disks in ReFS. Before, the server was working fine.

I've stopped SMS agent, removed the endpoint protection and now the server seems to work fine. It was using 95-100% CPU before ("system" process used 45%), but only 40GB out of 192GB RAM was used so it seems it is a different problem that the original ReFS one.

Server info:
W2016 1607 Build 14393.2551
Dell R730 with 2x Xeon E5-2643 v3 (12x3.40Ghz), 192 GB RAM.
Repos: 5x45TB NTFS, 1x122TB ReFS, 1x2TB ReFS.
SQL server and swap on SSD disks.
CCMExec.exe version: 5.00.8540.1000 (SCCM 2012)
Firmware for Veeam SAN and SAS arrays, HBAs, RAID, etc. + corresponding drivers are up to date.

[kubimike]

you're on 2515 ?

[Seve CH]

Sorry, yes, that one. I've forgotten to provide the ReFS.sys version: 10.0.14393.2515

[LBegnaud]

I think the CCM issue should be a separate thread, and this thread probably needs to be locked soon.

[Iain_Green]

Which driver do I check for the driver version?

I have refsv1.sys which has a description of NT ReFS FS Driver showing version 10.0.1.14393.2457
I also have refs.sys which has a description of NT ReFS FS Driver showing version 10.0.1.14393.2515

Tad confused after patching?? :s

[ejenner]

I think the CCM issue should be a separate thread, and this thread probably needs to be locked soon.

I'm glad someone else has been able to prove that there really is an issue with this and it's not just me imagining it.

Not sure if Veeam want to investigate that issue. I don't mind leaving the agent off the repository servers as a solution. It would be nice to know the reason though.

[Gostev]

I think the CCM issue should be a separate thread, and this thread probably needs to be locked soon.

Yes, I think it's time. Let me created the new placeholder thread and lock this one down.

@Iain it's refs.sys

@ejenner we can't do much from our end since since not having the source code we can't debug neither ReFS nor CCM, however if you open a support case with us we can then open a support case with Microsoft on this. It may require some of your time though, since they will want to troubleshoot it directly with you.

[Gostev]

Solution
Install September 2018 Windows Update (KB4343884) or later, since Windows Updates are cumulative.

If you have the latest updates installed but are still facing issues with ReFS, please refer to this topic > ReFS state post September 2018 Windows Updates

This thread is now locked.