-
- Service Provider
- Posts: 158
- Liked: 9 times
- Joined: Dec 05, 2014 2:13 pm
- Full Name: Iain Green
- Contact:
Re: REFS issues (server lockups, high CPU, high RAM)
Any news on testing of the new Driver @gostev?
Still holding off patching, until I know the safest REFS driver to use?
Still holding off patching, until I know the safest REFS driver to use?
Many thanks
Iain Green
Iain Green
-
- Chief Product Officer
- Posts: 31807
- Liked: 7300 times
- Joined: Jan 01, 2006 1:01 am
- Location: Baar, Switzerland
- Contact:
Re: REFS issues (server lockups, high CPU, high RAM)
Stress testing has just finished, everything looks golden with the ReFS driver version 2457 and beyond.
Looks like the new era of ReFS is finally here. I will write a more detailed report over the weekend.
Looks like the new era of ReFS is finally here. I will write a more detailed report over the weekend.
-
- Service Provider
- Posts: 454
- Liked: 86 times
- Joined: Jun 09, 2015 7:08 pm
- Full Name: JaySt
- Contact:
-
- Veteran
- Posts: 636
- Liked: 100 times
- Joined: Mar 23, 2018 4:43 pm
- Full Name: EJ
- Location: London
- Contact:
Re: REFS issues (server lockups, high CPU, high RAM)
I have mostly proved the Configuration Manager Client causes STOP errors on 2016 servers with ReFS formatted volumes.ejenner wrote: ↑Oct 05, 2018 1:42 pm One of my repositories crashed on 3/10/18.
I still think it's worth pursuing the possibility of incompatibility with the System Centre agent.
So what I've done today is to remove CCM from the crashing repository and put it back onto the stable repository. Then I can see if the currently stable repository starts to crash and the crashing one stops crashing.
I'll be giving it a few more weeks for final confirmation. I installed CCM client to my stable repository server on Friday and it crashed that evening after being stable for months.
So it may not be the main ReFS issue that this thread started with but the particular combination of ReFS, Windows 2016 Server and CCM client does cause fairly regular blue-screening / STOP errors. I've removed CCM client from both of my 2016 ReFS repositories.
CCM client is very common in corporate environments, it's Microsoft's own system management agent.
The problem does not happen on other Windows 2016 Servers which aren't running ReFS formatted drives for Veeam repositories.
CCM client version: 5.00.8692.1007
ReFS.sys version/s: 10.0.14393.2363 and 10.0.14393.2457
Windows version: 10.0.14393 / Windows 2016 Server
Server hardware: HP ProLiant DL380 Gen9, 64GB RAM, 50/100TB of ReFS formatted direct attached disk
-
- Service Provider
- Posts: 158
- Liked: 9 times
- Joined: Dec 05, 2014 2:13 pm
- Full Name: Iain Green
- Contact:
Re: REFS issues (server lockups, high CPU, high RAM)
Having now read your brief on the testing, does this also now mean we are safe moving away from 1gb/1tb?
I have two 176TB repos with 64GB of RAM and would like to know if this setup is safe?
Is there a new standard you would advise to use?
Many thanks
Iain Green
Iain Green
-
- Veteran
- Posts: 636
- Liked: 100 times
- Joined: Mar 23, 2018 4:43 pm
- Full Name: EJ
- Location: London
- Contact:
Re: REFS issues (server lockups, high CPU, high RAM)
I have 64GB with 100TB of disk and the 2457 (latest) ReFS driver. It works fine... if CCM client isn't on it.
-
- Influencer
- Posts: 24
- Liked: never
- Joined: Sep 28, 2015 8:35 am
- Full Name: Burak
- Contact:
Re: REFS issues (server lockups, high CPU, high RAM)
I am new to Veeam repositories running on ReFS. I will soon create a new ReFS repository with 16-20TB, because of the advantages with fast block cloning.
However, this thread has 83 pages and it seems that there is a lot of trouble with ReFS and it seems that some people still have problems with it.
Can someone sum up the current state and give me a recommendation, if I should make it as ReFS repository with 4k or 64k cluster size?
There is also this 1GB/1TB memory recommendation. So, the server which hosts the ReFS volume should have that amount of RAM or the server which runs Veeam software? Current plan is to create a Hyper-V host with 1x VM running Veeam B&R and on the host itself is the ReFS drive.
However, this thread has 83 pages and it seems that there is a lot of trouble with ReFS and it seems that some people still have problems with it.
Can someone sum up the current state and give me a recommendation, if I should make it as ReFS repository with 4k or 64k cluster size?
There is also this 1GB/1TB memory recommendation. So, the server which hosts the ReFS volume should have that amount of RAM or the server which runs Veeam software? Current plan is to create a Hyper-V host with 1x VM running Veeam B&R and on the host itself is the ReFS drive.
-
- Veteran
- Posts: 391
- Liked: 56 times
- Joined: Feb 03, 2017 2:34 pm
- Full Name: MikeO
- Contact:
Re: REFS issues (server lockups, high CPU, high RAM)
@burak, to sum this issue up from over a year yes .2515 is the one to run. I am backing up my machine with veeam agent then applying. I've never felt comfortable applying a patch until now. I've been running a test patch for almost a year waiting for a release like this.
-
- Veteran
- Posts: 391
- Liked: 56 times
- Joined: Feb 03, 2017 2:34 pm
- Full Name: MikeO
- Contact:
Re: REFS issues (server lockups, high CPU, high RAM)
Will all the registry keys we set over time be ignored by the new driver ?
-
- Chief Product Officer
- Posts: 31807
- Liked: 7300 times
- Joined: Jan 01, 2006 1:01 am
- Location: Baar, Switzerland
- Contact:
Re: REFS issues (server lockups, high CPU, high RAM)
When I last asked about it half a year ago, they recommended to remove all registry values used in conjunction with earlier drivers.
-
- Enthusiast
- Posts: 89
- Liked: 35 times
- Joined: May 09, 2016 2:34 pm
- Full Name: JM Severino
- Location: Switzerland
- Contact:
Re: REFS issues (server lockups, high CPU, high RAM)
I have too started to get performance issues after reformating some disks in ReFS. Before, the server was working fine.ejenner wrote: ↑Oct 08, 2018 9:08 am I have mostly proved the Configuration Manager Client causes STOP errors on 2016 servers with ReFS formatted volumes.
I'll be giving it a few more weeks for final confirmation. I installed CCM client to my stable repository server on Friday and it crashed that evening after being stable for months.
So it may not be the main ReFS issue that this thread started with but the particular combination of ReFS, Windows 2016 Server and CCM client does cause fairly regular blue-screening / STOP errors. I've removed CCM client from both of my 2016 ReFS repositories.
I've stopped SMS agent, removed the endpoint protection and now the server seems to work fine. It was using 95-100% CPU before ("system" process used 45%), but only 40GB out of 192GB RAM was used so it seems it is a different problem that the original ReFS one.
Server info:
W2016 1607 Build 14393.2551
Dell R730 with 2x Xeon E5-2643 v3 (12x3.40Ghz), 192 GB RAM.
Repos: 5x45TB NTFS, 1x122TB ReFS, 1x2TB ReFS.
SQL server and swap on SSD disks.
CCMExec.exe version: 5.00.8540.1000 (SCCM 2012)
Firmware for Veeam SAN and SAS arrays, HBAs, RAID, etc. + corresponding drivers are up to date.
-
- Veteran
- Posts: 391
- Liked: 56 times
- Joined: Feb 03, 2017 2:34 pm
- Full Name: MikeO
- Contact:
Re: REFS issues (server lockups, high CPU, high RAM)
you're on 2515 ?
-
- Enthusiast
- Posts: 89
- Liked: 35 times
- Joined: May 09, 2016 2:34 pm
- Full Name: JM Severino
- Location: Switzerland
- Contact:
Re: REFS issues (server lockups, high CPU, high RAM)
Sorry, yes, that one. I've forgotten to provide the ReFS.sys version: 10.0.14393.2515
-
- Service Provider
- Posts: 19
- Liked: 7 times
- Joined: Jan 24, 2018 12:08 am
- Contact:
Re: REFS issues (server lockups, high CPU, high RAM)
I think the CCM issue should be a separate thread, and this thread probably needs to be locked soon.
-
- Service Provider
- Posts: 158
- Liked: 9 times
- Joined: Dec 05, 2014 2:13 pm
- Full Name: Iain Green
- Contact:
Re: REFS issues (server lockups, high CPU, high RAM)
Which driver do I check for the driver version?
I have refsv1.sys which has a description of NT ReFS FS Driver showing version 10.0.1.14393.2457
I also have refs.sys which has a description of NT ReFS FS Driver showing version 10.0.1.14393.2515
Tad confused after patching?? :s
I have refsv1.sys which has a description of NT ReFS FS Driver showing version 10.0.1.14393.2457
I also have refs.sys which has a description of NT ReFS FS Driver showing version 10.0.1.14393.2515
Tad confused after patching?? :s
Many thanks
Iain Green
Iain Green
-
- Veteran
- Posts: 636
- Liked: 100 times
- Joined: Mar 23, 2018 4:43 pm
- Full Name: EJ
- Location: London
- Contact:
Re: REFS issues (server lockups, high CPU, high RAM)
I'm glad someone else has been able to prove that there really is an issue with this and it's not just me imagining it.
Not sure if Veeam want to investigate that issue. I don't mind leaving the agent off the repository servers as a solution. It would be nice to know the reason though.
-
- Chief Product Officer
- Posts: 31807
- Liked: 7300 times
- Joined: Jan 01, 2006 1:01 am
- Location: Baar, Switzerland
- Contact:
Re: REFS issues (server lockups, high CPU, high RAM)
Yes, I think it's time. Let me created the new placeholder thread and lock this one down.
@Iain it's refs.sys
@ejenner we can't do much from our end since since not having the source code we can't debug neither ReFS nor CCM, however if you open a support case with us we can then open a support case with Microsoft on this. It may require some of your time though, since they will want to troubleshoot it directly with you.
-
- Chief Product Officer
- Posts: 31807
- Liked: 7300 times
- Joined: Jan 01, 2006 1:01 am
- Location: Baar, Switzerland
- Contact:
Re: REFS issues (server lockups, high CPU, high RAM)
Solution
Install September 2018 Windows Update (KB4343884) or later, since Windows Updates are cumulative.
If you have the latest updates installed but are still facing issues with ReFS, please refer to this topic > ReFS state post September 2018 Windows Updates
This thread is now locked.
Install September 2018 Windows Update (KB4343884) or later, since Windows Updates are cumulative.
If you have the latest updates installed but are still facing issues with ReFS, please refer to this topic > ReFS state post September 2018 Windows Updates
This thread is now locked.
Who is online
Users browsing this forum: Amazon [Bot], Google [Bot], Majestic-12 [Bot], sivein, Steve-nIP and 271 guests