Comprehensive data protection for all workloads
Locked
Iain_Green
Service Provider
Posts: 158
Liked: 9 times
Joined: Dec 05, 2014 2:13 pm
Full Name: Iain Green
Contact:

Re: REFS issues (server lockups, high CPU, high RAM)

Post by Iain_Green » 1 person likes this post

Any news on testing of the new Driver @gostev?

Still holding off patching, until I know the safest REFS driver to use?
Many thanks

Iain Green
Gostev
Chief Product Officer
Posts: 31428
Liked: 6633 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: REFS issues (server lockups, high CPU, high RAM)

Post by Gostev » 3 people like this post

Stress testing has just finished, everything looks golden with the ReFS driver version 2457 and beyond.
Looks like the new era of ReFS is finally here. I will write a more detailed report over the weekend.
JaySt
Service Provider
Posts: 414
Liked: 73 times
Joined: Jun 09, 2015 7:08 pm
Full Name: JaySt
Contact:

Re: REFS issues (server lockups, high CPU, high RAM)

Post by JaySt »

great news!
Veeam Certified Engineer
ejenner
Veteran
Posts: 636
Liked: 100 times
Joined: Mar 23, 2018 4:43 pm
Full Name: EJ
Location: London
Contact:

Re: REFS issues (server lockups, high CPU, high RAM)

Post by ejenner » 1 person likes this post

ejenner wrote: Oct 05, 2018 1:42 pm One of my repositories crashed on 3/10/18.

I still think it's worth pursuing the possibility of incompatibility with the System Centre agent.

So what I've done today is to remove CCM from the crashing repository and put it back onto the stable repository. Then I can see if the currently stable repository starts to crash and the crashing one stops crashing.
I have mostly proved the Configuration Manager Client causes STOP errors on 2016 servers with ReFS formatted volumes.

I'll be giving it a few more weeks for final confirmation. I installed CCM client to my stable repository server on Friday and it crashed that evening after being stable for months.

So it may not be the main ReFS issue that this thread started with but the particular combination of ReFS, Windows 2016 Server and CCM client does cause fairly regular blue-screening / STOP errors. I've removed CCM client from both of my 2016 ReFS repositories.

CCM client is very common in corporate environments, it's Microsoft's own system management agent.

The problem does not happen on other Windows 2016 Servers which aren't running ReFS formatted drives for Veeam repositories.

CCM client version: 5.00.8692.1007
ReFS.sys version/s: 10.0.14393.2363 and 10.0.14393.2457
Windows version: 10.0.14393 / Windows 2016 Server
Server hardware: HP ProLiant DL380 Gen9, 64GB RAM, 50/100TB of ReFS formatted direct attached disk
Iain_Green
Service Provider
Posts: 158
Liked: 9 times
Joined: Dec 05, 2014 2:13 pm
Full Name: Iain Green
Contact:

Re: REFS issues (server lockups, high CPU, high RAM)

Post by Iain_Green »

Gostev wrote: Oct 05, 2018 11:18 pm Stress testing has just finished, everything looks golden with the ReFS driver version 2457 and beyond.
Looks like the new era of ReFS is finally here. I will write a more detailed report over the weekend.
Having now read your brief on the testing, does this also now mean we are safe moving away from 1gb/1tb?
I have two 176TB repos with 64GB of RAM and would like to know if this setup is safe?

Is there a new standard you would advise to use?
Many thanks

Iain Green
ejenner
Veteran
Posts: 636
Liked: 100 times
Joined: Mar 23, 2018 4:43 pm
Full Name: EJ
Location: London
Contact:

Re: REFS issues (server lockups, high CPU, high RAM)

Post by ejenner » 1 person likes this post

I have 64GB with 100TB of disk and the 2457 (latest) ReFS driver. It works fine... if CCM client isn't on it. :wink:
burak
Influencer
Posts: 24
Liked: never
Joined: Sep 28, 2015 8:35 am
Full Name: Burak
Contact:

Re: REFS issues (server lockups, high CPU, high RAM)

Post by burak »

I am new to Veeam repositories running on ReFS. I will soon create a new ReFS repository with 16-20TB, because of the advantages with fast block cloning.

However, this thread has 83 pages and it seems that there is a lot of trouble with ReFS and it seems that some people still have problems with it.

Can someone sum up the current state and give me a recommendation, if I should make it as ReFS repository with 4k or 64k cluster size?
There is also this 1GB/1TB memory recommendation. So, the server which hosts the ReFS volume should have that amount of RAM or the server which runs Veeam software? Current plan is to create a Hyper-V host with 1x VM running Veeam B&R and on the host itself is the ReFS drive.
kubimike
Veteran
Posts: 373
Liked: 41 times
Joined: Feb 03, 2017 2:34 pm
Full Name: MikeO
Contact:

Re: REFS issues (server lockups, high CPU, high RAM)

Post by kubimike »

@burak, to sum this issue up from over a year yes .2515 is the one to run. I am backing up my machine with veeam agent then applying. I've never felt comfortable applying a patch until now. I've been running a test patch for almost a year waiting for a release like this.
kubimike
Veteran
Posts: 373
Liked: 41 times
Joined: Feb 03, 2017 2:34 pm
Full Name: MikeO
Contact:

Re: REFS issues (server lockups, high CPU, high RAM)

Post by kubimike »

Gostev wrote: Oct 05, 2018 11:18 pm Stress testing has just finished, everything looks golden with the ReFS driver version 2457 and beyond.
Looks like the new era of ReFS is finally here. I will write a more detailed report over the weekend.
Will all the registry keys we set over time be ignored by the new driver ?
Gostev
Chief Product Officer
Posts: 31428
Liked: 6633 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: REFS issues (server lockups, high CPU, high RAM)

Post by Gostev »

When I last asked about it half a year ago, they recommended to remove all registry values used in conjunction with earlier drivers.
Seve CH
Enthusiast
Posts: 67
Liked: 29 times
Joined: May 09, 2016 2:34 pm
Full Name: JM Severino
Location: Switzerland
Contact:

Re: REFS issues (server lockups, high CPU, high RAM)

Post by Seve CH »

ejenner wrote: Oct 08, 2018 9:08 am I have mostly proved the Configuration Manager Client causes STOP errors on 2016 servers with ReFS formatted volumes.

I'll be giving it a few more weeks for final confirmation. I installed CCM client to my stable repository server on Friday and it crashed that evening after being stable for months.

So it may not be the main ReFS issue that this thread started with but the particular combination of ReFS, Windows 2016 Server and CCM client does cause fairly regular blue-screening / STOP errors. I've removed CCM client from both of my 2016 ReFS repositories.
I have too started to get performance issues after reformating some disks in ReFS. Before, the server was working fine.

I've stopped SMS agent, removed the endpoint protection and now the server seems to work fine. It was using 95-100% CPU before ("system" process used 45%), but only 40GB out of 192GB RAM was used so it seems it is a different problem that the original ReFS one.

Server info:
W2016 1607 Build 14393.2551
Dell R730 with 2x Xeon E5-2643 v3 (12x3.40Ghz), 192 GB RAM.
Repos: 5x45TB NTFS, 1x122TB ReFS, 1x2TB ReFS.
SQL server and swap on SSD disks.
CCMExec.exe version: 5.00.8540.1000 (SCCM 2012)
Firmware for Veeam SAN and SAS arrays, HBAs, RAID, etc. + corresponding drivers are up to date.
kubimike
Veteran
Posts: 373
Liked: 41 times
Joined: Feb 03, 2017 2:34 pm
Full Name: MikeO
Contact:

Re: REFS issues (server lockups, high CPU, high RAM)

Post by kubimike »

you're on 2515 ?
Seve CH
Enthusiast
Posts: 67
Liked: 29 times
Joined: May 09, 2016 2:34 pm
Full Name: JM Severino
Location: Switzerland
Contact:

Re: REFS issues (server lockups, high CPU, high RAM)

Post by Seve CH »

Sorry, yes, that one. I've forgotten to provide the ReFS.sys version: 10.0.14393.2515
LBegnaud
Service Provider
Posts: 19
Liked: 7 times
Joined: Jan 24, 2018 12:08 am
Contact:

Re: REFS issues (server lockups, high CPU, high RAM)

Post by LBegnaud »

I think the CCM issue should be a separate thread, and this thread probably needs to be locked soon.
Iain_Green
Service Provider
Posts: 158
Liked: 9 times
Joined: Dec 05, 2014 2:13 pm
Full Name: Iain Green
Contact:

Re: REFS issues (server lockups, high CPU, high RAM)

Post by Iain_Green »

Which driver do I check for the driver version?

I have refsv1.sys which has a description of NT ReFS FS Driver showing version 10.0.1.14393.2457
I also have refs.sys which has a description of NT ReFS FS Driver showing version 10.0.1.14393.2515

Tad confused after patching?? :s
Many thanks

Iain Green
ejenner
Veteran
Posts: 636
Liked: 100 times
Joined: Mar 23, 2018 4:43 pm
Full Name: EJ
Location: London
Contact:

Re: REFS issues (server lockups, high CPU, high RAM)

Post by ejenner »

LBegnaud wrote: Oct 12, 2018 2:00 pmI think the CCM issue should be a separate thread, and this thread probably needs to be locked soon.

I'm glad someone else has been able to prove that there really is an issue with this and it's not just me imagining it.

Not sure if Veeam want to investigate that issue. I don't mind leaving the agent off the repository servers as a solution. It would be nice to know the reason though.
Gostev
Chief Product Officer
Posts: 31428
Liked: 6633 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: REFS issues (server lockups, high CPU, high RAM)

Post by Gostev »

LBegnaud wrote: Oct 12, 2018 2:00 pmI think the CCM issue should be a separate thread, and this thread probably needs to be locked soon.
Yes, I think it's time. Let me created the new placeholder thread and lock this one down.

@Iain it's refs.sys

@ejenner we can't do much from our end since since not having the source code we can't debug neither ReFS nor CCM, however if you open a support case with us we can then open a support case with Microsoft on this. It may require some of your time though, since they will want to troubleshoot it directly with you.
Gostev
Chief Product Officer
Posts: 31428
Liked: 6633 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: REFS issues (server lockups, high CPU, high RAM)

Post by Gostev »

Solution
Install September 2018 Windows Update (KB4343884) or later, since Windows Updates are cumulative.

If you have the latest updates installed but are still facing issues with ReFS, please refer to this topic > ReFS state post September 2018 Windows Updates

This thread is now locked.
Locked

Who is online

Users browsing this forum: Google [Bot] and 81 guests