Comprehensive data protection for all workloads
Locked
Iain_Green
Service Provider
Posts: 148
Liked: 8 times
Joined: Dec 05, 2014 2:13 pm
Full Name: Iain Green
Contact:

Re: REFS issues (server lockups, high CPU, high RAM)

Post by Iain_Green » Oct 05, 2018 1:45 pm 1 person likes this post

Any news on testing of the new Driver @gostev?

Still holding off patching, until I know the safest REFS driver to use?
Many thanks

Iain Green

Gostev
SVP, Product Management
Posts: 24431
Liked: 3405 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: REFS issues (server lockups, high CPU, high RAM)

Post by Gostev » Oct 05, 2018 11:18 pm 3 people like this post

Stress testing has just finished, everything looks golden with the ReFS driver version 2457 and beyond.
Looks like the new era of ReFS is finally here. I will write a more detailed report over the weekend.

JaySt
Service Provider
Posts: 154
Liked: 24 times
Joined: Jun 09, 2015 7:08 pm
Full Name: JaySt
Contact:

Re: REFS issues (server lockups, high CPU, high RAM)

Post by JaySt » Oct 07, 2018 7:56 pm

great news!
Veeam Certified Engineer

ejenner
Expert
Posts: 331
Liked: 49 times
Joined: Mar 23, 2018 4:43 pm
Full Name: EJ
Location: London
Contact:

Re: REFS issues (server lockups, high CPU, high RAM)

Post by ejenner » Oct 08, 2018 9:08 am 1 person likes this post

ejenner wrote:
Oct 05, 2018 1:42 pm
One of my repositories crashed on 3/10/18.

I still think it's worth pursuing the possibility of incompatibility with the System Centre agent.

So what I've done today is to remove CCM from the crashing repository and put it back onto the stable repository. Then I can see if the currently stable repository starts to crash and the crashing one stops crashing.
I have mostly proved the Configuration Manager Client causes STOP errors on 2016 servers with ReFS formatted volumes.

I'll be giving it a few more weeks for final confirmation. I installed CCM client to my stable repository server on Friday and it crashed that evening after being stable for months.

So it may not be the main ReFS issue that this thread started with but the particular combination of ReFS, Windows 2016 Server and CCM client does cause fairly regular blue-screening / STOP errors. I've removed CCM client from both of my 2016 ReFS repositories.

CCM client is very common in corporate environments, it's Microsoft's own system management agent.

The problem does not happen on other Windows 2016 Servers which aren't running ReFS formatted drives for Veeam repositories.

CCM client version: 5.00.8692.1007
ReFS.sys version/s: 10.0.14393.2363 and 10.0.14393.2457
Windows version: 10.0.14393 / Windows 2016 Server
Server hardware: HP ProLiant DL380 Gen9, 64GB RAM, 50/100TB of ReFS formatted direct attached disk

Iain_Green
Service Provider
Posts: 148
Liked: 8 times
Joined: Dec 05, 2014 2:13 pm
Full Name: Iain Green
Contact:

Re: REFS issues (server lockups, high CPU, high RAM)

Post by Iain_Green » Oct 08, 2018 9:30 am

Gostev wrote:
Oct 05, 2018 11:18 pm
Stress testing has just finished, everything looks golden with the ReFS driver version 2457 and beyond.
Looks like the new era of ReFS is finally here. I will write a more detailed report over the weekend.
Having now read your brief on the testing, does this also now mean we are safe moving away from 1gb/1tb?
I have two 176TB repos with 64GB of RAM and would like to know if this setup is safe?

Is there a new standard you would advise to use?
Many thanks

Iain Green

ejenner
Expert
Posts: 331
Liked: 49 times
Joined: Mar 23, 2018 4:43 pm
Full Name: EJ
Location: London
Contact:

Re: REFS issues (server lockups, high CPU, high RAM)

Post by ejenner » Oct 08, 2018 9:47 am 1 person likes this post

I have 64GB with 100TB of disk and the 2457 (latest) ReFS driver. It works fine... if CCM client isn't on it. :wink:

burak
Influencer
Posts: 22
Liked: never
Joined: Sep 28, 2015 8:35 am
Full Name: Burak
Contact:

Re: REFS issues (server lockups, high CPU, high RAM)

Post by burak » Oct 08, 2018 9:55 am

I am new to Veeam repositories running on ReFS. I will soon create a new ReFS repository with 16-20TB, because of the advantages with fast block cloning.

However, this thread has 83 pages and it seems that there is a lot of trouble with ReFS and it seems that some people still have problems with it.

Can someone sum up the current state and give me a recommendation, if I should make it as ReFS repository with 4k or 64k cluster size?
There is also this 1GB/1TB memory recommendation. So, the server which hosts the ReFS volume should have that amount of RAM or the server which runs Veeam software? Current plan is to create a Hyper-V host with 1x VM running Veeam B&R and on the host itself is the ReFS drive.

kubimike
Expert
Posts: 334
Liked: 40 times
Joined: Feb 03, 2017 2:34 pm
Full Name: MikeO
Contact:

Re: REFS issues (server lockups, high CPU, high RAM)

Post by kubimike » Oct 08, 2018 1:26 pm

@burak, to sum this issue up from over a year yes .2515 is the one to run. I am backing up my machine with veeam agent then applying. I've never felt comfortable applying a patch until now. I've been running a test patch for almost a year waiting for a release like this.

kubimike
Expert
Posts: 334
Liked: 40 times
Joined: Feb 03, 2017 2:34 pm
Full Name: MikeO
Contact:

Re: REFS issues (server lockups, high CPU, high RAM)

Post by kubimike » Oct 08, 2018 1:27 pm

Gostev wrote:
Oct 05, 2018 11:18 pm
Stress testing has just finished, everything looks golden with the ReFS driver version 2457 and beyond.
Looks like the new era of ReFS is finally here. I will write a more detailed report over the weekend.
Will all the registry keys we set over time be ignored by the new driver ?

Gostev
SVP, Product Management
Posts: 24431
Liked: 3405 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: REFS issues (server lockups, high CPU, high RAM)

Post by Gostev » Oct 09, 2018 6:20 am

When I last asked about it half a year ago, they recommended to remove all registry values used in conjunction with earlier drivers.

Seve CH
Influencer
Posts: 18
Liked: 9 times
Joined: May 09, 2016 2:34 pm
Full Name: JM Severino
Location: Switzerland
Contact:

Re: REFS issues (server lockups, high CPU, high RAM)

Post by Seve CH » Oct 12, 2018 9:11 am

ejenner wrote:
Oct 08, 2018 9:08 am
I have mostly proved the Configuration Manager Client causes STOP errors on 2016 servers with ReFS formatted volumes.

I'll be giving it a few more weeks for final confirmation. I installed CCM client to my stable repository server on Friday and it crashed that evening after being stable for months.

So it may not be the main ReFS issue that this thread started with but the particular combination of ReFS, Windows 2016 Server and CCM client does cause fairly regular blue-screening / STOP errors. I've removed CCM client from both of my 2016 ReFS repositories.
I have too started to get performance issues after reformating some disks in ReFS. Before, the server was working fine.

I've stopped SMS agent, removed the endpoint protection and now the server seems to work fine. It was using 95-100% CPU before ("system" process used 45%), but only 40GB out of 192GB RAM was used so it seems it is a different problem that the original ReFS one.

Server info:
W2016 1607 Build 14393.2551
Dell R730 with 2x Xeon E5-2643 v3 (12x3.40Ghz), 192 GB RAM.
Repos: 5x45TB NTFS, 1x122TB ReFS, 1x2TB ReFS.
SQL server and swap on SSD disks.
CCMExec.exe version: 5.00.8540.1000 (SCCM 2012)
Firmware for Veeam SAN and SAS arrays, HBAs, RAID, etc. + corresponding drivers are up to date.

kubimike
Expert
Posts: 334
Liked: 40 times
Joined: Feb 03, 2017 2:34 pm
Full Name: MikeO
Contact:

Re: REFS issues (server lockups, high CPU, high RAM)

Post by kubimike » Oct 12, 2018 1:24 pm

you're on 2515 ?

Seve CH
Influencer
Posts: 18
Liked: 9 times
Joined: May 09, 2016 2:34 pm
Full Name: JM Severino
Location: Switzerland
Contact:

Re: REFS issues (server lockups, high CPU, high RAM)

Post by Seve CH » Oct 12, 2018 1:54 pm

Sorry, yes, that one. I've forgotten to provide the ReFS.sys version: 10.0.14393.2515

LBegnaud
Service Provider
Posts: 19
Liked: 7 times
Joined: Jan 24, 2018 12:08 am
Contact:

Re: REFS issues (server lockups, high CPU, high RAM)

Post by LBegnaud » Oct 12, 2018 2:00 pm

I think the CCM issue should be a separate thread, and this thread probably needs to be locked soon.

Iain_Green
Service Provider
Posts: 148
Liked: 8 times
Joined: Dec 05, 2014 2:13 pm
Full Name: Iain Green
Contact:

Re: REFS issues (server lockups, high CPU, high RAM)

Post by Iain_Green » Oct 15, 2018 8:57 am

Which driver do I check for the driver version?

I have refsv1.sys which has a description of NT ReFS FS Driver showing version 10.0.1.14393.2457
I also have refs.sys which has a description of NT ReFS FS Driver showing version 10.0.1.14393.2515

Tad confused after patching?? :s
Many thanks

Iain Green

ejenner
Expert
Posts: 331
Liked: 49 times
Joined: Mar 23, 2018 4:43 pm
Full Name: EJ
Location: London
Contact:

Re: REFS issues (server lockups, high CPU, high RAM)

Post by ejenner » Oct 15, 2018 1:49 pm

LBegnaud wrote:
Oct 12, 2018 2:00 pm
I think the CCM issue should be a separate thread, and this thread probably needs to be locked soon.

I'm glad someone else has been able to prove that there really is an issue with this and it's not just me imagining it.

Not sure if Veeam want to investigate that issue. I don't mind leaving the agent off the repository servers as a solution. It would be nice to know the reason though.

Gostev
SVP, Product Management
Posts: 24431
Liked: 3405 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: REFS issues (server lockups, high CPU, high RAM)

Post by Gostev » Oct 15, 2018 1:53 pm

LBegnaud wrote:
Oct 12, 2018 2:00 pm
I think the CCM issue should be a separate thread, and this thread probably needs to be locked soon.
Yes, I think it's time. Let me created the new placeholder thread and lock this one down.

@Iain it's refs.sys

@ejenner we can't do much from our end since since not having the source code we can't debug neither ReFS nor CCM, however if you open a support case with us we can then open a support case with Microsoft on this. It may require some of your time though, since they will want to troubleshoot it directly with you.

Gostev
SVP, Product Management
Posts: 24431
Liked: 3405 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: REFS issues (server lockups, high CPU, high RAM)

Post by Gostev » Oct 15, 2018 2:42 pm

Solution
Install September 2018 Windows Update (KB4343884) or later, since Windows Updates are cumulative.

If you have the latest updates installed but are still facing issues with ReFS, please refer to this topic > ReFS state post September 2018 Windows Updates

This thread is now locked.

Locked

Who is online

Users browsing this forum: Bing [Bot], Majestic-12 [Bot] and 32 guests