REFS issues (server lockups, high CPU, high RAM)

Availability for the Always-On Enterprise

[MERGED] New Server ReFS, i need an answer?

Veeam Logoby smid1584 » Sun Oct 08, 2017 9:39 am

Hello,

got a new backup server with 64 GB RAM, local disks and about 60 TB of space, windows server 2016.
I read a lot of bad things about ReFS, are these problems fixed now from MS and Veeam?
What are the recommended settings for ReFS? Format the partition as 64k ? What to do else?
smid1584
Lurker
 
Posts: 1
Liked: never
Joined: Sun Oct 08, 2017 9:30 am
Full Name: Smic dias

Re: REFS 4k horror story

Veeam Logoby Vitaliy S. » Sun Oct 08, 2017 12:55 pm

Hello Smic, yes, 64k seems like a recommended value. Please review the latest 2 pages for more info on the current state with ReFS issues.
Vitaliy S.
Veeam Software
 
Posts: 20106
Liked: 1160 times
Joined: Mon Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov

Re: REFS 4k horror story

Veeam Logoby DaveWatkins » Sun Oct 08, 2017 6:46 pm

mkretzer wrote:We had it on FC. Does anyone have the same performance issues on non-ISCSI and non-FC disks?

FC here as well, although we're running ok now with just a slower merge than I'd like compared to what it was to start with
DaveWatkins
Expert
 
Posts: 277
Liked: 71 times
Joined: Sun Dec 13, 2015 11:33 pm

Re: REFS 4k horror story

Veeam Logoby nhwanderer » Fri Oct 13, 2017 11:26 pm

I just looked up 1709, and problem fixed or not, it appears you need software assurance to gain access to it as a semi-annual build, so the back port is important to those of us without software assurance.

I'm just setting up and agonizing over my first VEEAM environment (moving from home grown backups). I'm going to give ReFS a shot and back off real real quick if it causes us trouble. I was kinda relying on it for efficient GFS retention.

nmdange wrote:The phrase "backported" says to me the fix is part of the next build (1709) which will be out in the next few weeks. I definitely plan on testing backups on the new build, but I haven't really had the same issues other ppl have had. Anyone who's been having problems willing to try out the new release?
nhwanderer
Lurker
 
Posts: 1
Liked: never
Joined: Fri Oct 13, 2017 7:37 pm
Full Name: Jordan Desroches

Re: REFS 4k horror story

Veeam Logoby Eamonn Deering » Sun Oct 29, 2017 12:34 pm

Hi tsightler
Just for clarification

HKLM\SYSTEM\CurrentControlSet\Control\FileSystem\RefsEnableLargeWorkingSetTrim = 1
HKLM\SYSTEM\CurrentControlSet\Control\FileSystem\RefsNumberOfChunksToTrim = 32
HKLM\SYSTEM\CurrentControlSet\Control\FileSystem\RefsDisableCachedPins = 1
HKLM\SYSTEM\CurrentControlSet\Control\FileSystem\RefsProcessedDeleteQueueEntryCountThreshold = 512

are these numbers 32 and 512 Hex or decimal ?

I had a ReFS repo lock up on me today and want to implement these keys.
I'm complementing this with a windows update to catch the necessary ReFS KB's and switching off syntec full on the backup jobs.

Thanks
Eamonn
Eamonn Deering
Service Provider
 
Posts: 26
Liked: 1 time
Joined: Wed Feb 29, 2012 1:42 pm
Location: Dublin, Ireland
Full Name: EamonnD

Re: REFS 4k horror story

Veeam Logoby tsightler » Sun Oct 29, 2017 3:33 pm 1 person likes this post

Good question, those are decimal values.
tsightler
Veeam Software
 
Posts: 4898
Liked: 1836 times
Joined: Fri Jun 05, 2009 12:57 pm
Full Name: Tom Sightler

Re: REFS 4k horror story

Veeam Logoby Cicadymn » Mon Oct 30, 2017 7:08 pm

After about a month of it working. My Veeam started up with merges and health checks for my file servers, causing it to lock out via CPU. After implementing the keys in this article: https://support.microsoft.com/en-us/hel ... erver-2016 It started working again for me.

ReFS is still really slow, but it's stable again, for now. Still waiting on the backported fix the ReFS team is supposed to be putting together.
Cicadymn
Influencer
 
Posts: 21
Liked: 5 times
Joined: Mon Jan 30, 2017 7:42 pm
Full Name: Sam

Re: REFS 4k horror story

Veeam Logoby JaySt » Tue Oct 31, 2017 10:03 am 10 people like this post

Is it perhaps possible for Veeam to create a KB article with the current status and best practices regarding the use of ReFS/Registry settings etc etc.
It's better than sending everyone off to a 43-page forum thread right?
Veeam Certified Engineer
JaySt
Service Provider
 
Posts: 70
Liked: 15 times
Joined: Tue Jun 09, 2015 7:08 pm
Full Name: JaySt

Re: REFS 4k horror story

Veeam Logoby jimmycartrette » Wed Nov 01, 2017 12:57 pm

Is there anyone else still having stability/hard crashes with ReFS (not just performance issues)?
I am, although lessened with synthetic fulls turned off and concurrent tasks cranked down.

I'm hopeful that the ReFS backport will fix my hard crashes. Otherwise I'll have to dump my entire primary and secondary chains because I don't have space to keep them while I start new ones.
jimmycartrette
Influencer
 
Posts: 14
Liked: 2 times
Joined: Thu Feb 02, 2017 2:13 pm
Full Name: JC

Re: REFS 4k horror story

Veeam Logoby kubimike » Wed Nov 01, 2017 7:26 pm 1 person likes this post

@jimmycartrette I went from being stable for a week to stable for about a month. It still crashes but works.
kubimike
Expert
 
Posts: 261
Liked: 25 times
Joined: Fri Feb 03, 2017 2:34 pm
Full Name: MikeO

Re: REFS 4k horror story

Veeam Logoby WimVD » Thu Nov 02, 2017 9:31 am

JaySt wrote:Is it perhaps possible for Veeam to create a KB article with the current status and best practices regarding the use of ReFS/Registry settings etc etc.
It's better than sending everyone off to a 43-page forum thread right?

This... Also, it would be nice if Gostev could reach out to the ReFS team for an update on the backport/patch.
Build 1709 is not a solution since it's a different branch with no in-place upgrade.
WimVD
Service Provider
 
Posts: 52
Liked: 11 times
Joined: Tue Dec 23, 2014 4:04 pm

Re: REFS 4k horror story

Veeam Logoby Gostev » Sun Nov 05, 2017 7:17 pm 2 people like this post

I will check with them.
Gostev
Veeam Software
 
Posts: 21648
Liked: 2433 times
Joined: Sun Jan 01, 2006 1:01 am
Location: Baar, Switzerland

Re: REFS 4k horror story

Veeam Logoby hennys » Thu Nov 09, 2017 8:27 am 1 person likes this post

Any update yet ?
hennys
Lurker
 
Posts: 2
Liked: 1 time
Joined: Fri Nov 02, 2012 2:14 pm
Full Name: Hen Savelkoul

Re: REFS 4k horror story

Veeam Logoby trenzif » Wed Nov 15, 2017 4:34 pm

Any news on this?
I also don't see a ReFS remark on the updates provided by microsoft november patch tuesday.

Can somebody confirm that the problems are gone with Windows Server 1709?
trenzif
Lurker
 
Posts: 2
Liked: never
Joined: Wed Nov 15, 2017 4:31 pm

Re: REFS 4k horror story

Veeam Logoby WimVD » Thu Nov 16, 2017 10:18 am

trenzif wrote:Can somebody confirm that the problems are gone with Windows Server 1709?

Would like to test but since this needs a OS reinstall I would like some feedback from the ReFS team if the fix was backported in 1709.
Anton, any news? Our merges are taking 10+ hours and getting desperate for a solution...
WimVD
Service Provider
 
Posts: 52
Liked: 11 times
Joined: Tue Dec 23, 2014 4:04 pm

PreviousNext

Return to Veeam Backup & Replication



Who is online

Users browsing this forum: Bing [Bot], Yahoo [Bot] and 1 guest