Comprehensive data protection for all workloads
Locked
smid1584
Lurker
Posts: 1
Liked: never
Joined: Oct 08, 2017 9:30 am
Full Name: Smic dias
Contact:

[MERGED] New Server ReFS, i need an answer?

Post by smid1584 »

Hello,

got a new backup server with 64 GB RAM, local disks and about 60 TB of space, windows server 2016.
I read a lot of bad things about ReFS, are these problems fixed now from MS and Veeam?
What are the recommended settings for ReFS? Format the partition as 64k ? What to do else?
Vitaliy S.
VP, Product Management
Posts: 27055
Liked: 2710 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: REFS 4k horror story

Post by Vitaliy S. »

Hello Smic, yes, 64k seems like a recommended value. Please review the latest 2 pages for more info on the current state with ReFS issues.
DaveWatkins
Veteran
Posts: 370
Liked: 97 times
Joined: Dec 13, 2015 11:33 pm
Contact:

Re: REFS 4k horror story

Post by DaveWatkins »

mkretzer wrote:We had it on FC. Does anyone have the same performance issues on non-ISCSI and non-FC disks?
FC here as well, although we're running ok now with just a slower merge than I'd like compared to what it was to start with
nhwanderer
Novice
Posts: 8
Liked: 1 time
Joined: Oct 13, 2017 7:37 pm
Full Name: Jordan Desroches
Contact:

Re: REFS 4k horror story

Post by nhwanderer »

I just looked up 1709, and problem fixed or not, it appears you need software assurance to gain access to it as a semi-annual build, so the back port is important to those of us without software assurance.

I'm just setting up and agonizing over my first VEEAM environment (moving from home grown backups). I'm going to give ReFS a shot and back off real real quick if it causes us trouble. I was kinda relying on it for efficient GFS retention.
nmdange wrote:The phrase "backported" says to me the fix is part of the next build (1709) which will be out in the next few weeks. I definitely plan on testing backups on the new build, but I haven't really had the same issues other ppl have had. Anyone who's been having problems willing to try out the new release?
Eamonn Deering
Service Provider
Posts: 33
Liked: 4 times
Joined: Feb 29, 2012 1:42 pm
Full Name: EamonnD
Location: Dublin, Ireland
Contact:

Re: REFS 4k horror story

Post by Eamonn Deering »

Hi tsightler
Just for clarification

HKLM\SYSTEM\CurrentControlSet\Control\FileSystem\RefsEnableLargeWorkingSetTrim = 1
HKLM\SYSTEM\CurrentControlSet\Control\FileSystem\RefsNumberOfChunksToTrim = 32
HKLM\SYSTEM\CurrentControlSet\Control\FileSystem\RefsDisableCachedPins = 1
HKLM\SYSTEM\CurrentControlSet\Control\FileSystem\RefsProcessedDeleteQueueEntryCountThreshold = 512

are these numbers 32 and 512 Hex or decimal ?

I had a ReFS repo lock up on me today and want to implement these keys.
I'm complementing this with a windows update to catch the necessary ReFS KB's and switching off syntec full on the backup jobs.

Thanks
Eamonn
tsightler
VP, Product Management
Posts: 6009
Liked: 2843 times
Joined: Jun 05, 2009 12:57 pm
Full Name: Tom Sightler
Contact:

Re: REFS 4k horror story

Post by tsightler » 1 person likes this post

Good question, those are decimal values.
Cicadymn
Enthusiast
Posts: 26
Liked: 12 times
Joined: Jan 30, 2017 7:42 pm
Full Name: Sam
Contact:

Re: REFS 4k horror story

Post by Cicadymn »

After about a month of it working. My Veeam started up with merges and health checks for my file servers, causing it to lock out via CPU. After implementing the keys in this article: https://support.microsoft.com/en-us/hel ... erver-2016 It started working again for me.

ReFS is still really slow, but it's stable again, for now. Still waiting on the backported fix the ReFS team is supposed to be putting together.
JaySt
Service Provider
Posts: 415
Liked: 75 times
Joined: Jun 09, 2015 7:08 pm
Full Name: JaySt
Contact:

Re: REFS 4k horror story

Post by JaySt » 11 people like this post

Is it perhaps possible for Veeam to create a KB article with the current status and best practices regarding the use of ReFS/Registry settings etc etc.
It's better than sending everyone off to a 43-page forum thread right?
Veeam Certified Engineer
jimmycartrette
Influencer
Posts: 14
Liked: 2 times
Joined: Feb 02, 2017 2:13 pm
Full Name: JC
Contact:

Re: REFS 4k horror story

Post by jimmycartrette »

Is there anyone else still having stability/hard crashes with ReFS (not just performance issues)?
I am, although lessened with synthetic fulls turned off and concurrent tasks cranked down.

I'm hopeful that the ReFS backport will fix my hard crashes. Otherwise I'll have to dump my entire primary and secondary chains because I don't have space to keep them while I start new ones.
kubimike
Veteran
Posts: 373
Liked: 41 times
Joined: Feb 03, 2017 2:34 pm
Full Name: MikeO
Contact:

Re: REFS 4k horror story

Post by kubimike » 1 person likes this post

@jimmycartrette I went from being stable for a week to stable for about a month. It still crashes but works.
WimVD
Service Provider
Posts: 60
Liked: 19 times
Joined: Dec 23, 2014 4:04 pm
Contact:

Re: REFS 4k horror story

Post by WimVD »

JaySt wrote:Is it perhaps possible for Veeam to create a KB article with the current status and best practices regarding the use of ReFS/Registry settings etc etc.
It's better than sending everyone off to a 43-page forum thread right?
This... Also, it would be nice if Gostev could reach out to the ReFS team for an update on the backport/patch.
Build 1709 is not a solution since it's a different branch with no in-place upgrade.
Gostev
Chief Product Officer
Posts: 31459
Liked: 6648 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: REFS 4k horror story

Post by Gostev » 2 people like this post

I will check with them.
hennys
Lurker
Posts: 2
Liked: 1 time
Joined: Nov 02, 2012 2:14 pm
Full Name: Hen Savelkoul
Contact:

Re: REFS 4k horror story

Post by hennys » 1 person likes this post

Any update yet ?
trenzif
Lurker
Posts: 2
Liked: never
Joined: Nov 15, 2017 4:31 pm
Contact:

Re: REFS 4k horror story

Post by trenzif »

Any news on this?
I also don't see a ReFS remark on the updates provided by microsoft november patch tuesday.

Can somebody confirm that the problems are gone with Windows Server 1709?
WimVD
Service Provider
Posts: 60
Liked: 19 times
Joined: Dec 23, 2014 4:04 pm
Contact:

Re: REFS 4k horror story

Post by WimVD »

trenzif wrote:Can somebody confirm that the problems are gone with Windows Server 1709?
Would like to test but since this needs a OS reinstall I would like some feedback from the ReFS team if the fix was backported in 1709.
Anton, any news? Our merges are taking 10+ hours and getting desperate for a solution...
Delo123
Veteran
Posts: 361
Liked: 109 times
Joined: Dec 28, 2012 5:20 pm
Full Name: Guido Meijers
Contact:

Re: REFS 4k horror story

Post by Delo123 »

You guys are all running Windows core?
WimVD
Service Provider
Posts: 60
Liked: 19 times
Joined: Dec 23, 2014 4:04 pm
Contact:

Re: REFS 4k horror story

Post by WimVD »

Delo123 wrote:You guys are all running Windows core?
Not me, running with desktop. I guess proxies are a good use case for core but I feel a desktop makes life easier when troubleshooting...
Steven Bricklayer
Influencer
Posts: 23
Liked: 2 times
Joined: Apr 04, 2017 8:42 am
Full Name: Steven Bricklayer
Contact:

Re: REFS 4k horror story

Post by Steven Bricklayer »

Hi guys,

For your information our REFS 3.0 volume crashed and came back as a RAW device during a painfull Veeam merge.

Microsoft confirmed the problem come from the file system and couldn't find a way to get our files back
JimmyO
Enthusiast
Posts: 55
Liked: 9 times
Joined: Apr 27, 2014 8:19 pm
Contact:

Re: REFS 4k horror story

Post by JimmyO »

Same thing happened for one of our servers. Backups lost forever...
thomas.raabo
Service Provider
Posts: 28
Liked: 11 times
Joined: Oct 31, 2016 6:27 pm
Full Name: Thomas Raabo
Location: infrastructure guy
Contact:

Re: REFS 4k horror story

Post by thomas.raabo »

Gostev wrote:I will check with them.
Any update?

Has this been fixed in 1709? When will the fix hit Windows 2016 1609

Where are we?
Gostev
Chief Product Officer
Posts: 31459
Liked: 6648 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: REFS 4k horror story

Post by Gostev »

Looks like they had some changes in the team, I need to work with the different person now. I will update once I have something material to share.
trenzif
Lurker
Posts: 2
Liked: never
Joined: Nov 15, 2017 4:31 pm
Contact:

Re: REFS 4k horror story

Post by trenzif »

I got this information from the veeam support regarding Windows Server 1709:

>> Windows release #1709 has not been well tested yet as it was released just month ago. Moreover, it is not supported in Veeam B&R U2 and will break FastClone integration.
Alan_ORiordan
Enthusiast
Posts: 95
Liked: 18 times
Joined: Jun 04, 2014 10:23 am
Full Name: Alan ORiordan
Contact:

Re: REFS 4k horror story

Post by Alan_ORiordan »

I haven't been able to find the answer to this query.

Is 64K Vs 4096K unit size ONLY an issue for the repository server drive? How about the virtual disk partition's of the VM's themselves and the underlying Host Hyper-V VM storage partition?

Are there any issues with say an Exchange or File Server VM virtual drives being formatted with ReFS 4096 unit size?

Sorry if this has already been covered in this lengthly thread

This was a good post for the repository server:

https://www.virtualtothecore.com/en/ref ... kb-or-4kb/
antipolis
Enthusiast
Posts: 73
Liked: 9 times
Joined: Oct 26, 2016 9:17 am
Contact:

Re: REFS 4k horror story

Post by antipolis » 1 person likes this post

Alan_ORiordan wrote:I haven't been able to find the answer to this query.

Are there any issues with say an Exchange or File Server VM virtual drives being formatted with ReFS 4096 unit size?
I must admit that I fail to see any advantages of running ReFS inside VMs... especially for a file server : you will be missing some ntfs only features (https://en.wikipedia.org/wiki/ReFS#Removed_features)

anyway the current problems are to the extent of my knowledge related to block cloning only so my guess is that there is no issues with in-guest refs for an exchange or a file server

just keep in mind that for FLR you will need to do the restores from the veeam console installed on a operating system with refs support
Alan_ORiordan
Enthusiast
Posts: 95
Liked: 18 times
Joined: Jun 04, 2014 10:23 am
Full Name: Alan ORiordan
Contact:

Re: REFS 4k horror story

Post by Alan_ORiordan »

Thanks, the only advantages I can see for a File Server is the end of the 255 file path limit and the added resilience. The new domain will be 100% Server 2016 so File Level Restores will be ok on that front but that did catch us out once before. I have never used disk compression, encryption or quotas which are the main missing features but thanks for the heads-up. I can now continue the build of the new domain feeling comfortable with the choices made.
Lunatic Magnet
Influencer
Posts: 17
Liked: 3 times
Joined: Oct 18, 2017 6:40 pm
Contact:

Re: REFS 4k horror story

Post by Lunatic Magnet » 1 person likes this post

Based on the size of this thread and the fact that myself and several overs are having an issue even with 64k allocation sizes (see veeam-backup-replication-f2/slow-backup ... ml#p244057), I would think it beneficial to remove/edit the KB article recommending REFS.

Many have stated that they're going back to NTFS, but the issue for us (and likely many customers) is that we don't have a large enough location to move the data so we can format the volume. Hoping there is a fix that addresses the issue very soon.
thomas.raabo
Service Provider
Posts: 28
Liked: 11 times
Joined: Oct 31, 2016 6:27 pm
Full Name: Thomas Raabo
Location: infrastructure guy
Contact:

Re: REFS 4k horror story

Post by thomas.raabo » 1 person likes this post

Gostev wrote:Looks like they had some changes in the team, I need to work with the different person now. I will update once I have something material to share.
I´m going all-in on microsoft right now on premier case.

If i do not get answer by tomorrow - I will bump it to Sev A and leave it there for 24/7 resolution.

This has gone on to long! omg!
the_mentor
Enthusiast
Posts: 48
Liked: 8 times
Joined: Jul 26, 2012 11:10 pm
Full Name: DeMentor
Contact:

Re: REFS 4k horror story

Post by the_mentor »

I keep monitoring this post and it reassures me that I've made the right decision
I had nothing but issues with REFS and finally gave up on it and went back to NTFS.
Since I switched back to NTFS everything is working perfectly I really feel like REFS is still a Beta file system that needs to mature and have all of its issues resolved before I can consider it to keep important information like backups.
-DeMentor
kubimike
Veteran
Posts: 373
Liked: 41 times
Joined: Feb 03, 2017 2:34 pm
Full Name: MikeO
Contact:

Re: REFS 4k horror story

Post by kubimike »

OK this is getting confusing. Im still running the beta ReFS driver from 5 months ago. Now I see Windows 1709, is this the eqiv of 'R2' ??? Its a bit annoying that msft hasn't fully addressed this issue with ReFS yet. I DO have a ticket open with Microsoft. Still in a holding pattern.
JaySt
Service Provider
Posts: 415
Liked: 75 times
Joined: Jun 09, 2015 7:08 pm
Full Name: JaySt
Contact:

Re: REFS 4k horror story

Post by JaySt »

quick reminder about the RefsVirtualSyntheticDisabled registry value. if i understand correctly, this disables fast clone functionality on ReFS volumes. I have not tried it myself.
If you like to switch due to fast-clone problems, It could be an option to not re-format a repository to NTFS but just keep data for a while and use ReFS without fastclone by setting this value. Right?
I still have hopes there will be a fix.... eventually...
Veeam Certified Engineer
Locked

Who is online

Users browsing this forum: Bing [Bot], Paul.Loewenkamp and 238 guests