Comprehensive data protection for all workloads
Locked
seritlj
Service Provider
Posts: 3
Liked: never
Joined: Nov 30, 2017 2:27 pm
Full Name: Lars Johansen
Contact:

Re: REFS 4k horror story

Post by seritlj »

I think I read some where that 9.5 U3 will disable REFS Fast Clone on 2016 1609 - is this correct?
florian.meier
Service Provider
Posts: 53
Liked: never
Joined: Dec 01, 2014 11:40 am
Contact:

Re: REFS 4k horror story

Post by florian.meier »

seritlj

as i understand, only if you use deduplication on refs.
right veeam?

TY
thomas.raabo
Service Provider
Posts: 28
Liked: 11 times
Joined: Oct 31, 2016 6:27 pm
Full Name: Thomas Raabo
Location: infrastructure guy
Contact:

Re: REFS 4k horror story

Post by thomas.raabo »

JaySt wrote:quick reminder about the RefsVirtualSyntheticDisabled registry value. if i understand correctly, this disables fast clone functionality on ReFS volumes. I have not tried it myself.
If you like to switch due to fast-clone problems, It could be an option to not re-format a repository to NTFS but just keep data for a while and use ReFS without fastclone by setting this value. Right?
I still have hopes there will be a fix.... eventually...

Yes but it DOES NOT disable general interaction with the API - integrity streams are still getting pushed by veeam
JaySt
Service Provider
Posts: 415
Liked: 75 times
Joined: Jun 09, 2015 7:08 pm
Full Name: JaySt
Contact:

Re: REFS 4k horror story

Post by JaySt »

thomas.raabo wrote: Yes but it DOES NOT disable general interaction with the API - integrity streams are still getting pushed by veeam
Ok, but are you saying you expect to still see issues when this registry value is used?
Veeam Certified Engineer
kubimike
Veteran
Posts: 373
Liked: 41 times
Joined: Feb 03, 2017 2:34 pm
Full Name: MikeO
Contact:

Re: REFS 4k horror story

Post by kubimike »

Just an FYI, my freezing issue is back. After one particular job runs an it deletes old retention points the box has to be powered off over ilo. I am going to try an active full see if that fixes it.
kubimike
Veteran
Posts: 373
Liked: 41 times
Joined: Feb 03, 2017 2:34 pm
Full Name: MikeO
Contact:

Re: REFS 4k horror story

Post by kubimike »

FWIW, active full stopped the freezes. Back to normal.
thomas.raabo
Service Provider
Posts: 28
Liked: 11 times
Joined: Oct 31, 2016 6:27 pm
Full Name: Thomas Raabo
Location: infrastructure guy
Contact:

Re: REFS 4k horror story

Post by thomas.raabo »

JaySt wrote:
Ok, but are you saying you expect to still see issues when this registry value is used?
Yes... been running with the reg since almost day one

and there are still big problems
dmartenstyn
Lurker
Posts: 2
Liked: never
Joined: Apr 24, 2017 1:35 pm
Contact:

Re: REFS 4k horror story

Post by dmartenstyn »

Ok this has gone beyond ridiculous. I have been carefully monitoring this thread and associated issues for months now. Personally I think 2 things.

1) Microsoft should not have released ReFS to the general public. In my opinion with the underlying issues and considering it is a file system, you know, a core component of the OS, its still in beta retrospectively speaking.

2) Veeam should pull all mention of using ReFS from their documentation and best practices (as well as advertisements regarding interoperability and its benefits, etc).

I am just about to deploy my 2nd Veeam server and there is absolutely no way I consider ReFS production ready. I will also be re-formatting the initial Veeam server to NTFS (luckily I have space to do this). Microsoft and Veeam both need to pull their respective fingers out and get this sorted, it has quite literally dragged on for months now and frankly is unacceptable. I can only imagine a 'newbie' not doing a search beforehand and finding this thread and going with best practices, etc and using ReFS :(
Steven Bricklayer
Influencer
Posts: 23
Liked: 2 times
Joined: Apr 04, 2017 8:42 am
Full Name: Steven Bricklayer
Contact:

Re: REFS 4k horror story

Post by Steven Bricklayer »

I agree with you dmartenstyn

I really think that Veeam don't understand how bad the situation is about ReFs

The big performance problems are a thing, but the real big problem is that 2 customers on that thread lost the Refs Volume with no way to recover it!

That storage contains BACKUPS, it's really critical.

I'm with the Microsoft support since 1 month, on the maximum level with the maximum parternship you can imagine, and they can't recover the data!

So if you get a quick resume :
- Refs perf are bads most of the time
- You can loose all your data because of Refs

Maybe it's a Microsoft issue but who care?

You veeam guys really must STOP advice customers to go Refs, this is to dangerous
ferrus
Veeam ProPartner
Posts: 299
Liked: 43 times
Joined: Dec 03, 2015 3:41 pm
Location: UK
Contact:

Re: REFS 4k horror story

Post by ferrus »

In ReFS' defence ... :shock: :!:

For some customers (us included), NTFS is not sufficient for our needs at the moment, and keeping the status quo brings with it as many issues as the ones currently reported with ReFS.
Some of the posts make an assumption that people are migrating from a position of stability and high performance, to something much worse. The opposite can be true.

NTFS is sufficient on four out of five of our five Veeam repositories, but the nature of the VMs being backed up on the fifth, means multiple synthetic/active fulls break the storage capacity for our current RPO strategy, and merge jobs break the backup window.
ReFS/Fast Clone - provides a solution for both of these, and if an Active Full is required occasionally to reset the performance - so be it. It's no worse than our current position, in fact much, much better.

The recent reports of data loss on this thread, are very troubling however. That has me concerned.
We exceed the 3-2-1 rule for backups, but that's to plan against the possibility of data loss - not the expectation.
kubimike
Veteran
Posts: 373
Liked: 41 times
Joined: Feb 03, 2017 2:34 pm
Full Name: MikeO
Contact:

Re: REFS 4k horror story

Post by kubimike »

This issue with ReFS only comes up if you have a lot of synthetic fulls like myself. I do one per day to keep performance up when doing a restore, else it suffers and you have freezing problems. Other then that I really have no complains. Is it a hassle? Sure but configured probably the benefits for us outweigh switching back to NTFS.
DaStivi
Service Provider
Posts: 250
Liked: 34 times
Joined: Jun 30, 2015 9:13 am
Full Name: Stephan Lang
Location: Austria
Contact:

Re: REFS 4k horror story

Post by DaStivi » 1 person likes this post

as far as i understand right now:
  • never go up to more than 64TB per REFS Volume!!!! (issues starts directly -> slow performance etc..)
  • only use forward incr. forever (no syn fulls, this is primary the root cause of many problems, refs would make syn fulls "fast" because it just repointers tables... but basically this looks like to be a big issue, because more syn fulls means more and more pointers to be made and taken care of it)
  • never let the refs volume get full, i guess leave at least 10, better 15 or 20% of free space
  • don't backup very large vms to refs (vms bigger couple of TBs)
  • when formating refs make totally sure that you select 64k Block size!! -> 64k was default till server 2012r2, but now with server 2016 "default" settings is 4k be carefull to select 64k there!!!
i've couple of Veeam installations where refs does its job very good, and i've installations where i had big troubles with refs and gone back to ntfs now!
kubimike
Veteran
Posts: 373
Liked: 41 times
Joined: Feb 03, 2017 2:34 pm
Full Name: MikeO
Contact:

Re: REFS 4k horror story

Post by kubimike »

DaStivi wrote:as far as i understand right now:
  • never go up to more than 64TB per REFS Volume!!!! (issues starts directly -> slow performance etc..) True
  • only use forward incr. forever (no syn fulls, this is primary the root cause of many problems, refs would make syn fulls "fast" because it just repointers tables... but basically this looks like to be a big issue, because more syn fulls means more and more pointers to be made and taken care of it) Not sure I agree, more chance of corruption not to mention slow restores.
  • never let the refs volume get full, i guess leave at least 10, better 15 or 20% of free space I wouldn't let any volume get full
  • don't backup very large vms to refs (vms bigger couple of TBs) I have an 8TB job that runs synthetics daily. As long as I run an active full every 2 months no issues.
  • when formating refs make totally sure that you select 64k Block size!! -> 64k was default till server 2012r2, but now with server 2016 "default" settings is 4k be carefull to select 64k there!!!
Totally!
DaStivi wrote:i've couple of Veeam installations where refs does its job very good, and i've installations where i had big troubles with refs and gone back to ntfs now!
Get the beta driver, apply all the settings mentioned in this thread.
Lunatic Magnet
Influencer
Posts: 17
Liked: 3 times
Joined: Oct 18, 2017 6:40 pm
Contact:

Re: REFS 4k horror story

Post by Lunatic Magnet »

kubimike wrote: Get the beta driver, apply all the settings mentioned in this thread.
Based on threads earlier around page 35-39 several people stated that the beta driver was included in the July Cumulative updates. My server was built in September and has the latest updates and it states that KB4025334 isn't applicable. My refs.sys driver version is 10.0.14393.1770.

The MS article says Specify the indicated values in the following subkey: Is that indicating they should already be there? In my case they are not.
thomas.raabo
Service Provider
Posts: 28
Liked: 11 times
Joined: Oct 31, 2016 6:27 pm
Full Name: Thomas Raabo
Location: infrastructure guy
Contact:

Re: REFS 4k horror story

Post by thomas.raabo » 2 people like this post

Lunatic Magnet wrote: Based on threads earlier around page 35-39 several people stated that the beta driver was included in the July Cumulative updates. My server was built in September and has the latest updates and it states that KB4025334 isn't applicable. My refs.sys driver version is 10.0.14393.1770.

The MS article says Specify the indicated values in the following subkey: Is that indicating they should already be there? In my case they are not.
Hi Guys!

I´ve been working very very hard with Microsoft and all RS4 fixes have been backported to RS1 and i can confirm that it fixes all the problems i have been seeing.

The driver version you need to get from Microsoft is 10.0.14939.1934

You can refer to my Microsoft case :
[REG:117112217202384] ReFS volumes goes offline while Veeam is using BlockCloneAPI

The fixes also includes best practice registrys that does not at all look like the KB on ReFS problems
rvvliet78
Novice
Posts: 3
Liked: 1 time
Joined: Apr 26, 2017 9:43 am
Full Name: Rick van Vliet
Contact:

Re: REFS 4k horror story

Post by rvvliet78 »

Hi Guys!

I´ve been working very very hard with Microsoft and all RS4 fixes have been backported to RS1 and i can confirm that it fixes all the problems i have been seeing.

The driver version you need to get from Microsoft is 10.0.14939.1934

You can refer to my Microsoft case :
[REG:117112217202384] ReFS volumes goes offline while Veeam is using BlockCloneAPI

The fixes also includes best practice registrys that does not at all look like the KB on ReFS problems
This sounds good, did they give any indication when this will be released?
kubimike
Veteran
Posts: 373
Liked: 41 times
Joined: Feb 03, 2017 2:34 pm
Full Name: MikeO
Contact:

Re: REFS 4k horror story

Post by kubimike »

@lunatic magnet
My microsoft rep and veeam rep have told me the opposite. Not to mention back a few months ago I tried the latest refs driver release to the public and the system freezes. I still to this date running the beta refs driver. Its the only thing that keeps my system running.
kubimike
Veteran
Posts: 373
Liked: 41 times
Joined: Feb 03, 2017 2:34 pm
Full Name: MikeO
Contact:

Re: REFS 4k horror story

Post by kubimike »

thomas.raabo wrote:The fixes also includes best practice registrys that does not at all look like the KB on ReFS problems
Could you clarify what you're saying here ? Where can we get a copy of the build you're referring to ?
Lunatic Magnet
Influencer
Posts: 17
Liked: 3 times
Joined: Oct 18, 2017 6:40 pm
Contact:

Re: REFS 4k horror story

Post by Lunatic Magnet »

kubimike wrote:@lunatic magnet
My microsoft rep and veeam rep have told me the opposite. Not to mention back a few months ago I tried the latest refs driver release to the public and the system freezes. I still to this date running the beta refs driver. Its the only thing that keeps my system running.
Sorry, what part is opposite?
kubimike
Veteran
Posts: 373
Liked: 41 times
Joined: Feb 03, 2017 2:34 pm
Full Name: MikeO
Contact:

Re: REFS 4k horror story

Post by kubimike »

@lunatic magnet - that the beta refs fixes haven't been ported to public release.
Lunatic Magnet
Influencer
Posts: 17
Liked: 3 times
Joined: Oct 18, 2017 6:40 pm
Contact:

Re: REFS 4k horror story

Post by Lunatic Magnet »

kubimike wrote:@lunatic magnet - that the beta refs fixes haven't been ported to public release.
I just logged a Premiere call with MS and referenced that case. I'll update what the scoop is and my ticket # for reference if useful.
kubimike
Veteran
Posts: 373
Liked: 41 times
Joined: Feb 03, 2017 2:34 pm
Full Name: MikeO
Contact:

Re: REFS 4k horror story

Post by kubimike »

@lunatic magnet - so you have refs version 10.0.14939.1934 ?
Lunatic Magnet
Influencer
Posts: 17
Liked: 3 times
Joined: Oct 18, 2017 6:40 pm
Contact:

Re: REFS 4k horror story

Post by Lunatic Magnet »

kubimike wrote:@lunatic magnet - so you have refs version 10.0.14939.1934 ?
No, mine is currently 10.0.14393.1770. Windows 2016 build 1607 (14393.1884) with all latest patches. I'll see what support has to say
thomas.raabo
Service Provider
Posts: 28
Liked: 11 times
Joined: Oct 31, 2016 6:27 pm
Full Name: Thomas Raabo
Location: infrastructure guy
Contact:

Re: REFS 4k horror story

Post by thomas.raabo »

kubimike wrote:@lunatic magnet - that the beta refs fixes haven't been ported to public release.
Thats just to string you along.
Like i said - It was no easy task getting this update.

I had my case running for 3 weeks at Sev A to get it... And it has only been released to six customers.
kubimike
Veteran
Posts: 373
Liked: 41 times
Joined: Feb 03, 2017 2:34 pm
Full Name: MikeO
Contact:

Re: REFS 4k horror story

Post by kubimike »

@thomas Well that’s not so cool. Veeam is a Microsoft partner and they don’t even know it exists. Could you clarify your ‘registry’ statement above ?
thomas.raabo
Service Provider
Posts: 28
Liked: 11 times
Joined: Oct 31, 2016 6:27 pm
Full Name: Thomas Raabo
Location: infrastructure guy
Contact:

Re: REFS 4k horror story

Post by thomas.raabo » 1 person likes this post

kubimike wrote:@thomas Well that’s not so cool. Veeam is a Microsoft partner and they don’t even know it exists. Could you clarify your ‘registry’ statement above ?
With this new update i was told to remove all ReFS registry I already had.

Along the update came the new recommendation

Windows Registry Editor Version 5.00

[HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\FileSystem]
"RefsProcessedDeleteQueueEntryCountThreshold"=dword:00010000
"RefsEnableLargeWorkingSetTrim"=dword:00000001
"RefsCheckpointSampleInterval"=dword:00000001
"RefsContainerRotationSampleInterval"=dword:00000001
"RefsLogFileFullSampleInterval"=dword:00000001
;"RefsDisableRefCountParallelDelete"=dword:00000001
;"RefsNumberOfChunksToTrim"=dword:0
;"RefsEnableInlineTrim"=dword:00000001
;"RefsDisableCachedPins"=dword:00000001
pesos
Expert
Posts: 205
Liked: 17 times
Joined: Nov 12, 2014 9:40 am
Full Name: John Johnson
Contact:

Re: REFS 4k horror story

Post by pesos »

Since we do on-host, I assume we would need these refs fixes on our hyper-v hosts, not necessarily on the veeam server(s)?
WimVD
Service Provider
Posts: 60
Liked: 19 times
Joined: Dec 23, 2014 4:04 pm
Contact:

Re: REFS 4k horror story

Post by WimVD » 1 person likes this post

Just read Anton's digest and finally feeling a bit confident things will get fixed. Thank you all and merry christmas :D
nmdange
Veteran
Posts: 527
Liked: 142 times
Joined: Aug 20, 2015 9:30 pm
Contact:

Re: REFS 4k horror story

Post by nmdange »

pesos wrote:Since we do on-host, I assume we would need these refs fixes on our hyper-v hosts, not necessarily on the veeam server(s)?
The ReFS issue applies to backup repositories formatted with ReFS, so it would be whatever Windows Server hosts the backup repository volumes. Your Hyper-V hosts would not need the fix.
kubimike
Veteran
Posts: 373
Liked: 41 times
Joined: Feb 03, 2017 2:34 pm
Full Name: MikeO
Contact:

Re: REFS 4k horror story

Post by kubimike »

thomas.raabo wrote: Windows Registry Editor Version 5.00

[HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\FileSystem]
"RefsProcessedDeleteQueueEntryCountThreshold"=dword:00010000
"RefsEnableLargeWorkingSetTrim"=dword:00000001
"RefsCheckpointSampleInterval"=dword:00000001
"RefsContainerRotationSampleInterval"=dword:00000001
"RefsLogFileFullSampleInterval"=dword:00000001
;"RefsDisableRefCountParallelDelete"=dword:00000001
;"RefsNumberOfChunksToTrim"=dword:0
;"RefsEnableInlineTrim"=dword:00000001
;"RefsDisableCachedPins"=dword:00000001
I see new goodies there. Interesting they now have you set '"RefsProcessedDeleteQueueEntryCountThreshold"=dword:00010000' , with the beta driver the only thing that works for me is value 'dword:00000200'

I curious about 'RefsDisableRefCountParallelDelete' I wonder, if this throttles the pointers when deleting multiple cloned files at once. Any reason why they've had you not use RefsDisableCachedPins ?
Locked

Who is online

Users browsing this forum: rjv1971, Semrush [Bot] and 202 guests