Availability for the Always-On Enterprise
Locked
seritlj
Service Provider
Posts: 3
Liked: never
Joined: Nov 30, 2017 2:27 pm
Full Name: Lars Johansen
Contact:

Re: REFS 4k horror story

Post by seritlj » Nov 30, 2017 2:31 pm

I think I read some where that 9.5 U3 will disable REFS Fast Clone on 2016 1609 - is this correct?

florian.meier
Service Provider
Posts: 52
Liked: never
Joined: Dec 01, 2014 11:40 am
Contact:

Re: REFS 4k horror story

Post by florian.meier » Dec 01, 2017 7:42 am

seritlj

as i understand, only if you use deduplication on refs.
right veeam?

TY

thomas.raabo
Service Provider
Posts: 28
Liked: 11 times
Joined: Oct 31, 2016 6:27 pm
Full Name: Thomas Raabo
Location: infrastructure guy
Contact:

Re: REFS 4k horror story

Post by thomas.raabo » Dec 01, 2017 9:29 am

JaySt wrote:quick reminder about the RefsVirtualSyntheticDisabled registry value. if i understand correctly, this disables fast clone functionality on ReFS volumes. I have not tried it myself.
If you like to switch due to fast-clone problems, It could be an option to not re-format a repository to NTFS but just keep data for a while and use ReFS without fastclone by setting this value. Right?
I still have hopes there will be a fix.... eventually...

Yes but it DOES NOT disable general interaction with the API - integrity streams are still getting pushed by veeam

JaySt
Service Provider
Posts: 88
Liked: 19 times
Joined: Jun 09, 2015 7:08 pm
Full Name: JaySt
Contact:

Re: REFS 4k horror story

Post by JaySt » Dec 01, 2017 10:03 am

thomas.raabo wrote: Yes but it DOES NOT disable general interaction with the API - integrity streams are still getting pushed by veeam
Ok, but are you saying you expect to still see issues when this registry value is used?
Veeam Certified Engineer

kubimike
Expert
Posts: 324
Liked: 37 times
Joined: Feb 03, 2017 2:34 pm
Full Name: MikeO
Contact:

Re: REFS 4k horror story

Post by kubimike » Dec 01, 2017 4:25 pm

Just an FYI, my freezing issue is back. After one particular job runs an it deletes old retention points the box has to be powered off over ilo. I am going to try an active full see if that fixes it.

kubimike
Expert
Posts: 324
Liked: 37 times
Joined: Feb 03, 2017 2:34 pm
Full Name: MikeO
Contact:

Re: REFS 4k horror story

Post by kubimike » Dec 03, 2017 8:39 am

FWIW, active full stopped the freezes. Back to normal.

thomas.raabo
Service Provider
Posts: 28
Liked: 11 times
Joined: Oct 31, 2016 6:27 pm
Full Name: Thomas Raabo
Location: infrastructure guy
Contact:

Re: REFS 4k horror story

Post by thomas.raabo » Dec 04, 2017 11:02 am

JaySt wrote:
Ok, but are you saying you expect to still see issues when this registry value is used?
Yes... been running with the reg since almost day one

and there are still big problems

dmartenstyn
Lurker
Posts: 2
Liked: never
Joined: Apr 24, 2017 1:35 pm
Contact:

Re: REFS 4k horror story

Post by dmartenstyn » Dec 04, 2017 11:43 am

Ok this has gone beyond ridiculous. I have been carefully monitoring this thread and associated issues for months now. Personally I think 2 things.

1) Microsoft should not have released ReFS to the general public. In my opinion with the underlying issues and considering it is a file system, you know, a core component of the OS, its still in beta retrospectively speaking.

2) Veeam should pull all mention of using ReFS from their documentation and best practices (as well as advertisements regarding interoperability and its benefits, etc).

I am just about to deploy my 2nd Veeam server and there is absolutely no way I consider ReFS production ready. I will also be re-formatting the initial Veeam server to NTFS (luckily I have space to do this). Microsoft and Veeam both need to pull their respective fingers out and get this sorted, it has quite literally dragged on for months now and frankly is unacceptable. I can only imagine a 'newbie' not doing a search beforehand and finding this thread and going with best practices, etc and using ReFS :(

Steven Bricklayer
Influencer
Posts: 17
Liked: never
Joined: Apr 04, 2017 8:42 am
Full Name: Steven Bricklayer
Contact:

Re: REFS 4k horror story

Post by Steven Bricklayer » Dec 04, 2017 1:50 pm

I agree with you dmartenstyn

I really think that Veeam don't understand how bad the situation is about ReFs

The big performance problems are a thing, but the real big problem is that 2 customers on that thread lost the Refs Volume with no way to recover it!

That storage contains BACKUPS, it's really critical.

I'm with the Microsoft support since 1 month, on the maximum level with the maximum parternship you can imagine, and they can't recover the data!

So if you get a quick resume :
- Refs perf are bads most of the time
- You can loose all your data because of Refs

Maybe it's a Microsoft issue but who care?

You veeam guys really must STOP advice customers to go Refs, this is to dangerous

ferrus
Veeam ProPartner
Posts: 190
Liked: 23 times
Joined: Dec 03, 2015 3:41 pm
Location: UK
Contact:

Re: REFS 4k horror story

Post by ferrus » Dec 04, 2017 4:59 pm

In ReFS' defence ... :shock: :!:

For some customers (us included), NTFS is not sufficient for our needs at the moment, and keeping the status quo brings with it as many issues as the ones currently reported with ReFS.
Some of the posts make an assumption that people are migrating from a position of stability and high performance, to something much worse. The opposite can be true.

NTFS is sufficient on four out of five of our five Veeam repositories, but the nature of the VMs being backed up on the fifth, means multiple synthetic/active fulls break the storage capacity for our current RPO strategy, and merge jobs break the backup window.
ReFS/Fast Clone - provides a solution for both of these, and if an Active Full is required occasionally to reset the performance - so be it. It's no worse than our current position, in fact much, much better.

The recent reports of data loss on this thread, are very troubling however. That has me concerned.
We exceed the 3-2-1 rule for backups, but that's to plan against the possibility of data loss - not the expectation.

kubimike
Expert
Posts: 324
Liked: 37 times
Joined: Feb 03, 2017 2:34 pm
Full Name: MikeO
Contact:

Re: REFS 4k horror story

Post by kubimike » Dec 04, 2017 7:42 pm

This issue with ReFS only comes up if you have a lot of synthetic fulls like myself. I do one per day to keep performance up when doing a restore, else it suffers and you have freezing problems. Other then that I really have no complains. Is it a hassle? Sure but configured probably the benefits for us outweigh switching back to NTFS.

DaStivi
Service Provider
Posts: 93
Liked: 10 times
Joined: Jun 30, 2015 9:13 am
Full Name: Stephan Lang
Contact:

Re: REFS 4k horror story

Post by DaStivi » Dec 07, 2017 8:53 am 1 person likes this post

as far as i understand right now:
  • never go up to more than 64TB per REFS Volume!!!! (issues starts directly -> slow performance etc..)
  • only use forward incr. forever (no syn fulls, this is primary the root cause of many problems, refs would make syn fulls "fast" because it just repointers tables... but basically this looks like to be a big issue, because more syn fulls means more and more pointers to be made and taken care of it)
  • never let the refs volume get full, i guess leave at least 10, better 15 or 20% of free space
  • don't backup very large vms to refs (vms bigger couple of TBs)
  • when formating refs make totally sure that you select 64k Block size!! -> 64k was default till server 2012r2, but now with server 2016 "default" settings is 4k be carefull to select 64k there!!!
i've couple of Veeam installations where refs does its job very good, and i've installations where i had big troubles with refs and gone back to ntfs now!

kubimike
Expert
Posts: 324
Liked: 37 times
Joined: Feb 03, 2017 2:34 pm
Full Name: MikeO
Contact:

Re: REFS 4k horror story

Post by kubimike » Dec 07, 2017 7:48 pm

DaStivi wrote:as far as i understand right now:
  • never go up to more than 64TB per REFS Volume!!!! (issues starts directly -> slow performance etc..) True
  • only use forward incr. forever (no syn fulls, this is primary the root cause of many problems, refs would make syn fulls "fast" because it just repointers tables... but basically this looks like to be a big issue, because more syn fulls means more and more pointers to be made and taken care of it) Not sure I agree, more chance of corruption not to mention slow restores.
  • never let the refs volume get full, i guess leave at least 10, better 15 or 20% of free space I wouldn't let any volume get full
  • don't backup very large vms to refs (vms bigger couple of TBs) I have an 8TB job that runs synthetics daily. As long as I run an active full every 2 months no issues.
  • when formating refs make totally sure that you select 64k Block size!! -> 64k was default till server 2012r2, but now with server 2016 "default" settings is 4k be carefull to select 64k there!!!
Totally!
DaStivi wrote:i've couple of Veeam installations where refs does its job very good, and i've installations where i had big troubles with refs and gone back to ntfs now!
Get the beta driver, apply all the settings mentioned in this thread.

Lunatic Magnet
Influencer
Posts: 15
Liked: 3 times
Joined: Oct 18, 2017 6:40 pm
Contact:

Re: REFS 4k horror story

Post by Lunatic Magnet » Dec 07, 2017 8:39 pm

kubimike wrote: Get the beta driver, apply all the settings mentioned in this thread.
Based on threads earlier around page 35-39 several people stated that the beta driver was included in the July Cumulative updates. My server was built in September and has the latest updates and it states that KB4025334 isn't applicable. My refs.sys driver version is 10.0.14393.1770.

The MS article says Specify the indicated values in the following subkey: Is that indicating they should already be there? In my case they are not.

thomas.raabo
Service Provider
Posts: 28
Liked: 11 times
Joined: Oct 31, 2016 6:27 pm
Full Name: Thomas Raabo
Location: infrastructure guy
Contact:

Re: REFS 4k horror story

Post by thomas.raabo » Dec 08, 2017 12:16 pm 2 people like this post

Lunatic Magnet wrote: Based on threads earlier around page 35-39 several people stated that the beta driver was included in the July Cumulative updates. My server was built in September and has the latest updates and it states that KB4025334 isn't applicable. My refs.sys driver version is 10.0.14393.1770.

The MS article says Specify the indicated values in the following subkey: Is that indicating they should already be there? In my case they are not.
Hi Guys!

I´ve been working very very hard with Microsoft and all RS4 fixes have been backported to RS1 and i can confirm that it fixes all the problems i have been seeing.

The driver version you need to get from Microsoft is 10.0.14939.1934

You can refer to my Microsoft case :
[REG:117112217202384] ReFS volumes goes offline while Veeam is using BlockCloneAPI

The fixes also includes best practice registrys that does not at all look like the KB on ReFS problems

Locked

Who is online

Users browsing this forum: Bing [Bot] and 30 guests