REFS issues (server lockups, high CPU, high RAM)

Availability for the Always-On Enterprise

Re: REFS 4k horror story

Veeam Logoby seritlj » Thu Nov 30, 2017 2:31 pm

I think I read some where that 9.5 U3 will disable REFS Fast Clone on 2016 1609 - is this correct?
seritlj
Service Provider
 
Posts: 1
Liked: never
Joined: Thu Nov 30, 2017 2:27 pm
Full Name: Lars Johansen

Re: REFS 4k horror story

Veeam Logoby florian.meier » Fri Dec 01, 2017 7:42 am

seritlj

as i understand, only if you use deduplication on refs.
right veeam?

TY
florian.meier
Service Provider
 
Posts: 47
Liked: never
Joined: Mon Dec 01, 2014 11:40 am

Re: REFS 4k horror story

Veeam Logoby thomas.raabo » Fri Dec 01, 2017 9:29 am

JaySt wrote:quick reminder about the RefsVirtualSyntheticDisabled registry value. if i understand correctly, this disables fast clone functionality on ReFS volumes. I have not tried it myself.
If you like to switch due to fast-clone problems, It could be an option to not re-format a repository to NTFS but just keep data for a while and use ReFS without fastclone by setting this value. Right?
I still have hopes there will be a fix.... eventually...



Yes but it DOES NOT disable general interaction with the API - integrity streams are still getting pushed by veeam
thomas.raabo
Service Provider
 
Posts: 21
Liked: 9 times
Joined: Mon Oct 31, 2016 6:27 pm
Location: infrastructure guy
Full Name: Thomas Raabo

Re: REFS 4k horror story

Veeam Logoby JaySt » Fri Dec 01, 2017 10:03 am

thomas.raabo wrote:Yes but it DOES NOT disable general interaction with the API - integrity streams are still getting pushed by veeam


Ok, but are you saying you expect to still see issues when this registry value is used?
Veeam Certified Engineer
JaySt
Service Provider
 
Posts: 70
Liked: 15 times
Joined: Tue Jun 09, 2015 7:08 pm
Full Name: JaySt

Re: REFS 4k horror story

Veeam Logoby kubimike » Fri Dec 01, 2017 4:25 pm

Just an FYI, my freezing issue is back. After one particular job runs an it deletes old retention points the box has to be powered off over ilo. I am going to try an active full see if that fixes it.
kubimike
Expert
 
Posts: 260
Liked: 25 times
Joined: Fri Feb 03, 2017 2:34 pm
Full Name: MikeO

Re: REFS 4k horror story

Veeam Logoby kubimike » Sun Dec 03, 2017 8:39 am

FWIW, active full stopped the freezes. Back to normal.
kubimike
Expert
 
Posts: 260
Liked: 25 times
Joined: Fri Feb 03, 2017 2:34 pm
Full Name: MikeO

Re: REFS 4k horror story

Veeam Logoby thomas.raabo » Mon Dec 04, 2017 11:02 am

JaySt wrote:
Ok, but are you saying you expect to still see issues when this registry value is used?


Yes... been running with the reg since almost day one

and there are still big problems
thomas.raabo
Service Provider
 
Posts: 21
Liked: 9 times
Joined: Mon Oct 31, 2016 6:27 pm
Location: infrastructure guy
Full Name: Thomas Raabo

Re: REFS 4k horror story

Veeam Logoby dmartenstyn » Mon Dec 04, 2017 11:43 am

Ok this has gone beyond ridiculous. I have been carefully monitoring this thread and associated issues for months now. Personally I think 2 things.

1) Microsoft should not have released ReFS to the general public. In my opinion with the underlying issues and considering it is a file system, you know, a core component of the OS, its still in beta retrospectively speaking.

2) Veeam should pull all mention of using ReFS from their documentation and best practices (as well as advertisements regarding interoperability and its benefits, etc).

I am just about to deploy my 2nd Veeam server and there is absolutely no way I consider ReFS production ready. I will also be re-formatting the initial Veeam server to NTFS (luckily I have space to do this). Microsoft and Veeam both need to pull their respective fingers out and get this sorted, it has quite literally dragged on for months now and frankly is unacceptable. I can only imagine a 'newbie' not doing a search beforehand and finding this thread and going with best practices, etc and using ReFS :(
dmartenstyn
Lurker
 
Posts: 2
Liked: never
Joined: Mon Apr 24, 2017 1:35 pm

Re: REFS 4k horror story

Veeam Logoby Steven Bricklayer » Mon Dec 04, 2017 1:50 pm

I agree with you dmartenstyn

I really think that Veeam don't understand how bad the situation is about ReFs

The big performance problems are a thing, but the real big problem is that 2 customers on that thread lost the Refs Volume with no way to recover it!

That storage contains BACKUPS, it's really critical.

I'm with the Microsoft support since 1 month, on the maximum level with the maximum parternship you can imagine, and they can't recover the data!

So if you get a quick resume :
- Refs perf are bads most of the time
- You can loose all your data because of Refs

Maybe it's a Microsoft issue but who care?

You veeam guys really must STOP advice customers to go Refs, this is to dangerous
Steven Bricklayer
Influencer
 
Posts: 15
Liked: never
Joined: Tue Apr 04, 2017 8:42 am
Full Name: Steven Bricklayer

Re: REFS 4k horror story

Veeam Logoby ferrus » Mon Dec 04, 2017 4:59 pm

In ReFS' defence ... :shock: :!:

For some customers (us included), NTFS is not sufficient for our needs at the moment, and keeping the status quo brings with it as many issues as the ones currently reported with ReFS.
Some of the posts make an assumption that people are migrating from a position of stability and high performance, to something much worse. The opposite can be true.

NTFS is sufficient on four out of five of our five Veeam repositories, but the nature of the VMs being backed up on the fifth, means multiple synthetic/active fulls break the storage capacity for our current RPO strategy, and merge jobs break the backup window.
ReFS/Fast Clone - provides a solution for both of these, and if an Active Full is required occasionally to reset the performance - so be it. It's no worse than our current position, in fact much, much better.

The recent reports of data loss on this thread, are very troubling however. That has me concerned.
We exceed the 3-2-1 rule for backups, but that's to plan against the possibility of data loss - not the expectation.
ferrus
Veeam ProPartner
 
Posts: 149
Liked: 20 times
Joined: Thu Dec 03, 2015 3:41 pm
Location: UK

Re: REFS 4k horror story

Veeam Logoby kubimike » Mon Dec 04, 2017 7:42 pm

This issue with ReFS only comes up if you have a lot of synthetic fulls like myself. I do one per day to keep performance up when doing a restore, else it suffers and you have freezing problems. Other then that I really have no complains. Is it a hassle? Sure but configured probably the benefits for us outweigh switching back to NTFS.
kubimike
Expert
 
Posts: 260
Liked: 25 times
Joined: Fri Feb 03, 2017 2:34 pm
Full Name: MikeO

Re: REFS 4k horror story

Veeam Logoby DaStivi » Thu Dec 07, 2017 8:53 am 1 person likes this post

as far as i understand right now:

  • never go up to more than 64TB per REFS Volume!!!! (issues starts directly -> slow performance etc..)
  • only use forward incr. forever (no syn fulls, this is primary the root cause of many problems, refs would make syn fulls "fast" because it just repointers tables... but basically this looks like to be a big issue, because more syn fulls means more and more pointers to be made and taken care of it)
  • never let the refs volume get full, i guess leave at least 10, better 15 or 20% of free space
  • don't backup very large vms to refs (vms bigger couple of TBs)
  • when formating refs make totally sure that you select 64k Block size!! -> 64k was default till server 2012r2, but now with server 2016 "default" settings is 4k be carefull to select 64k there!!!

i've couple of Veeam installations where refs does its job very good, and i've installations where i had big troubles with refs and gone back to ntfs now!
DaStivi
Enthusiast
 
Posts: 57
Liked: 5 times
Joined: Tue Jun 30, 2015 9:13 am
Full Name: Stephan Lang

Re: REFS 4k horror story

Veeam Logoby kubimike » Thu Dec 07, 2017 7:48 pm

DaStivi wrote:as far as i understand right now:

  • never go up to more than 64TB per REFS Volume!!!! (issues starts directly -> slow performance etc..) True
  • only use forward incr. forever (no syn fulls, this is primary the root cause of many problems, refs would make syn fulls "fast" because it just repointers tables... but basically this looks like to be a big issue, because more syn fulls means more and more pointers to be made and taken care of it) Not sure I agree, more chance of corruption not to mention slow restores.
  • never let the refs volume get full, i guess leave at least 10, better 15 or 20% of free space I wouldn't let any volume get full
  • don't backup very large vms to refs (vms bigger couple of TBs) I have an 8TB job that runs synthetics daily. As long as I run an active full every 2 months no issues.
  • when formating refs make totally sure that you select 64k Block size!! -> 64k was default till server 2012r2, but now with server 2016 "default" settings is 4k be carefull to select 64k there!!!

Totally!

DaStivi wrote:i've couple of Veeam installations where refs does its job very good, and i've installations where i had big troubles with refs and gone back to ntfs now!

Get the beta driver, apply all the settings mentioned in this thread.
kubimike
Expert
 
Posts: 260
Liked: 25 times
Joined: Fri Feb 03, 2017 2:34 pm
Full Name: MikeO

Re: REFS 4k horror story

Veeam Logoby Lunatic Magnet » Thu Dec 07, 2017 8:39 pm

kubimike wrote:Get the beta driver, apply all the settings mentioned in this thread.

Based on threads earlier around page 35-39 several people stated that the beta driver was included in the July Cumulative updates. My server was built in September and has the latest updates and it states that KB4025334 isn't applicable. My refs.sys driver version is 10.0.14393.1770.

The MS article says Specify the indicated values in the following subkey: Is that indicating they should already be there? In my case they are not.
Lunatic Magnet
Influencer
 
Posts: 13
Liked: 2 times
Joined: Wed Oct 18, 2017 6:40 pm

Re: REFS 4k horror story

Veeam Logoby thomas.raabo » Fri Dec 08, 2017 12:16 pm 2 people like this post

Lunatic Magnet wrote:Based on threads earlier around page 35-39 several people stated that the beta driver was included in the July Cumulative updates. My server was built in September and has the latest updates and it states that KB4025334 isn't applicable. My refs.sys driver version is 10.0.14393.1770.

The MS article says Specify the indicated values in the following subkey: Is that indicating they should already be there? In my case they are not.

Hi Guys!

I´ve been working very very hard with Microsoft and all RS4 fixes have been backported to RS1 and i can confirm that it fixes all the problems i have been seeing.

The driver version you need to get from Microsoft is 10.0.14939.1934

You can refer to my Microsoft case :
[REG:117112217202384] ReFS volumes goes offline while Veeam is using BlockCloneAPI

The fixes also includes best practice registrys that does not at all look like the KB on ReFS problems
thomas.raabo
Service Provider
 
Posts: 21
Liked: 9 times
Joined: Mon Oct 31, 2016 6:27 pm
Location: infrastructure guy
Full Name: Thomas Raabo

PreviousNext

Return to Veeam Backup & Replication



Who is online

Users browsing this forum: baber, Google [Bot], tdewin and 1 guest