Comprehensive data protection for all workloads
Locked
SBarrett847
Service Provider
Posts: 315
Liked: 41 times
Joined: Feb 02, 2016 5:02 pm
Full Name: Stephen Barrett
Contact:

Re: REFS issues (server lockups, high CPU, high RAM)

Post by SBarrett847 »

firewall68 wrote:Ok i`ve did it, but .......
According MS the update will fix the issue but you have to format the disk again with REFS. therfor all data on the RAW Device are lost.
"We had to destroy the Village in order to save it........."

I really hope this isn't true.
Neil Flanagan
Novice
Posts: 4
Liked: never
Joined: Aug 26, 2011 8:24 am
Full Name: Neil Flanagan
Contact:

Re: REFS issues (server lockups, high CPU, high RAM)

Post by Neil Flanagan »

Cullan wrote:Seems to still be Refs.sys10.0.14393.203511-Jan-18 :(
We weren't expecting a new REFS fix with the 13 February Cumulative Update (security patch day). It was supposed to be in the later February Cumulative Update on 20 February or later, the one you have to manually trigger or get from the Microsoft Catalog.
Gostev
Chief Product Officer
Posts: 31428
Liked: 6633 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: REFS issues (server lockups, high CPU, high RAM)

Post by Gostev » 2 people like this post

Neil is correct, these aren't the droids you're looking for. The fix is expected to be released next week (2/20) as part of a separate KB.
Mike Resseler
Product Manager
Posts: 8042
Liked: 1262 times
Joined: Feb 08, 2013 3:08 pm
Full Name: Mike Resseler
Location: Belgium
Contact:

Re: REFS issues (server lockups, high CPU, high RAM)

Post by Mike Resseler »

That is not so good news :-(

So after installing the CU, to be able to use the "fixed" ReFS driver you need to reformat. That is painful
EricJ
Influencer
Posts: 20
Liked: 4 times
Joined: Jan 12, 2017 7:06 pm
Contact:

Re: REFS issues (server lockups, high CPU, high RAM)

Post by EricJ »

Mike Resseler wrote:That is not so good news :-(

So after installing the CU, to be able to use the "fixed" ReFS driver you need to reformat. That is painful
I must have missed something - but where did we find out this to be true? Haven't some users been beta testing it with success? I don't remember any of the mentioning a reformat.

Thanks,
Eric
Mildur
Product Manager
Posts: 8481
Liked: 2203 times
Joined: May 13, 2017 4:51 pm
Full Name: Fabian K.
Location: Switzerland
Contact:

Re: REFS issues (server lockups, high CPU, high RAM)

Post by Mildur »

Mike Resseler wrote:That is not so good news :-(

So after installing the CU, to be able to use the "fixed" ReFS driver you need to reformat. That is painful
He's telling about raw disks.
firewall68 wrote: According MS the update will fix the issue but you have to format the disk again with REFS. therfor all data on the RAW Device are lost.
the only way is to buy a software which can recover REFS.
For me, this is a normal behaviour. RAW disks are destroyed data. No patch from microsoft can rescue this data. :(

For Installations with a working refs Partition, Patch should be ok without reformat. I hope, I'm not wrong ;-)
Product Management Analyst @ Veeam Software
DaveWatkins
Veteran
Posts: 370
Liked: 97 times
Joined: Dec 13, 2015 11:33 pm
Contact:

Re: REFS issues (server lockups, high CPU, high RAM)

Post by DaveWatkins »

Mike Resseler wrote:Keep us informed gentleman. I am not in the possibility to install updates on a workload the next couple of days but I am eager to know if it is included. From the list of files, it does indeed seem that ReFS has not received an update compared to the January CU. :-(
Based on comments earlier, it was never going to as far as I can tell. The updates was possibly going to be released later in Feb as a standalone update, and otherwise included in the March rollup
Zew
Veteran
Posts: 365
Liked: 80 times
Joined: Mar 17, 2015 9:50 pm
Full Name: Aemilianus Kehler
Contact:

Re: REFS issues (server lockups, high CPU, high RAM)

Post by Zew »

Just asked about proper ReFS setup, and then I found this lengthy painful blog post. Waiting for the Feb 20 Optional Update before I even build my ReFS storage.
Mike Resseler
Product Manager
Posts: 8042
Liked: 1262 times
Joined: Feb 08, 2013 3:08 pm
Full Name: Mike Resseler
Location: Belgium
Contact:

Re: REFS issues (server lockups, high CPU, high RAM)

Post by Mike Resseler »

Hi Zew,

Yes. Best to wait. And I hope indeed that Neil and Gostev are correct (and now Gostev will come back saying to me he is always right ;-)) and that the manual one is the droid we are looking for...
Gostev
Chief Product Officer
Posts: 31428
Liked: 6633 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: REFS issues (server lockups, high CPU, high RAM)

Post by Gostev »

Mike Resseler wrote:So after installing the CU, to be able to use the "fixed" ReFS driver you need to reformat. That is painful
First time I hear. Who told you this?
kubimike
Veteran
Posts: 373
Liked: 41 times
Joined: Feb 03, 2017 2:34 pm
Full Name: MikeO
Contact:

Re: REFS issues (server lockups, high CPU, high RAM)

Post by kubimike » 2 people like this post

Hi I didn’t need to reformat to use Beta v2. Indeed RAW means raw. I’ve been up for about a month no freezes active fills kick off daily. Couldn’t he happier! Now to get Veeam to address an issue where my hotadd box isn’t releasing disks sometime. I’ll start a ticket :mrgreen:
sarbaron
Service Provider
Posts: 9
Liked: 2 times
Joined: Apr 29, 2013 8:11 pm
Full Name: Adam Baron
Location: Rochdale, UK
Contact:

Re: REFS issues (server lockups, high CPU, high RAM)

Post by sarbaron »

Gostev wrote:Neil is correct, these aren't the droids you're looking for. The fix is expected to be released next week (2/20) as part of a separate KB.
So, here I am, all eager, looking for the patch, while all my failed backup jobs are stopping and snapshots removing before I reboot my repo again... and I can't find anything....

Please tell me that it will be today, not sure how many more times I can go through this! :roll:
bigyellocup
Lurker
Posts: 2
Liked: never
Joined: Feb 20, 2018 3:22 pm
Contact:

Re: REFS issues (server lockups, high CPU, high RAM)

Post by bigyellocup »

Gostev
Chief Product Officer
Posts: 31428
Liked: 6633 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: REFS issues (server lockups, high CPU, high RAM)

Post by Gostev » 1 person likes this post

Remember Seattle is almost 1 day back with their time zone ;)
sarbaron
Service Provider
Posts: 9
Liked: 2 times
Joined: Apr 29, 2013 8:11 pm
Full Name: Adam Baron
Location: Rochdale, UK
Contact:

Re: REFS issues (server lockups, high CPU, high RAM)

Post by sarbaron »

Gostev wrote:Remember Seattle is almost 1 day back with their time zone ;)
It must just feel like an eternity! :lol:
BartP
Veeam Software
Posts: 230
Liked: 62 times
Joined: Aug 31, 2015 8:24 am
Full Name: Bart Pellegrino
Location: Netherlands
Contact:

Re: REFS issues (server lockups, high CPU, high RAM)

Post by BartP » 3 people like this post

Wakey-wakey Seattle!
*shameless post to keep myself notified for updates about the update :roll: *
Bart Pellegrino,
Technical Account Manager - EMEA
NightBird
Expert
Posts: 242
Liked: 57 times
Joined: Apr 28, 2009 8:33 am
Location: Strasbourg, FRANCE
Contact:

Re: REFS issues (server lockups, high CPU, high RAM)

Post by NightBird »

No update at the moment :'(
Gostev
Chief Product Officer
Posts: 31428
Liked: 6633 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: REFS issues (server lockups, high CPU, high RAM)

Post by Gostev » 4 people like this post

I checked with Microsoft folks and it appears that the KB has been delayed by a few days because of some internal issues with the update. I guess we have to wait a few more days!
bib_ak
Enthusiast
Posts: 33
Liked: 1 time
Joined: Jan 31, 2010 12:57 pm
Contact:

Re: REFS issues (server lockups, high CPU, high RAM)

Post by bib_ak »

I have two installations on hold waiting for this KB - it's OK if it's delayed - as long as it resolves some of the issues :)
pinkerton
Enthusiast
Posts: 82
Liked: 4 times
Joined: Sep 29, 2011 9:57 am
Contact:

Re: REFS issues (server lockups, high CPU, high RAM)

Post by pinkerton »

Just got off a phone call with one some Veeam technican (3rd party) discussing WAN acceleration. At the end we talked about ReFS issues and he told me that Veeam now recommends to use 4K formatting for ReFS volumes instead of 64K. Is that true? We implemented ReFS a year ago and IIRC the recommendation was 64K back then. We also have problems with our ReFS setup and are waiting for the hotfix. However, if 64K is also cause of the issue we are going to change to 4K as well.
bonzovt
Enthusiast
Posts: 42
Liked: 1 time
Joined: Sep 09, 2016 6:15 pm
Full Name: Adam Fisher
Contact:

Re: REFS issues (server lockups, high CPU, high RAM)

Post by bonzovt »

pinkerton wrote:Just got off a phone call with one some Veeam technican (3rd party) discussing WAN acceleration. At the end we talked about ReFS issues and he told me that Veeam now recommends to use 4K formatting for ReFS volumes instead of 64K. Is that true? We implemented ReFS a year ago and IIRC the recommendation was 64K back then. We also have problems with our ReFS setup and are waiting for the hotfix. However, if 64K is also cause of the issue we are going to change to 4K as well.
I feel like if a change from 64k to 4k was now the preferred option, Veeam would have communicated that somehow? All the guides I see still explicitly show using 64k during the format, and granted the guides are a little bit older but I'd think such a large shift would have been announced at some point.
BartP
Veeam Software
Posts: 230
Liked: 62 times
Joined: Aug 31, 2015 8:24 am
Full Name: Bart Pellegrino
Location: Netherlands
Contact:

Re: REFS issues (server lockups, high CPU, high RAM)

Post by BartP » 2 people like this post

The reason why 64k block sizes are usually advised is due the speed increase with backup merges.
It however does mean you consume about 10% more space.

source: https://www.virtualtothecore.com/en/ref ... kb-or-4kb/

As extra, when ReFS was introduced and the problems started arising with the filesystem. The problems occurred less frequently when 64k block where used instead of 4k.
This makes sense because less blocks to merge = less metadata to rewrite = less cpu/ram needed.
Bart Pellegrino,
Technical Account Manager - EMEA
bonzovt
Enthusiast
Posts: 42
Liked: 1 time
Joined: Sep 09, 2016 6:15 pm
Full Name: Adam Fisher
Contact:

Re: REFS issues (server lockups, high CPU, high RAM)

Post by bonzovt »

thanks bart, that makes a ton of sense!
bryand82487
Influencer
Posts: 14
Liked: 2 times
Joined: Dec 13, 2017 3:31 pm
Full Name: Bryan Dennison
Contact:

Re: REFS issues (server lockups, high CPU, high RAM)

Post by bryand82487 » 2 people like this post

This was released today. Is this the update?

2018-02 Cumulative Update for Windows Server 2016 for x64-based Systems (KB4077525)

Improves ReFS performance by more thoroughly unmapping multiple views of a file. See KB4090104 for additional tunable registry parameters to address large ReFS metadata streams.

Improves ReFS performance by removing idle containers from its hash table.
jayscarff
Service Provider
Posts: 114
Liked: 12 times
Joined: Nov 15, 2016 6:56 pm
Location: Cayman Islands
Contact:

Re: REFS issues (server lockups, high CPU, high RAM)

Post by jayscarff »

Thanks, currently downloading to test!
Jason
VMCE
bryand82487
Influencer
Posts: 14
Liked: 2 times
Joined: Dec 13, 2017 3:31 pm
Full Name: Bryan Dennison
Contact:

Re: REFS issues (server lockups, high CPU, high RAM)

Post by bryand82487 »

Good Luck! Let me know how it works out for you. Can you tell me what the REFS driver file version is after you install the update please?
SteelContainer
Service Provider
Posts: 146
Liked: 21 times
Joined: May 21, 2014 8:47 am
Location: New Zealand
Contact:

Re: REFS issues (server lockups, high CPU, high RAM)

Post by SteelContainer »

Refs.sys
10.0.14393.2097
12-Feb-18
jayscarff
Service Provider
Posts: 114
Liked: 12 times
Joined: Nov 15, 2016 6:56 pm
Location: Cayman Islands
Contact:

Re: REFS issues (server lockups, high CPU, high RAM)

Post by jayscarff »

Same version here! Testing some backup jobs currently, copy jobs kick off in the morning which will hit the repository with multiple GFS at the same time!
Jason
VMCE
tsightler
VP, Product Management
Posts: 6003
Liked: 2840 times
Joined: Jun 05, 2009 12:57 pm
Full Name: Tom Sightler
Contact:

Re: REFS issues (server lockups, high CPU, high RAM)

Post by tsightler » 1 person likes this post

pinkerton wrote:Just got off a phone call with one some Veeam technican (3rd party) discussing WAN acceleration. At the end we talked about ReFS issues and he told me that Veeam now recommends to use 4K formatting for ReFS volumes instead of 64K. Is that true? We implemented ReFS a year ago and IIRC the recommendation was 64K back then. We also have problems with our ReFS setup and are waiting for the hotfix. However, if 64K is also cause of the issue we are going to change to 4K as well.
I'm not sure why someone from support would say that. We recommend 64K not just due to performance of merges/synthetics, although that is a factor, but primarily due to the fact that using 4K clusters leads to the problems documented in this forum thread much more often and in worse ways, even with much smaller datasets. For example, with 4K clusters I've personally seen ReFS lockups on repositories as small as a few TBs in size, and common (nightly), lockups with 100TB repositories when they contained <10TB of data. On the other hand, with 64K clusters, these smaller repos were stable as long as they had enough memory. Of course, 64K clusters did not eliminate all of the issues, as this thread well documents, it just made them less likely.

If the fixes for ReFS in the latest patches fix all of the resource consumption and performance issues even with repositories at scale, even when used with 4K clusters, it might be possible for us to change this recommendation in the future, but that will only be proven over time in the field as we see the results. Initial results from customers that I know were using 4K clusters, and were testing the patches, seems promising, but it will take some time and more results before we really know.

For now, let's just all test with these new ReFS updates and hope that we may finally be near the end of this long road with ReFS stability and please post your results here. It's a great technology when it works, but it's been a bumpy ride to this point.
jayscarff
Service Provider
Posts: 114
Liked: 12 times
Joined: Nov 15, 2016 6:56 pm
Location: Cayman Islands
Contact:

Re: REFS issues (server lockups, high CPU, high RAM)

Post by jayscarff » 1 person likes this post

jayscarff wrote:Same version here! Testing some backup jobs currently, copy jobs kick off in the morning which will hit the repository with multiple GFS at the same time!
First job finished, pre patch the fast clone took 2 hours, post patch, 25 minutes. :D
Jason
VMCE
Locked

Who is online

Users browsing this forum: Bing [Bot], legil.miguel and 142 guests