REFS issues (server lockups, high CPU, high RAM)

Availability for the Always-On Enterprise

Re: REFS issues (server lockups, high CPU, high RAM)

Veeam Logoby jozne » Fri Mar 09, 2018 7:49 am 1 person likes this post

Gostev wrote:Thanks for the update, Jimmy. Very happy to hear the new code is holding up well and there's no performance degradation over time as before. And since "no news is good news" in this case, I assume everyone else is seeing the same!

I guess you did not read my posts on veeam-backup-replication-f2/refs-4k-horror-story-t40629-870.html then ^^
We still have some lockups and Offsite backups to 3rd party cloud take 300h for 2TB backup , using fast clone...
jozne
Influencer
 
Posts: 16
Liked: 6 times
Joined: Wed Apr 18, 2012 6:55 pm
Full Name: Jari Haikonen

Re: REFS issues (server lockups, high CPU, high RAM)

Veeam Logoby Gostev » Fri Mar 09, 2018 7:25 pm

Did your service provider install the patch on their ReFS repository server?

As for local backups, you should investigate your particular case with Microsoft Support to see if the patch has installed properly. I did see your and a couple of other reports similar to yours above, but so far reports from the majority of users have been positive.

Thanks!
Gostev
Veeam Software
 
Posts: 22078
Liked: 2569 times
Joined: Sun Jan 01, 2006 1:01 am
Location: Baar, Switzerland

Re: REFS issues (server lockups, high CPU, high RAM)

Veeam Logoby mweissen13 » Mon Mar 12, 2018 9:14 am 1 person likes this post

Hi!
As we know by know Microsoft pulled KB4077525 which included the long-awaited ReFS fixes, at least for Windows Server 2016. Not because of ReFS but because of other, unrelated problems. I installed it on a few of our Veeam Repository Servers before it was pulled with great results so far. Does anyone know if those ReFS fixes will be included in the regular patch Tuesday fixes coming up this week or will we have to wait even longer?
mweissen13
Service Provider
 
Posts: 11
Liked: 3 times
Joined: Thu Dec 28, 2017 3:22 pm
Full Name: Michael Weissenbacher

Re: REFS issues (server lockups, high CPU, high RAM)

Veeam Logoby jshiflet » Mon Mar 12, 2018 4:01 pm

JimmyO wrote:Update: Almost 1 week running with KB4077525 now and it´s still looking good. No signs of decreasing performance. Memory utilization is approx. 40% lower than before but CPU load approx. 50%higher (as expected I guess).

I just wish MS released this before I spent 1 month reverting my main 500TB backup server to NTFS. Here we go again...

I'm a bit curious here...how much RAM do you have on your system with the 500TB repository? We have that same size in NTFS as well, but am looking at switching to ReFS now that the patch is out because of it taking several days for synthetic fulls to complete on NTFS and I'm curious as to your RAM specs since based on the Microsoft sizing recommendation (what was it like 4GB for every 1TB, which would mean needing 2TB of RAM on my backup server).
--
jason shiflet
systems enginner
jshiflet
Service Provider
 
Posts: 13
Liked: never
Joined: Mon Sep 17, 2012 6:35 pm
Location: Dallas, TX
Full Name: Jason Shiflet

Re: REFS issues (server lockups, high CPU, high RAM)

Veeam Logoby jozne » Tue Mar 13, 2018 10:24 am

Gostev wrote:Did your service provider install the patch on their ReFS repository server?

As for local backups, you should investigate your particular case with Microsoft Support to see if the patch has installed properly. I did see your and a couple of other reports similar to yours above, but so far reports from the majority of users have been positive.

Thanks!

yup they did. And we already have case open with MS and Veeam :)
but just so that others know the situation as well ^^
jozne
Influencer
 
Posts: 16
Liked: 6 times
Joined: Wed Apr 18, 2012 6:55 pm
Full Name: Jari Haikonen

Re: REFS issues (server lockups, high CPU, high RAM)

Veeam Logoby jozne » Tue Mar 13, 2018 10:25 am 1 person likes this post

mweissen13 wrote:Hi!
As we know by know Microsoft pulled KB4077525 which included the long-awaited ReFS fixes, at least for Windows Server 2016. Not because of ReFS but because of other, unrelated problems. I installed it on a few of our Veeam Repository Servers before it was pulled with great results so far. Does anyone know if those ReFS fixes will be included in the regular patch Tuesday fixes coming up this week or will we have to wait even longer?

Got this from MS support

"The fix for the ReFS issue released week C of February and will be included in week two updates of March."
"fix for memory pressure similar to that seen in your last dumpfile is being released week C of April."
jozne
Influencer
 
Posts: 16
Liked: 6 times
Joined: Wed Apr 18, 2012 6:55 pm
Full Name: Jari Haikonen

Re: REFS issues (server lockups, high CPU, high RAM)

Veeam Logoby Gostev » Wed Mar 14, 2018 11:20 am

Jari, thanks so much for sharing this response from Microsoft Support - this will save me some back and forth finding these dates out through my sources.

And it looks like I was right in my thinking that those still having issues after installing the patch have them due to lack of RAM. But you did share some good news - I knew the memory pressure fix was "in the works" but didn't actually expect it to become available so soon!
Gostev
Veeam Software
 
Posts: 22078
Liked: 2569 times
Joined: Sun Jan 01, 2006 1:01 am
Location: Baar, Switzerland

Re: REFS issues (server lockups, high CPU, high RAM)

Veeam Logoby SBarrett847 » Wed Mar 14, 2018 11:24 am

What is the memory pressure issue? I still have Hard Lockups/crashes with Dynamic Ram on the Repo server after the Patch, it this expected to address that?
SBarrett847
Service Provider
 
Posts: 227
Liked: 35 times
Joined: Tue Feb 02, 2016 5:02 pm
Full Name: Stephen Barrett

Re: REFS issues (server lockups, high CPU, high RAM)

Veeam Logoby Gostev » Wed Mar 14, 2018 4:09 pm 1 person likes this post

See this KB article > https://support.microsoft.com/en-us/hel ... windows-10

Basically, there are some tweaks you can try that may improve the situation with the existing ReFS memory management code. And, it sounds like the proper fix (improved memory management) is coming in April.
Gostev
Veeam Software
 
Posts: 22078
Liked: 2569 times
Joined: Sun Jan 01, 2006 1:01 am
Location: Baar, Switzerland

Re: REFS issues (server lockups, high CPU, high RAM)

Veeam Logoby enoch » Thu Mar 15, 2018 10:44 am

MANY information on this subject. As I read, the current "new" fix for REFS from Microsoft is no longer possible to download because of other none-REFS issues in that patch?

Possible to get someone or when is it possible to get new patch that include the REFS fixes? (need is badly)
enoch
Service Provider
 
Posts: 134
Liked: 9 times
Joined: Sat Jun 29, 2013 12:14 pm
Full Name: Peter Enoch

Re: REFS issues (server lockups, high CPU, high RAM)

Veeam Logoby Gostev » Thu Mar 15, 2018 1:28 pm

See post by jozne earlier on this page for the current estimations from Microsoft Support.
Gostev
Veeam Software
 
Posts: 22078
Liked: 2569 times
Joined: Sun Jan 01, 2006 1:01 am
Location: Baar, Switzerland

Re: REFS issues (server lockups, high CPU, high RAM)

Veeam Logoby enoch » Thu Mar 15, 2018 1:59 pm

So week C of April = 3. week of April?
enoch
Service Provider
 
Posts: 134
Liked: 9 times
Joined: Sat Jun 29, 2013 12:14 pm
Full Name: Peter Enoch

Re: REFS issues (server lockups, high CPU, high RAM)

Veeam Logoby jamesharper-bsol » Thu Mar 15, 2018 4:16 pm 3 people like this post

KB4088787 is now out, it supersedes KB4077525 and contains ReFS.sys 10.0.14393.2097
Also, it doesn't need the AV QualityCompat registry key.
jamesharper-bsol
Novice
 
Posts: 4
Liked: 4 times
Joined: Mon Jan 16, 2012 10:30 am
Full Name: James Harper

Re: REFS issues (server lockups, high CPU, high RAM)

Veeam Logoby JimmyO » Wed Mar 21, 2018 7:21 am

Has anyone been able to compare 4k and 64k yet? Most likely 64k is faster, but it would be nice to see some figures...
JimmyO
Enthusiast
 
Posts: 49
Liked: 7 times
Joined: Sun Apr 27, 2014 8:19 pm

Re: REFS issues (server lockups, high CPU, high RAM)

Veeam Logoby Iain_Green » Wed Mar 21, 2018 2:29 pm

Hi,

Sorry if this has been asked and answered in this now very long thread but....

Is REFS now usable once again, and do I require anything special to ensure it works?

Do I need 64k block size?
Do I need certain updates?
Do I need any reg changes?
Anything else I may have missed?

I have a new install for a customer and would like to use REFS if it is now working.

Cheers
Many thanks

Iain Green
Iain_Green
Service Provider
 
Posts: 111
Liked: 7 times
Joined: Fri Dec 05, 2014 2:13 pm
Full Name: Iain Green

PreviousNext

Return to Veeam Backup & Replication



Who is online

Users browsing this forum: Google [Bot] and 12 guests