-
- Service Provider
- Posts: 246
- Liked: 28 times
- Joined: Dec 14, 2015 8:20 pm
- Full Name: Mehmet Istanbullu
- Location: Türkiye
- Contact:
Re: REFS issues (server lockups, high CPU, high RAM)
What is the best using case ReFS?
Physical Windows 2016 installing or Server 2016 on Vsphere 6.5?
Some people have issue about raid card drivers on psychical 2016 installation.
Physical Windows 2016 installing or Server 2016 on Vsphere 6.5?
Some people have issue about raid card drivers on psychical 2016 installation.
-
- Veteran
- Posts: 527
- Liked: 142 times
- Joined: Aug 20, 2015 9:30 pm
- Contact:
Re: REFS issues (server lockups, high CPU, high RAM)
On Windows Server, the mitigations for meltdown and spectre are not enabled by default. You have to create a registry key, and also for spectre it only works if you have updated firmware on your device. Also remember the specter/meltdown patch has been included in the security patches since January, it's not like the ReFS fix is the first time you're getting those patches.operations wrote:From what I understand the rollup that contains the fix for ReFS also contains patches for meltdown and spectre, has anyone tried to extract the patch or updates and just install the ReFS fix so as to avoid the melt/spec patches ?
-
- Veteran
- Posts: 391
- Liked: 56 times
- Joined: Feb 03, 2017 2:34 pm
- Full Name: MikeO
- Contact:
Re: REFS issues (server lockups, high CPU, high RAM)
/Sigh , what a mess . Im staying on the second beta until this is sorted out. Everything works for me ATM. Maybe you can wave your magic wand and just get the driver only.Gostev wrote:Actually, this is an interesting thought. The pre-release driver was provided by itself, and I know for the fact that it helped every single person who installed it... but now we're installing a huge patch to get the same driver, so it is certainly a possibility something else in this patch is misbehaving. Although this is not very likely, because in that case it would misbehave for everyone. Usually, when experiences are so dramatically different, this indicates some environment-specific trigger such as computer configuration (lack of system resources) or 3rd party software conflict (where antivirus is the primary suspect). I hope those still having issues after installing the patch are able to investigate them with Microsoft.
-
- Service Provider
- Posts: 12
- Liked: never
- Joined: Nov 25, 2017 6:49 pm
- Full Name: operations
- Contact:
Re: REFS issues (server lockups, high CPU, high RAM)
Fully understood, and personally I have been avoiding installing melt/spec patches by not using the rollups just individual patches due to the perf hit my backup servers took when I did install it, so if people are installing the all the patches since Jan then they will need to have compensated for the reduction in IO performance.nmdange wrote: On Windows Server, the mitigations for meltdown and spectre are not enabled by default. You have to create a registry key, and also for spectre it only works if you have updated firmware on your device. Also remember the specter/meltdown patch has been included in the security patches since January, it's not like the ReFS fix is the first time you're getting those patches.
-
- Enthusiast
- Posts: 55
- Liked: 9 times
- Joined: Apr 27, 2014 8:19 pm
- Contact:
Re: REFS issues (server lockups, high CPU, high RAM)
Update: Almost 1 week running with KB4077525 now and it´s still looking good. No signs of decreasing performance. Memory utilization is approx. 40% lower than before but CPU load approx. 50%higher (as expected I guess).JimmyO wrote:Installed KB4077525 on one of my smaller backup servers with 58TB volume (backup file: 6TB full, 200GB increment) and merge time went from 8h to 1h after first daily backup.
Of course - the server has just been restarted, but first impressions are good
I just wish MS released this before I spent 1 month reverting my main 500TB backup server to NTFS. Here we go again...
-
- Chief Product Officer
- Posts: 31606
- Liked: 7095 times
- Joined: Jan 01, 2006 1:01 am
- Location: Baar, Switzerland
- Contact:
Re: REFS issues (server lockups, high CPU, high RAM)
Thanks for the update, Jimmy. Very happy to hear the new code is holding up well and there's no performance degradation over time as before. And since "no news is good news" in this case, I assume everyone else is seeing the same!
-
- Service Provider
- Posts: 14
- Liked: 1 time
- Joined: May 19, 2016 12:52 am
- Full Name: Leith Magon
- Contact:
Re: REFS issues (server lockups, high CPU, high RAM)
So when 2018-02 first came out I installed it on a bunch of servers using REFS repos (Veeam & *cough* dpm).
Haven't had any issues that I can see and speeds improved for sure. Fast forward to yesterday and I've been unable to locate this patch on the update catalogue or windows update anymore?
I have the msu file downloaded from previous installations but Im hesitant to install it if it's been pulled? No-one except this thread has mentioned its missing.
Haven't had any issues that I can see and speeds improved for sure. Fast forward to yesterday and I've been unable to locate this patch on the update catalogue or windows update anymore?
I have the msu file downloaded from previous installations but Im hesitant to install it if it's been pulled? No-one except this thread has mentioned its missing.
-
- Chief Product Officer
- Posts: 31606
- Liked: 7095 times
- Joined: Jan 01, 2006 1:01 am
- Location: Baar, Switzerland
- Contact:
Re: REFS issues (server lockups, high CPU, high RAM)
I covered this in the forum digest this past Sunday. In short, Microsoft pulled the patch due to ADFS corruption bug it contains. ReFS is just one of many fixes this patch brings, and one of them appeared to misbehave. But, if you don't run ADFS configuration database on your backup repository server, you're don't need to worry about installing the patch.
-
- Service Provider
- Posts: 12
- Liked: never
- Joined: Nov 25, 2017 6:49 pm
- Full Name: operations
- Contact:
Re: REFS issues (server lockups, high CPU, high RAM)
Would be nice if they would release the patches independently, or give the ReFS patch to veeam as a hotfix to rollout.
-
- Enthusiast
- Posts: 26
- Liked: 12 times
- Joined: Jan 30, 2017 7:42 pm
- Full Name: Sam
- Contact:
Re: REFS issues (server lockups, high CPU, high RAM)
I've checked in every so often during the past year of these issues.
It looks like this patch had a major affect in getting things to work for me. Despite having 4k block size, I'm no longer crashing and my jobs are completing in a fraction of the time before. I'd see merges that would last 8-12 hours, now they run 20-30 minutes. It's really like a new system! Hopefully things continue to get even better!
It looks like this patch had a major affect in getting things to work for me. Despite having 4k block size, I'm no longer crashing and my jobs are completing in a fraction of the time before. I'd see merges that would last 8-12 hours, now they run 20-30 minutes. It's really like a new system! Hopefully things continue to get even better!
-
- Service Provider
- Posts: 60
- Liked: 6 times
- Joined: Dec 06, 2010 7:51 pm
- Full Name: Eric Thompson
- Location: Boston, MA
- Contact:
Re: REFS issues (server lockups, high CPU, high RAM)
Since applying this patch my Windows 2016 File Server cluster has been crashing and taking the SoBR extents with it when trying to evacuate a small extent. Anyone else running into that? Going to pull the patch off and see if that is the only culprit.
-
- Chief Product Officer
- Posts: 31606
- Liked: 7095 times
- Joined: Jan 01, 2006 1:01 am
- Location: Baar, Switzerland
- Contact:
Re: REFS issues (server lockups, high CPU, high RAM)
That's some precious feedback considering there are few people running with 4KB blocks, so we have very little data on such deployments. If you could share, I would be very interested to know the amount of RAM in your backup repository, ReFS volume capacity and utilization, and the largest backup file size. Thanks!Cicadymn wrote:It looks like this patch had a major affect in getting things to work for me. Despite having 4k block size, I'm no longer crashing and my jobs are completing in a fraction of the time before. I'd see merges that would last 8-12 hours, now they run 20-30 minutes. It's really like a new system! Hopefully things continue to get even better!
-
- Enthusiast
- Posts: 26
- Liked: 12 times
- Joined: Jan 30, 2017 7:42 pm
- Full Name: Sam
- Contact:
Re: REFS issues (server lockups, high CPU, high RAM)
Yeah, we accidentally ended up with 4K size and it's been a challenge. But I'm happy to say that it's working much better.Gostev wrote:That's some precious feedback considering there are few people running with 4KB blocks, so we have very little data on such deployments. If you could share, I would be very interested to know the amount of RAM in your backup repository, ReFS volume capacity and utilization, and the largest backup file size. Thanks!
So for your questions:
- RAM: 48GB
We've got three repositories at 35TB a piece, using 28, 26.5, and 5.7TB respectively
Our largest backup file is 4.7TB, though the largest amount of data processed in Veeam is listed at 6.5TB.
The merge on the largest job lasted 26 minutes.
-
- Chief Product Officer
- Posts: 31606
- Liked: 7095 times
- Joined: Jan 01, 2006 1:01 am
- Location: Baar, Switzerland
- Contact:
Re: REFS issues (server lockups, high CPU, high RAM)
Thanks so much. Sounds like we should remove that 64KB cluster size recommendation from backup repository wizard down the road, at least for smaller ReFS volumes...
-
- Service Provider
- Posts: 12
- Liked: never
- Joined: Nov 25, 2017 6:49 pm
- Full Name: operations
- Contact:
Re: REFS issues (server lockups, high CPU, high RAM)
sounds like its time to do some side by side performance tests 4k vs 64k on the same storage ?
-
- Influencer
- Posts: 17
- Liked: 7 times
- Joined: Apr 18, 2012 6:55 pm
- Full Name: Jari Haikonen
- Contact:
Re: REFS issues (server lockups, high CPU, high RAM)
I guess you did not read my posts on https://forums.veeam.com/veeam-backup-r ... 9-870.html then ^^Gostev wrote:Thanks for the update, Jimmy. Very happy to hear the new code is holding up well and there's no performance degradation over time as before. And since "no news is good news" in this case, I assume everyone else is seeing the same!
We still have some lockups and Offsite backups to 3rd party cloud take 300h for 2TB backup , using fast clone...
-
- Chief Product Officer
- Posts: 31606
- Liked: 7095 times
- Joined: Jan 01, 2006 1:01 am
- Location: Baar, Switzerland
- Contact:
Re: REFS issues (server lockups, high CPU, high RAM)
Did your service provider install the patch on their ReFS repository server?
As for local backups, you should investigate your particular case with Microsoft Support to see if the patch has installed properly. I did see your and a couple of other reports similar to yours above, but so far reports from the majority of users have been positive.
Thanks!
As for local backups, you should investigate your particular case with Microsoft Support to see if the patch has installed properly. I did see your and a couple of other reports similar to yours above, but so far reports from the majority of users have been positive.
Thanks!
-
- Enthusiast
- Posts: 93
- Liked: 54 times
- Joined: Dec 28, 2017 3:22 pm
- Full Name: Michael Weissenbacher
- Contact:
Re: REFS issues (server lockups, high CPU, high RAM)
Hi!
As we know by know Microsoft pulled KB4077525 which included the long-awaited ReFS fixes, at least for Windows Server 2016. Not because of ReFS but because of other, unrelated problems. I installed it on a few of our Veeam Repository Servers before it was pulled with great results so far. Does anyone know if those ReFS fixes will be included in the regular patch Tuesday fixes coming up this week or will we have to wait even longer?
As we know by know Microsoft pulled KB4077525 which included the long-awaited ReFS fixes, at least for Windows Server 2016. Not because of ReFS but because of other, unrelated problems. I installed it on a few of our Veeam Repository Servers before it was pulled with great results so far. Does anyone know if those ReFS fixes will be included in the regular patch Tuesday fixes coming up this week or will we have to wait even longer?
-
- Influencer
- Posts: 13
- Liked: never
- Joined: Sep 17, 2012 6:35 pm
- Full Name: Jason Shiflet
- Location: Dallas, TX
- Contact:
Re: REFS issues (server lockups, high CPU, high RAM)
I'm a bit curious here...how much RAM do you have on your system with the 500TB repository? We have that same size in NTFS as well, but am looking at switching to ReFS now that the patch is out because of it taking several days for synthetic fulls to complete on NTFS and I'm curious as to your RAM specs since based on the Microsoft sizing recommendation (what was it like 4GB for every 1TB, which would mean needing 2TB of RAM on my backup server).JimmyO wrote:Update: Almost 1 week running with KB4077525 now and it´s still looking good. No signs of decreasing performance. Memory utilization is approx. 40% lower than before but CPU load approx. 50%higher (as expected I guess).
I just wish MS released this before I spent 1 month reverting my main 500TB backup server to NTFS. Here we go again...
--
jason shiflet
systems enginner
jason shiflet
systems enginner
-
- Influencer
- Posts: 17
- Liked: 7 times
- Joined: Apr 18, 2012 6:55 pm
- Full Name: Jari Haikonen
- Contact:
Re: REFS issues (server lockups, high CPU, high RAM)
yup they did. And we already have case open with MS and VeeamGostev wrote:Did your service provider install the patch on their ReFS repository server?
As for local backups, you should investigate your particular case with Microsoft Support to see if the patch has installed properly. I did see your and a couple of other reports similar to yours above, but so far reports from the majority of users have been positive.
Thanks!
but just so that others know the situation as well ^^
-
- Influencer
- Posts: 17
- Liked: 7 times
- Joined: Apr 18, 2012 6:55 pm
- Full Name: Jari Haikonen
- Contact:
Re: REFS issues (server lockups, high CPU, high RAM)
Got this from MS supportmweissen13 wrote:Hi!
As we know by know Microsoft pulled KB4077525 which included the long-awaited ReFS fixes, at least for Windows Server 2016. Not because of ReFS but because of other, unrelated problems. I installed it on a few of our Veeam Repository Servers before it was pulled with great results so far. Does anyone know if those ReFS fixes will be included in the regular patch Tuesday fixes coming up this week or will we have to wait even longer?
"The fix for the ReFS issue released week C of February and will be included in week two updates of March."
"fix for memory pressure similar to that seen in your last dumpfile is being released week C of April."
-
- Chief Product Officer
- Posts: 31606
- Liked: 7095 times
- Joined: Jan 01, 2006 1:01 am
- Location: Baar, Switzerland
- Contact:
Re: REFS issues (server lockups, high CPU, high RAM)
Jari, thanks so much for sharing this response from Microsoft Support - this will save me some back and forth finding these dates out through my sources.
And it looks like I was right in my thinking that those still having issues after installing the patch have them due to lack of RAM. But you did share some good news - I knew the memory pressure fix was "in the works" but didn't actually expect it to become available so soon!
And it looks like I was right in my thinking that those still having issues after installing the patch have them due to lack of RAM. But you did share some good news - I knew the memory pressure fix was "in the works" but didn't actually expect it to become available so soon!
-
- Service Provider
- Posts: 315
- Liked: 41 times
- Joined: Feb 02, 2016 5:02 pm
- Full Name: Stephen Barrett
- Contact:
Re: REFS issues (server lockups, high CPU, high RAM)
What is the memory pressure issue? I still have Hard Lockups/crashes with Dynamic Ram on the Repo server after the Patch, it this expected to address that?
-
- Chief Product Officer
- Posts: 31606
- Liked: 7095 times
- Joined: Jan 01, 2006 1:01 am
- Location: Baar, Switzerland
- Contact:
Re: REFS issues (server lockups, high CPU, high RAM)
See this KB article > https://support.microsoft.com/en-us/hel ... windows-10
Basically, there are some tweaks you can try that may improve the situation with the existing ReFS memory management code. And, it sounds like the proper fix (improved memory management) is coming in April.
Basically, there are some tweaks you can try that may improve the situation with the existing ReFS memory management code. And, it sounds like the proper fix (improved memory management) is coming in April.
-
- Service Provider
- Posts: 171
- Liked: 13 times
- Joined: Jun 29, 2013 12:14 pm
- Full Name: Peter Enoch
- Contact:
Re: REFS issues (server lockups, high CPU, high RAM)
MANY information on this subject. As I read, the current "new" fix for REFS from Microsoft is no longer possible to download because of other none-REFS issues in that patch?
Possible to get someone or when is it possible to get new patch that include the REFS fixes? (need is badly)
Possible to get someone or when is it possible to get new patch that include the REFS fixes? (need is badly)
-
- Chief Product Officer
- Posts: 31606
- Liked: 7095 times
- Joined: Jan 01, 2006 1:01 am
- Location: Baar, Switzerland
- Contact:
Re: REFS issues (server lockups, high CPU, high RAM)
See post by jozne earlier on this page for the current estimations from Microsoft Support.
-
- Service Provider
- Posts: 171
- Liked: 13 times
- Joined: Jun 29, 2013 12:14 pm
- Full Name: Peter Enoch
- Contact:
Re: REFS issues (server lockups, high CPU, high RAM)
So week C of April = 3. week of April?
-
- Influencer
- Posts: 15
- Liked: 4 times
- Joined: Jan 16, 2012 10:30 am
- Full Name: James Harper
- Contact:
Re: REFS issues (server lockups, high CPU, high RAM)
KB4088787 is now out, it supersedes KB4077525 and contains ReFS.sys 10.0.14393.2097
Also, it doesn't need the AV QualityCompat registry key.
Also, it doesn't need the AV QualityCompat registry key.
-
- Enthusiast
- Posts: 55
- Liked: 9 times
- Joined: Apr 27, 2014 8:19 pm
- Contact:
Re: REFS issues (server lockups, high CPU, high RAM)
Has anyone been able to compare 4k and 64k yet? Most likely 64k is faster, but it would be nice to see some figures...
-
- Service Provider
- Posts: 158
- Liked: 9 times
- Joined: Dec 05, 2014 2:13 pm
- Full Name: Iain Green
- Contact:
Re: REFS issues (server lockups, high CPU, high RAM)
Hi,
Sorry if this has been asked and answered in this now very long thread but....
Is REFS now usable once again, and do I require anything special to ensure it works?
Do I need 64k block size?
Do I need certain updates?
Do I need any reg changes?
Anything else I may have missed?
I have a new install for a customer and would like to use REFS if it is now working.
Cheers
Sorry if this has been asked and answered in this now very long thread but....
Is REFS now usable once again, and do I require anything special to ensure it works?
Do I need 64k block size?
Do I need certain updates?
Do I need any reg changes?
Anything else I may have missed?
I have a new install for a customer and would like to use REFS if it is now working.
Cheers
Many thanks
Iain Green
Iain Green
Who is online
Users browsing this forum: Bing [Bot], Google [Bot], Semrush [Bot] and 47 guests