REFS issues (server lockups, high CPU, high RAM)

Availability for the Always-On Enterprise

Re: REFS issues (server lockups, high CPU, high RAM)

Veeam Logoby crackocain » Mon Mar 05, 2018 8:42 am

What is the best using case ReFS?

Physical Windows 2016 installing or Server 2016 on Vsphere 6.5?

Some people have issue about raid card drivers on psychical 2016 installation.
EMC, IBM Storage Specialist - VMCEv9 - Dosbil - Turkey
crackocain
Service Provider
 
Posts: 81
Liked: 7 times
Joined: Mon Dec 14, 2015 8:20 pm
Location: Turkey
Full Name: Mehmet Istanbullu

Re: REFS issues (server lockups, high CPU, high RAM)

Veeam Logoby nmdange » Mon Mar 05, 2018 3:42 pm

operations wrote:From what I understand the rollup that contains the fix for ReFS also contains patches for meltdown and spectre, has anyone tried to extract the patch or updates and just install the ReFS fix so as to avoid the melt/spec patches ?

On Windows Server, the mitigations for meltdown and spectre are not enabled by default. You have to create a registry key, and also for spectre it only works if you have updated firmware on your device. Also remember the specter/meltdown patch has been included in the security patches since January, it's not like the ReFS fix is the first time you're getting those patches.
nmdange
Expert
 
Posts: 308
Liked: 74 times
Joined: Thu Aug 20, 2015 9:30 pm

Re: REFS issues (server lockups, high CPU, high RAM)

Veeam Logoby kubimike » Mon Mar 05, 2018 7:33 pm 1 person likes this post

Gostev wrote:Actually, this is an interesting thought. The pre-release driver was provided by itself, and I know for the fact that it helped every single person who installed it... but now we're installing a huge patch to get the same driver, so it is certainly a possibility something else in this patch is misbehaving. Although this is not very likely, because in that case it would misbehave for everyone. Usually, when experiences are so dramatically different, this indicates some environment-specific trigger such as computer configuration (lack of system resources) or 3rd party software conflict (where antivirus is the primary suspect). I hope those still having issues after installing the patch are able to investigate them with Microsoft.

/Sigh , what a mess . Im staying on the second beta until this is sorted out. Everything works for me ATM. Maybe you can wave your magic wand and just get the driver only.
kubimike
Expert
 
Posts: 306
Liked: 37 times
Joined: Fri Feb 03, 2017 2:34 pm
Full Name: MikeO

Re: REFS issues (server lockups, high CPU, high RAM)

Veeam Logoby operations » Mon Mar 05, 2018 10:36 pm

nmdange wrote:On Windows Server, the mitigations for meltdown and spectre are not enabled by default. You have to create a registry key, and also for spectre it only works if you have updated firmware on your device. Also remember the specter/meltdown patch has been included in the security patches since January, it's not like the ReFS fix is the first time you're getting those patches.

Fully understood, and personally I have been avoiding installing melt/spec patches by not using the rollups just individual patches due to the perf hit my backup servers took when I did install it, so if people are installing the all the patches since Jan then they will need to have compensated for the reduction in IO performance.
operations
Service Provider
 
Posts: 12
Liked: never
Joined: Sat Nov 25, 2017 6:49 pm
Full Name: operations

Re: REFS issues (server lockups, high CPU, high RAM)

Veeam Logoby JimmyO » Tue Mar 06, 2018 12:41 pm 1 person likes this post

JimmyO wrote:Installed KB4077525 on one of my smaller backup servers with 58TB volume (backup file: 6TB full, 200GB increment) and merge time went from 8h to 1h after first daily backup.
Of course - the server has just been restarted, but first impressions are good :)

Update: Almost 1 week running with KB4077525 now and it´s still looking good. No signs of decreasing performance. Memory utilization is approx. 40% lower than before but CPU load approx. 50%higher (as expected I guess).

I just wish MS released this before I spent 1 month reverting my main 500TB backup server to NTFS. Here we go again...
JimmyO
Enthusiast
 
Posts: 51
Liked: 7 times
Joined: Sun Apr 27, 2014 8:19 pm

Re: REFS issues (server lockups, high CPU, high RAM)

Veeam Logoby Gostev » Tue Mar 06, 2018 6:37 pm

Thanks for the update, Jimmy. Very happy to hear the new code is holding up well and there's no performance degradation over time as before. And since "no news is good news" in this case, I assume everyone else is seeing the same!
Gostev
Veeam Software
 
Posts: 22404
Liked: 2677 times
Joined: Sun Jan 01, 2006 1:01 am
Location: Baar, Switzerland

Re: REFS issues (server lockups, high CPU, high RAM)

Veeam Logoby leithm » Tue Mar 06, 2018 7:57 pm

So when 2018-02 first came out I installed it on a bunch of servers using REFS repos (Veeam & *cough* dpm).

Haven't had any issues that I can see and speeds improved for sure. Fast forward to yesterday and I've been unable to locate this patch on the update catalogue or windows update anymore?

I have the msu file downloaded from previous installations but Im hesitant to install it if it's been pulled? No-one except this thread has mentioned its missing.
leithm
Service Provider
 
Posts: 10
Liked: never
Joined: Thu May 19, 2016 12:52 am
Full Name: Leith Magon

Re: REFS issues (server lockups, high CPU, high RAM)

Veeam Logoby Gostev » Wed Mar 07, 2018 12:58 am 1 person likes this post

I covered this in the forum digest this past Sunday. In short, Microsoft pulled the patch due to ADFS corruption bug it contains. ReFS is just one of many fixes this patch brings, and one of them appeared to misbehave. But, if you don't run ADFS configuration database on your backup repository server, you're don't need to worry about installing the patch.
Gostev
Veeam Software
 
Posts: 22404
Liked: 2677 times
Joined: Sun Jan 01, 2006 1:01 am
Location: Baar, Switzerland

Re: REFS issues (server lockups, high CPU, high RAM)

Veeam Logoby operations » Wed Mar 07, 2018 10:23 am

Would be nice if they would release the patches independently, or give the ReFS patch to veeam as a hotfix to rollout.
operations
Service Provider
 
Posts: 12
Liked: never
Joined: Sat Nov 25, 2017 6:49 pm
Full Name: operations

Re: REFS issues (server lockups, high CPU, high RAM)

Veeam Logoby Cicadymn » Wed Mar 07, 2018 5:42 pm 1 person likes this post

I've checked in every so often during the past year of these issues.

It looks like this patch had a major affect in getting things to work for me. Despite having 4k block size, I'm no longer crashing and my jobs are completing in a fraction of the time before. I'd see merges that would last 8-12 hours, now they run 20-30 minutes. It's really like a new system! Hopefully things continue to get even better!
Cicadymn
Enthusiast
 
Posts: 26
Liked: 12 times
Joined: Mon Jan 30, 2017 7:42 pm
Full Name: Sam

Re: REFS issues (server lockups, high CPU, high RAM)

Veeam Logoby vipthomps » Wed Mar 07, 2018 6:50 pm

Since applying this patch my Windows 2016 File Server cluster has been crashing and taking the SoBR extents with it when trying to evacuate a small extent. Anyone else running into that? Going to pull the patch off and see if that is the only culprit.
vipthomps
Service Provider
 
Posts: 32
Liked: 1 time
Joined: Mon Dec 06, 2010 7:51 pm
Location: Boston, MA
Full Name: Eric Thompson

Re: REFS issues (server lockups, high CPU, high RAM)

Veeam Logoby Gostev » Wed Mar 07, 2018 8:27 pm

Cicadymn wrote:It looks like this patch had a major affect in getting things to work for me. Despite having 4k block size, I'm no longer crashing and my jobs are completing in a fraction of the time before. I'd see merges that would last 8-12 hours, now they run 20-30 minutes. It's really like a new system! Hopefully things continue to get even better!

That's some precious feedback considering there are few people running with 4KB blocks, so we have very little data on such deployments. If you could share, I would be very interested to know the amount of RAM in your backup repository, ReFS volume capacity and utilization, and the largest backup file size. Thanks!
Gostev
Veeam Software
 
Posts: 22404
Liked: 2677 times
Joined: Sun Jan 01, 2006 1:01 am
Location: Baar, Switzerland

Re: REFS issues (server lockups, high CPU, high RAM)

Veeam Logoby Cicadymn » Thu Mar 08, 2018 3:27 pm 1 person likes this post

Gostev wrote:That's some precious feedback considering there are few people running with 4KB blocks, so we have very little data on such deployments. If you could share, I would be very interested to know the amount of RAM in your backup repository, ReFS volume capacity and utilization, and the largest backup file size. Thanks!

Yeah, we accidentally ended up with 4K size and it's been a challenge. But I'm happy to say that it's working much better.

So for your questions:
    RAM: 48GB
    We've got three repositories at 35TB a piece, using 28, 26.5, and 5.7TB respectively
    Our largest backup file is 4.7TB, though the largest amount of data processed in Veeam is listed at 6.5TB.
    The merge on the largest job lasted 26 minutes.
Cicadymn
Enthusiast
 
Posts: 26
Liked: 12 times
Joined: Mon Jan 30, 2017 7:42 pm
Full Name: Sam

Re: REFS issues (server lockups, high CPU, high RAM)

Veeam Logoby Gostev » Thu Mar 08, 2018 6:11 pm

Thanks so much. Sounds like we should remove that 64KB cluster size recommendation from backup repository wizard down the road, at least for smaller ReFS volumes...
Gostev
Veeam Software
 
Posts: 22404
Liked: 2677 times
Joined: Sun Jan 01, 2006 1:01 am
Location: Baar, Switzerland

Re: REFS issues (server lockups, high CPU, high RAM)

Veeam Logoby operations » Thu Mar 08, 2018 6:57 pm

sounds like its time to do some side by side performance tests 4k vs 64k on the same storage ?
operations
Service Provider
 
Posts: 12
Liked: never
Joined: Sat Nov 25, 2017 6:49 pm
Full Name: operations

PreviousNext

Return to Veeam Backup & Replication



Who is online

Users browsing this forum: Bing [Bot], Google [Bot] and 26 guests