REFS issues (server lockups, high CPU, high RAM)

Availability for the Always-On Enterprise

Re: REFS issues (server lockups, high CPU, high RAM)

Veeam Logoby JimmyO » Mon Apr 09, 2018 10:22 am 1 person likes this post

I´ve only installed it on one of my repos but it works really well so I plan to reformat my other ones to ReFS in the next couple of months.
JimmyO
Enthusiast
 
Posts: 51
Liked: 7 times
Joined: Sun Apr 27, 2014 8:19 pm

Re: REFS issues (server lockups, high CPU, high RAM)

Veeam Logoby myFist » Wed Apr 11, 2018 10:12 am

Gostev wrote:
Until 4KB reliability is proven in the field, it is safer to stay with 64KB block size. Especially since Microsoft is reportedly working on optimizing ReFS memory pressure in the April update, which of course will be much higher with the smaller block size.

Thanks!


Hello,

is the memory optimization included in April Update KB4093119?
myFist
Influencer
 
Posts: 20
Liked: 3 times
Joined: Wed Nov 29, 2017 1:06 pm

Re: REFS issues (server lockups, high CPU, high RAM)

Veeam Logoby stoal76 » Thu Apr 12, 2018 12:44 am

Hi everyone,

When our Veeam backups run out of disk space on our Server 2016 REFS volume, the server stops allowing logins via RDP and on the console gets stuck at this message

Setting up personalized settings for web platform customizations

I can get to task manager on the console and see some services stopping and starting, if I try to logout it gets stuck, only option is to force the server off.

This message appears in the logs around the time after the backup runs and fulls the disk:

tcp/ip failed to establish an outgoing connection - and it looks like lots of Veeam event logs message about File is locked by 1 processes

I upgraded the server to the March cumulative update mentioned above but it still had the same problem, maybe it will be fixed in the April update.

We are only using reverse incremental backups but it does seem to be using this Fast cloning Dedupe thing as the total of the backups is more than the total disk size

Does anyone know if you can disable Veeam from using this new REFS Fast cloning feature without rolling back to NTFS??

Thanks
stoal76
Novice
 
Posts: 6
Liked: never
Joined: Tue Mar 10, 2015 8:18 am
Full Name: Shane

Re: REFS issues (server lockups, high CPU, high RAM)

Veeam Logoby Bacon » Fri Apr 13, 2018 11:28 am

I still have the problem on the target repo for my backpcopy job. It only happen when veeam is trying to merge the oldest restore point to the full backup file. The merge is starting, after 4-6 hours the servers loads up the cpu to 100% and is stuck. I can see it on the vmware performance overview, the cpu load goes to 100% the disk down to 0%.

I installed the April updates but no luck there.
Bacon
Novice
 
Posts: 3
Liked: 1 time
Joined: Fri Jan 26, 2018 9:35 am
Full Name: Alexander

Re: REFS issues (server lockups, high CPU, high RAM)

Veeam Logoby Gostev » Sun Apr 15, 2018 9:50 pm

myFist wrote:is the memory optimization included in April Update KB4093119?

I haven't heard the news but I am expecting the update we're waiting as "download only" update at first (available only via Microsoft Update Catalog). This one looks to be published on Windows Update.
Gostev
Veeam Software
 
Posts: 22404
Liked: 2677 times
Joined: Sun Jan 01, 2006 1:01 am
Location: Baar, Switzerland

Re: REFS issues (server lockups, high CPU, high RAM)

Veeam Logoby dive7 » Mon Apr 16, 2018 11:16 am

Is anyone still experiencing this issue when using 64k block size ReFS volumes? If so, are there any workarounds?
dive7
Lurker
 
Posts: 2
Liked: never
Joined: Wed Mar 21, 2018 8:41 pm

Re: REFS issues (server lockups, high CPU, high RAM)

Veeam Logoby kubimike » Mon Apr 16, 2018 7:01 pm 1 person likes this post

@dive7 what issue ?
kubimike
Expert
 
Posts: 306
Liked: 37 times
Joined: Fri Feb 03, 2017 2:34 pm
Full Name: MikeO

Re: REFS issues (server lockups, high CPU, high RAM)

Veeam Logoby dive7 » Wed Apr 18, 2018 9:21 pm

Sorry, I should have clarified. We were seeing our repo server lock up and become unresponsive to RDP and HP ILO console. I originally typed out a post with more info but it got denied for not having a support case to reference. The engineer we worked with suggested offloading the proxy function to a different server. Previously we allowed the jobs to automatically select a proxy, and each job would auto-select the repo server as the proxy. Issue hasn't come up since the last server reboot, and we have since set some of our jobs to go to a new proxy.

We have never used 4k block size ReFS volumes, but we were seeing similiar symptoms with our 64k block size volumes (server lockups/high RAM).


case 02767982
dive7
Lurker
 
Posts: 2
Liked: never
Joined: Wed Mar 21, 2018 8:41 pm

Re: REFS issues (server lockups, high CPU, high RAM)

Veeam Logoby DonZoomik » Wed Apr 18, 2018 10:22 pm

myFist wrote:is the memory optimization included in April Update KB4093119?

I extracted both KB4093119 and KB4093120 and refs.sys is still at 10.14393.2097 (from February).

I don't use ReFS and therefor don't have any issues but I've been following this thread for a long time. :)
DonZoomik
Enthusiast
 
Posts: 43
Liked: 13 times
Joined: Fri Nov 25, 2016 1:56 pm

Re: REFS issues (server lockups, high CPU, high RAM)

Veeam Logoby Cicadymn » Tue Apr 24, 2018 5:10 pm 1 person likes this post

Anybody seen any word on the April update that's supposed to help out a bit too?

I'm doing really good now with the previous update, but I'm almost scared to patch it and fall back into the world of ReFS horror!
Cicadymn
Enthusiast
 
Posts: 26
Liked: 12 times
Joined: Mon Jan 30, 2017 7:42 pm
Full Name: Sam

Re: REFS issues (server lockups, high CPU, high RAM)

Veeam Logoby BigJack » Fri Apr 27, 2018 9:48 pm

dive7 wrote:Is anyone still experiencing this issue when using 64k block size ReFS volumes? If so, are there any workarounds?

I am currently experiencing this issue. I have a pair of 64-disk arrays, each formatted as ReFS with 64K blocks. Fully updated, these Windows 2016 servers still suffer terrible performance with fast-clone operations. The onsite repository has enough time to perform backups with weekly synthetic fulls. The offsite repository can't process all of our backup copy jobs. The data copies from onsite to offsite quickly, but merge operations take forever. Veeam support suggested enabling "Defragment and compact full backup file" within the backup copy jobs. Not only did this not help, but it added to the copy job lengths. I don't understand how this option would benefit an ReFS volume anyway. Also, I think our off-site copy repository is in such bad shape because I believe copy jobs can only use the "Forward Incremental-Forever" storage method, which triggers a fast-clone (SLOW) operation with every backup copy instead of just during synthetic fulls. Help!
BigJack
Lurker
 
Posts: 1
Liked: never
Joined: Fri Apr 27, 2018 9:29 pm
Full Name: Jack Clark

Re: REFS issues (server lockups, high CPU, high RAM)

Veeam Logoby EricJ » Fri May 04, 2018 1:18 pm

Has anyone heard anything new about the ReFS memory fix due out in April?

We don't have server lockups anymore, but during nightly backup copy jobs, the filesystem gets bogged down enough that ongoing SQL transaction jobs fail. Each night we'd reduce the transfer limit on each repo, and had to crank it down to 30MB/sec to keep the jobs from failing. It's a band-aid, but backup windows are much larger and we're not getting anywhere near the max performance of the disk arrays that we paid all this money for. :?

Hoping the rumored "April fix" will address our issue.

Thanks,
Eric
EricJ
Influencer
 
Posts: 18
Liked: 4 times
Joined: Thu Jan 12, 2017 7:06 pm

Re: REFS issues (server lockups, high CPU, high RAM)

Veeam Logoby Gostev » Fri May 04, 2018 2:14 pm 2 people like this post

I already asked my contact for an update about a week ago, but got OOO back as he's on vacation. I will update once I hear anything back... after he's back. Thanks!
Gostev
Veeam Software
 
Posts: 22404
Liked: 2677 times
Joined: Sun Jan 01, 2006 1:01 am
Location: Baar, Switzerland

Re: REFS issues (server lockups, high CPU, high RAM)

Veeam Logoby DonZoomik » Tue May 08, 2018 8:28 pm

KB4103723 has patched refs.sys to 10.0.14393.2248 (April 28th) but nothing in release notes.
DonZoomik
Enthusiast
 
Posts: 43
Liked: 13 times
Joined: Fri Nov 25, 2016 1:56 pm

Re: REFS issues (server lockups, high CPU, high RAM)

Veeam Logoby EzE » Thu May 24, 2018 8:10 pm

BigJack wrote: Fully updated, these Windows 2016 servers still suffer terrible performance with fast-clone operations.

Just rebuilt my backup server and having trouble with ReFS fast-clone slowness as well. I figured with the MS patches out, I could move forward with ReFS. The server is fully up to date, and ReFS.sys file is at version 10.0.14393.2273, which comes down as part of CU KB4103720. I'm using 64k clusters as recommended, but I see terrible merge times for both local disk repo and external USB repo (temporary while changing some cloud provider stuff). Local RAID 10 repo takes about 1 hour per TB for fast clone backup chain transformation. So disappointed that even after all the fixes, I need to consider going back to NTFS before I get too far down the rabbit hole. I hate to add yet another ReFS ticket to the support queue... Anyone see performance degrade with the newer version of ReFS? Maybe MS fixed it, and then broke it again?
EzE
Influencer
 
Posts: 18
Liked: never
Joined: Fri Feb 06, 2015 3:48 pm
Full Name: Eric H

PreviousNext

Return to Veeam Backup & Replication



Who is online

Users browsing this forum: Bing [Bot], Google [Bot] and 21 guests