Availability for the Always-On Enterprise
Locked
JimmyO
Enthusiast
Posts: 55
Liked: 9 times
Joined: Apr 27, 2014 8:19 pm
Contact:

Re: REFS issues (server lockups, high CPU, high RAM)

Post by JimmyO » Apr 09, 2018 10:22 am 1 person likes this post

I´ve only installed it on one of my repos but it works really well so I plan to reformat my other ones to ReFS in the next couple of months.

myFist
Influencer
Posts: 20
Liked: 3 times
Joined: Nov 29, 2017 1:06 pm
Contact:

Re: REFS issues (server lockups, high CPU, high RAM)

Post by myFist » Apr 11, 2018 10:12 am

Gostev wrote:
Until 4KB reliability is proven in the field, it is safer to stay with 64KB block size. Especially since Microsoft is reportedly working on optimizing ReFS memory pressure in the April update, which of course will be much higher with the smaller block size.

Thanks!
Hello,

is the memory optimization included in April Update KB4093119?

stoal76
Novice
Posts: 6
Liked: never
Joined: Mar 10, 2015 8:18 am
Full Name: Shane
Contact:

Re: REFS issues (server lockups, high CPU, high RAM)

Post by stoal76 » Apr 12, 2018 12:44 am

Hi everyone,

When our Veeam backups run out of disk space on our Server 2016 REFS volume, the server stops allowing logins via RDP and on the console gets stuck at this message

Setting up personalized settings for web platform customizations

I can get to task manager on the console and see some services stopping and starting, if I try to logout it gets stuck, only option is to force the server off.

This message appears in the logs around the time after the backup runs and fulls the disk:

tcp/ip failed to establish an outgoing connection - and it looks like lots of Veeam event logs message about File is locked by 1 processes

I upgraded the server to the March cumulative update mentioned above but it still had the same problem, maybe it will be fixed in the April update.

We are only using reverse incremental backups but it does seem to be using this Fast cloning Dedupe thing as the total of the backups is more than the total disk size

Does anyone know if you can disable Veeam from using this new REFS Fast cloning feature without rolling back to NTFS??

Thanks

Bacon
Novice
Posts: 3
Liked: 1 time
Joined: Jan 26, 2018 9:35 am
Full Name: Alexander
Contact:

Re: REFS issues (server lockups, high CPU, high RAM)

Post by Bacon » Apr 13, 2018 11:28 am

I still have the problem on the target repo for my backpcopy job. It only happen when veeam is trying to merge the oldest restore point to the full backup file. The merge is starting, after 4-6 hours the servers loads up the cpu to 100% and is stuck. I can see it on the vmware performance overview, the cpu load goes to 100% the disk down to 0%.

I installed the April updates but no luck there.

Gostev
Veeam Software
Posts: 23215
Liked: 2977 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: REFS issues (server lockups, high CPU, high RAM)

Post by Gostev » Apr 15, 2018 9:50 pm

myFist wrote:is the memory optimization included in April Update KB4093119?
I haven't heard the news but I am expecting the update we're waiting as "download only" update at first (available only via Microsoft Update Catalog). This one looks to be published on Windows Update.

dive7
Lurker
Posts: 2
Liked: never
Joined: Mar 21, 2018 8:41 pm
Contact:

Re: REFS issues (server lockups, high CPU, high RAM)

Post by dive7 » Apr 16, 2018 11:16 am

Is anyone still experiencing this issue when using 64k block size ReFS volumes? If so, are there any workarounds?

kubimike
Expert
Posts: 324
Liked: 37 times
Joined: Feb 03, 2017 2:34 pm
Full Name: MikeO
Contact:

Re: REFS issues (server lockups, high CPU, high RAM)

Post by kubimike » Apr 16, 2018 7:01 pm 1 person likes this post

@dive7 what issue ?

dive7
Lurker
Posts: 2
Liked: never
Joined: Mar 21, 2018 8:41 pm
Contact:

Re: REFS issues (server lockups, high CPU, high RAM)

Post by dive7 » Apr 18, 2018 9:21 pm

Sorry, I should have clarified. We were seeing our repo server lock up and become unresponsive to RDP and HP ILO console. I originally typed out a post with more info but it got denied for not having a support case to reference. The engineer we worked with suggested offloading the proxy function to a different server. Previously we allowed the jobs to automatically select a proxy, and each job would auto-select the repo server as the proxy. Issue hasn't come up since the last server reboot, and we have since set some of our jobs to go to a new proxy.

We have never used 4k block size ReFS volumes, but we were seeing similiar symptoms with our 64k block size volumes (server lockups/high RAM).


case 02767982

DonZoomik
Enthusiast
Posts: 62
Liked: 17 times
Joined: Nov 25, 2016 1:56 pm
Contact:

Re: REFS issues (server lockups, high CPU, high RAM)

Post by DonZoomik » Apr 18, 2018 10:22 pm

myFist wrote:is the memory optimization included in April Update KB4093119?
I extracted both KB4093119 and KB4093120 and refs.sys is still at 10.14393.2097 (from February).

I don't use ReFS and therefor don't have any issues but I've been following this thread for a long time. :)

Cicadymn
Enthusiast
Posts: 26
Liked: 12 times
Joined: Jan 30, 2017 7:42 pm
Full Name: Sam
Contact:

Re: REFS issues (server lockups, high CPU, high RAM)

Post by Cicadymn » Apr 24, 2018 5:10 pm 1 person likes this post

Anybody seen any word on the April update that's supposed to help out a bit too?

I'm doing really good now with the previous update, but I'm almost scared to patch it and fall back into the world of ReFS horror!

BigJack
Lurker
Posts: 2
Liked: never
Joined: Apr 27, 2018 9:29 pm
Full Name: Jack Clark
Contact:

Re: REFS issues (server lockups, high CPU, high RAM)

Post by BigJack » Apr 27, 2018 9:48 pm

dive7 wrote:Is anyone still experiencing this issue when using 64k block size ReFS volumes? If so, are there any workarounds?
I am currently experiencing this issue. I have a pair of 64-disk arrays, each formatted as ReFS with 64K blocks. Fully updated, these Windows 2016 servers still suffer terrible performance with fast-clone operations. The onsite repository has enough time to perform backups with weekly synthetic fulls. The offsite repository can't process all of our backup copy jobs. The data copies from onsite to offsite quickly, but merge operations take forever. Veeam support suggested enabling "Defragment and compact full backup file" within the backup copy jobs. Not only did this not help, but it added to the copy job lengths. I don't understand how this option would benefit an ReFS volume anyway. Also, I think our off-site copy repository is in such bad shape because I believe copy jobs can only use the "Forward Incremental-Forever" storage method, which triggers a fast-clone (SLOW) operation with every backup copy instead of just during synthetic fulls. Help!

EricJ
Influencer
Posts: 18
Liked: 4 times
Joined: Jan 12, 2017 7:06 pm
Contact:

Re: REFS issues (server lockups, high CPU, high RAM)

Post by EricJ » May 04, 2018 1:18 pm

Has anyone heard anything new about the ReFS memory fix due out in April?

We don't have server lockups anymore, but during nightly backup copy jobs, the filesystem gets bogged down enough that ongoing SQL transaction jobs fail. Each night we'd reduce the transfer limit on each repo, and had to crank it down to 30MB/sec to keep the jobs from failing. It's a band-aid, but backup windows are much larger and we're not getting anywhere near the max performance of the disk arrays that we paid all this money for. :?

Hoping the rumored "April fix" will address our issue.

Thanks,
Eric

Gostev
Veeam Software
Posts: 23215
Liked: 2977 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: REFS issues (server lockups, high CPU, high RAM)

Post by Gostev » May 04, 2018 2:14 pm 2 people like this post

I already asked my contact for an update about a week ago, but got OOO back as he's on vacation. I will update once I hear anything back... after he's back. Thanks!

DonZoomik
Enthusiast
Posts: 62
Liked: 17 times
Joined: Nov 25, 2016 1:56 pm
Contact:

Re: REFS issues (server lockups, high CPU, high RAM)

Post by DonZoomik » May 08, 2018 8:28 pm

KB4103723 has patched refs.sys to 10.0.14393.2248 (April 28th) but nothing in release notes.

EzE
Influencer
Posts: 19
Liked: never
Joined: Feb 06, 2015 3:48 pm
Full Name: Eric H
Contact:

Re: REFS issues (server lockups, high CPU, high RAM)

Post by EzE » May 24, 2018 8:10 pm

BigJack wrote: Fully updated, these Windows 2016 servers still suffer terrible performance with fast-clone operations.
Just rebuilt my backup server and having trouble with ReFS fast-clone slowness as well. I figured with the MS patches out, I could move forward with ReFS. The server is fully up to date, and ReFS.sys file is at version 10.0.14393.2273, which comes down as part of CU KB4103720. I'm using 64k clusters as recommended, but I see terrible merge times for both local disk repo and external USB repo (temporary while changing some cloud provider stuff). Local RAID 10 repo takes about 1 hour per TB for fast clone backup chain transformation. So disappointed that even after all the fixes, I need to consider going back to NTFS before I get too far down the rabbit hole. I hate to add yet another ReFS ticket to the support queue... Anyone see performance degrade with the newer version of ReFS? Maybe MS fixed it, and then broke it again?

Locked

Who is online

Users browsing this forum: Bing [Bot], Ildar, ivica.vujovic and 67 guests