REFS issues (server lockups, high CPU, high RAM)

Availability for the Always-On Enterprise

Re: REFS issues (server lockups, high CPU, high RAM)

Veeam Logoby JimmyO » Mon Apr 09, 2018 10:22 am 1 person likes this post

I´ve only installed it on one of my repos but it works really well so I plan to reformat my other ones to ReFS in the next couple of months.
JimmyO
Enthusiast
 
Posts: 49
Liked: 7 times
Joined: Sun Apr 27, 2014 8:19 pm

Re: REFS issues (server lockups, high CPU, high RAM)

Veeam Logoby myFist » Wed Apr 11, 2018 10:12 am

Gostev wrote:
Until 4KB reliability is proven in the field, it is safer to stay with 64KB block size. Especially since Microsoft is reportedly working on optimizing ReFS memory pressure in the April update, which of course will be much higher with the smaller block size.

Thanks!


Hello,

is the memory optimization included in April Update KB4093119?
myFist
Influencer
 
Posts: 11
Liked: never
Joined: Wed Nov 29, 2017 1:06 pm

Re: REFS issues (server lockups, high CPU, high RAM)

Veeam Logoby stoal76 » Thu Apr 12, 2018 12:44 am

Hi everyone,

When our Veeam backups run out of disk space on our Server 2016 REFS volume, the server stops allowing logins via RDP and on the console gets stuck at this message

Setting up personalized settings for web platform customizations

I can get to task manager on the console and see some services stopping and starting, if I try to logout it gets stuck, only option is to force the server off.

This message appears in the logs around the time after the backup runs and fulls the disk:

tcp/ip failed to establish an outgoing connection - and it looks like lots of Veeam event logs message about File is locked by 1 processes

I upgraded the server to the March cumulative update mentioned above but it still had the same problem, maybe it will be fixed in the April update.

We are only using reverse incremental backups but it does seem to be using this Fast cloning Dedupe thing as the total of the backups is more than the total disk size

Does anyone know if you can disable Veeam from using this new REFS Fast cloning feature without rolling back to NTFS??

Thanks
stoal76
Novice
 
Posts: 6
Liked: never
Joined: Tue Mar 10, 2015 8:18 am
Full Name: Shane

Re: REFS issues (server lockups, high CPU, high RAM)

Veeam Logoby Bacon » Fri Apr 13, 2018 11:28 am

I still have the problem on the target repo for my backpcopy job. It only happen when veeam is trying to merge the oldest restore point to the full backup file. The merge is starting, after 4-6 hours the servers loads up the cpu to 100% and is stuck. I can see it on the vmware performance overview, the cpu load goes to 100% the disk down to 0%.

I installed the April updates but no luck there.
Bacon
Novice
 
Posts: 3
Liked: 1 time
Joined: Fri Jan 26, 2018 9:35 am
Full Name: Alexander

Re: REFS issues (server lockups, high CPU, high RAM)

Veeam Logoby Gostev » Sun Apr 15, 2018 9:50 pm

myFist wrote:is the memory optimization included in April Update KB4093119?

I haven't heard the news but I am expecting the update we're waiting as "download only" update at first (available only via Microsoft Update Catalog). This one looks to be published on Windows Update.
Gostev
Veeam Software
 
Posts: 21992
Liked: 2536 times
Joined: Sun Jan 01, 2006 1:01 am
Location: Baar, Switzerland

Re: REFS issues (server lockups, high CPU, high RAM)

Veeam Logoby dive7 » Mon Apr 16, 2018 11:16 am

Is anyone still experiencing this issue when using 64k block size ReFS volumes? If so, are there any workarounds?
dive7
Lurker
 
Posts: 2
Liked: never
Joined: Wed Mar 21, 2018 8:41 pm

Re: REFS issues (server lockups, high CPU, high RAM)

Veeam Logoby kubimike » Mon Apr 16, 2018 7:01 pm 1 person likes this post

@dive7 what issue ?
kubimike
Expert
 
Posts: 305
Liked: 37 times
Joined: Fri Feb 03, 2017 2:34 pm
Full Name: MikeO

Re: REFS issues (server lockups, high CPU, high RAM)

Veeam Logoby dive7 » Wed Apr 18, 2018 9:21 pm

Sorry, I should have clarified. We were seeing our repo server lock up and become unresponsive to RDP and HP ILO console. I originally typed out a post with more info but it got denied for not having a support case to reference. The engineer we worked with suggested offloading the proxy function to a different server. Previously we allowed the jobs to automatically select a proxy, and each job would auto-select the repo server as the proxy. Issue hasn't come up since the last server reboot, and we have since set some of our jobs to go to a new proxy.

We have never used 4k block size ReFS volumes, but we were seeing similiar symptoms with our 64k block size volumes (server lockups/high RAM).


case 02767982
dive7
Lurker
 
Posts: 2
Liked: never
Joined: Wed Mar 21, 2018 8:41 pm

Re: REFS issues (server lockups, high CPU, high RAM)

Veeam Logoby DonZoomik » Wed Apr 18, 2018 10:22 pm

myFist wrote:is the memory optimization included in April Update KB4093119?

I extracted both KB4093119 and KB4093120 and refs.sys is still at 10.14393.2097 (from February).

I don't use ReFS and therefor don't have any issues but I've been following this thread for a long time. :)
DonZoomik
Enthusiast
 
Posts: 33
Liked: 9 times
Joined: Fri Nov 25, 2016 1:56 pm

Re: REFS issues (server lockups, high CPU, high RAM)

Veeam Logoby Cicadymn » Tue Apr 24, 2018 5:10 pm 1 person likes this post

Anybody seen any word on the April update that's supposed to help out a bit too?

I'm doing really good now with the previous update, but I'm almost scared to patch it and fall back into the world of ReFS horror!
Cicadymn
Enthusiast
 
Posts: 26
Liked: 12 times
Joined: Mon Jan 30, 2017 7:42 pm
Full Name: Sam

Previous

Return to Veeam Backup & Replication



Who is online

Users browsing this forum: Bing [Bot], Google [Bot] and 44 guests