Availability for the Always-On Enterprise
Post Reply
Mike Resseler
Veeam Software
Posts: 4691
Liked: 500 times
Joined: Feb 08, 2013 3:08 pm
Full Name: Mike Resseler
Location: Belgium
Contact:

Re: REFS issues (server lockups, high CPU, high RAM)

Post by Mike Resseler » Jan 09, 2018 6:22 am

Hi Warwickb,

First: Welcome to the forums!
Second: Is that public information or did you get it from a system engineer?

Thanks
Mike

jamesmay
Lurker
Posts: 2
Liked: never
Joined: Dec 13, 2017 10:04 pm
Contact:

Re: REFS issues (server lockups, high CPU, high RAM)

Post by jamesmay » Jan 09, 2018 6:41 am

Not sure if it's the same issue but our physical Veeam server running 2016 with a ~80TB REFS volume started hanging (requiring a reset on the iLO) at the completion of every nightly backup of our hyper v environment.

After few days of this the server now hangs about a minute after boot - in safe mode this can be after login but normal mode it only gets up to "Applying computer settings...". Was on the 2017-11 and upgraded to 2017-12 but it didn't help. MS support tried their best but seem to think it's somehow network related because of the where in the normal boot up process it gets - even though it gets past this is safe mode only to hang later.

Mike Resseler
Veeam Software
Posts: 4691
Liked: 500 times
Joined: Feb 08, 2013 3:08 pm
Full Name: Mike Resseler
Location: Belgium
Contact:

Re: REFS issues (server lockups, high CPU, high RAM)

Post by Mike Resseler » Jan 09, 2018 7:40 am

Hi James,
First: Welcome to the forums
Second: The issue that is discussed here is that ReFS becomes very unstable if there is a lot of activity on it and the size large. Not being able to boot it is not something I have heard off with this issue. Something might be related but I am not sure. Please keep working with MSFT support for now and keep us posted. Who knows this is a new problem with ReFS (I hope not though)
Mike

GarethUK
Influencer
Posts: 21
Liked: 2 times
Joined: Mar 21, 2014 11:41 am
Full Name: Gareth
Contact:

Re: REFS issues (server lockups, high CPU, high RAM)

Post by GarethUK » Jan 09, 2018 12:36 pm

James is indeed correct. This is behaviour I have observed. We have 16 backup repo servers 5 of which are 70TB REFS enabled Windows 2016 servers.

I have previously raised this issue with MS support. I did get some registry keys which temporarily fixed the issue. However, it has reoccured. I have again today raised a premium support case with Microsoft support and they have advised me they can provide no fix, no workaround and that I must wait for the permanent fix which they are unable to provide a timescale for. They have closed the case.

Regards,


Gareth

operations
Service Provider
Posts: 12
Liked: never
Joined: Nov 25, 2017 6:49 pm
Full Name: operations
Contact:

Re: REFS issues (server lockups, high CPU, high RAM)

Post by operations » Jan 09, 2018 3:24 pm

Based on the above and the fact that I have a 240TB REPO on ReFS, I need to do something so I was wondering what direction people are going ..

Migrate to NTFS ?
Run Full or Synthetics but stay on ReFS ?

... I cannot continue as my backup merges are killing my server and resulting backups taking days not a day.

Gostev
Veeam Software
Posts: 22818
Liked: 2807 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: REFS issues (server lockups, high CPU, high RAM)

Post by Gostev » Jan 09, 2018 5:44 pm

operations wrote:240TB REPO on ReFS
From what I know based on the conversation with ReFS devs, it may be possible to work around this particular bug around huge volumes by adding lots of RAM to the backup repository server. If you can't do this, then I'm afraid the only option is to fall back to NTFS until Microsoft ships that patch.

operations
Service Provider
Posts: 12
Liked: never
Joined: Nov 25, 2017 6:49 pm
Full Name: operations
Contact:

Re: REFS issues (server lockups, high CPU, high RAM)

Post by operations » Jan 09, 2018 6:27 pm

I do not see a memory issue my Backup is at a crawl at the moment but the server that has 256GB ram in it says there is 120GB free, RAM is not an issue I could easily put in 768GB if that would help.

I presume based on what you are saying that converting to Full backups or synthetic fulls will not fix the issue ?

kubimike
Expert
Posts: 315
Liked: 37 times
Joined: Feb 03, 2017 2:34 pm
Full Name: MikeO
Contact:

Re: REFS issues (server lockups, high CPU, high RAM)

Post by kubimike » Jan 09, 2018 7:18 pm

operations wrote:I do not see a memory issue my Backup is at a crawl at the moment but the server that has 256GB ram in it says there is 120GB free, RAM is not an issue I could easily put in 768GB if that would help.

I presume based on what you are saying that converting to Full backups or synthetic fulls will not fix the issue ?
Use RAMMAP while a clone/merge is running and get back to us. You'll see :mrgreen:

kubimike
Expert
Posts: 315
Liked: 37 times
Joined: Feb 03, 2017 2:34 pm
Full Name: MikeO
Contact:

Re: REFS issues (server lockups, high CPU, high RAM)

Post by kubimike » Jan 09, 2018 7:21 pm

operations wrote:Based on the above and the fact that I have a 240TB REPO on ReFS, I need to do something so I was wondering what direction people are going ..

Migrate to NTFS ?
Run Full or Synthetics but stay on ReFS ?

... I cannot continue as my backup merges are killing my server and resulting backups taking days not a day.
Type of storage?
How many synthetic fulls per week ?
Size of VM causing the issue ?
Frequency of backups ?
Last Active full ?
Space available on disk (REPO)
Cluster size ?
How Much ram ?
Which registry keys + driver are you using ?

operations
Service Provider
Posts: 12
Liked: never
Joined: Nov 25, 2017 6:49 pm
Full Name: operations
Contact:

Re: REFS issues (server lockups, high CPU, high RAM)

Post by operations » Jan 09, 2018 8:33 pm

Type of storage?
IBM SVC with 900GB 10K SAS
How many synthetic fulls per week ?
NONE
Size of VM causing the issue ?
from 127GB to 5TB
Frequency of backups ?
Daily
Last Active full ?
30 days running incr forever no scheduled active fulls hence the merge progress
Space available on disk (REPO)
49Tb free
Cluster size ?
?
How Much ram ?
256GB
Which registry keys + driver are you using ?
No keys + default driver shipped with OS

thomas.raabo
Service Provider
Posts: 28
Liked: 11 times
Joined: Oct 31, 2016 6:27 pm
Full Name: Thomas Raabo
Location: infrastructure guy
Contact:

Re: REFS issues (server lockups, high CPU, high RAM)

Post by thomas.raabo » Jan 10, 2018 8:04 am

operations wrote:Type of storage?
IBM SVC with 900GB 10K SAS
How many synthetic fulls per week ?
NONE
Size of VM causing the issue ?
from 127GB to 5TB
Frequency of backups ?
Daily
Last Active full ?
30 days running incr forever no scheduled active fulls hence the merge progress
Space available on disk (REPO)
49Tb free
Cluster size ?
?
How Much ram ?
256GB
Which registry keys + driver are you using ?
No keys + default driver shipped with OS
That will not work! contact MS and get them to help you.

You do have a support contract right?

operations
Service Provider
Posts: 12
Liked: never
Joined: Nov 25, 2017 6:49 pm
Full Name: operations
Contact:

Re: REFS issues (server lockups, high CPU, high RAM)

Post by operations » Jan 10, 2018 9:03 am

No like many I have no support contract.

Which why I ask what are people doing that cannot get M$ to fix the issue but still need to run production ?

thomas.raabo
Service Provider
Posts: 28
Liked: 11 times
Joined: Oct 31, 2016 6:27 pm
Full Name: Thomas Raabo
Location: infrastructure guy
Contact:

Re: REFS issues (server lockups, high CPU, high RAM)

Post by thomas.raabo » Jan 10, 2018 9:54 am

suprnova wrote:Im not saying this is the silver bullet. I do still have freezes but that means its time for an active full. However, when any public release refs driver from msft it freezes no matter what.
Definitely understand, at the moment even running one incremental merge with [fast clone] makes the repository drive become unstable. Everything was fine for awhile, but now it's back to ground zero. I have your same registry keys, but I am running the latest refs.sys driver.
I recommend you to disable fastclone API in veeam

RefsVirtualSyntheticDisabled DWORD = 1


what version of the driver do you have?

suprnova
Service Provider
Posts: 33
Liked: never
Joined: Apr 08, 2016 5:15 pm
Contact:

Re: REFS issues (server lockups, high CPU, high RAM)

Post by suprnova » Jan 10, 2018 2:18 pm

thomas.raabo wrote:
That will not work! contact MS and get them to help you.

You do have a support contract right?
Can you explain why this doesn't work? Thank you!

suprnova
Service Provider
Posts: 33
Liked: never
Joined: Apr 08, 2016 5:15 pm
Contact:

Re: REFS issues (server lockups, high CPU, high RAM)

Post by suprnova » Jan 10, 2018 2:59 pm

thomas.raabo wrote: what version of the driver do you have?
10.0.14393.1770

Post Reply

Who is online

Users browsing this forum: BogdanT, JimmyO and 54 guests