REFS issues (server lockups, high CPU, high RAM)

Availability for the Always-On Enterprise

Re: REFS issues (server lockups, high CPU, high RAM)

Veeam Logoby Mike Resseler » Tue Jan 09, 2018 6:22 am

Hi Warwickb,

First: Welcome to the forums!
Second: Is that public information or did you get it from a system engineer?

Thanks
Mike
Mike Resseler
Veeam Software
 
Posts: 4220
Liked: 471 times
Joined: Fri Feb 08, 2013 3:08 pm
Location: Belgium, the land of the fries, the beer, the chocolate and the diamonds...
Full Name: Mike Resseler

Re: REFS issues (server lockups, high CPU, high RAM)

Veeam Logoby jamesmay » Tue Jan 09, 2018 6:41 am

Not sure if it's the same issue but our physical Veeam server running 2016 with a ~80TB REFS volume started hanging (requiring a reset on the iLO) at the completion of every nightly backup of our hyper v environment.

After few days of this the server now hangs about a minute after boot - in safe mode this can be after login but normal mode it only gets up to "Applying computer settings...". Was on the 2017-11 and upgraded to 2017-12 but it didn't help. MS support tried their best but seem to think it's somehow network related because of the where in the normal boot up process it gets - even though it gets past this is safe mode only to hang later.
jamesmay
Lurker
 
Posts: 2
Liked: never
Joined: Wed Dec 13, 2017 10:04 pm

Re: REFS issues (server lockups, high CPU, high RAM)

Veeam Logoby Mike Resseler » Tue Jan 09, 2018 7:40 am

Hi James,
First: Welcome to the forums
Second: The issue that is discussed here is that ReFS becomes very unstable if there is a lot of activity on it and the size large. Not being able to boot it is not something I have heard off with this issue. Something might be related but I am not sure. Please keep working with MSFT support for now and keep us posted. Who knows this is a new problem with ReFS (I hope not though)
Mike
Mike Resseler
Veeam Software
 
Posts: 4220
Liked: 471 times
Joined: Fri Feb 08, 2013 3:08 pm
Location: Belgium, the land of the fries, the beer, the chocolate and the diamonds...
Full Name: Mike Resseler

Re: REFS issues (server lockups, high CPU, high RAM)

Veeam Logoby GarethUK » Tue Jan 09, 2018 12:36 pm

James is indeed correct. This is behaviour I have observed. We have 16 backup repo servers 5 of which are 70TB REFS enabled Windows 2016 servers.

I have previously raised this issue with MS support. I did get some registry keys which temporarily fixed the issue. However, it has reoccured. I have again today raised a premium support case with Microsoft support and they have advised me they can provide no fix, no workaround and that I must wait for the permanent fix which they are unable to provide a timescale for. They have closed the case.

Regards,


Gareth
GarethUK
Influencer
 
Posts: 21
Liked: 2 times
Joined: Fri Mar 21, 2014 11:41 am
Full Name: Gareth

Re: REFS issues (server lockups, high CPU, high RAM)

Veeam Logoby operations » Tue Jan 09, 2018 3:24 pm

Based on the above and the fact that I have a 240TB REPO on ReFS, I need to do something so I was wondering what direction people are going ..

Migrate to NTFS ?
Run Full or Synthetics but stay on ReFS ?

... I cannot continue as my backup merges are killing my server and resulting backups taking days not a day.
operations
Service Provider
 
Posts: 12
Liked: never
Joined: Sat Nov 25, 2017 6:49 pm
Full Name: operations

Re: REFS issues (server lockups, high CPU, high RAM)

Veeam Logoby Gostev » Tue Jan 09, 2018 5:44 pm

operations wrote:240TB REPO on ReFS

From what I know based on the conversation with ReFS devs, it may be possible to work around this particular bug around huge volumes by adding lots of RAM to the backup repository server. If you can't do this, then I'm afraid the only option is to fall back to NTFS until Microsoft ships that patch.
Gostev
Veeam Software
 
Posts: 22179
Liked: 2610 times
Joined: Sun Jan 01, 2006 1:01 am
Location: Baar, Switzerland

Re: REFS issues (server lockups, high CPU, high RAM)

Veeam Logoby operations » Tue Jan 09, 2018 6:27 pm

I do not see a memory issue my Backup is at a crawl at the moment but the server that has 256GB ram in it says there is 120GB free, RAM is not an issue I could easily put in 768GB if that would help.

I presume based on what you are saying that converting to Full backups or synthetic fulls will not fix the issue ?
operations
Service Provider
 
Posts: 12
Liked: never
Joined: Sat Nov 25, 2017 6:49 pm
Full Name: operations

Re: REFS issues (server lockups, high CPU, high RAM)

Veeam Logoby kubimike » Tue Jan 09, 2018 7:18 pm

operations wrote:I do not see a memory issue my Backup is at a crawl at the moment but the server that has 256GB ram in it says there is 120GB free, RAM is not an issue I could easily put in 768GB if that would help.

I presume based on what you are saying that converting to Full backups or synthetic fulls will not fix the issue ?


Use RAMMAP while a clone/merge is running and get back to us. You'll see :mrgreen:
kubimike
Expert
 
Posts: 305
Liked: 37 times
Joined: Fri Feb 03, 2017 2:34 pm
Full Name: MikeO

Re: REFS issues (server lockups, high CPU, high RAM)

Veeam Logoby kubimike » Tue Jan 09, 2018 7:21 pm

operations wrote:Based on the above and the fact that I have a 240TB REPO on ReFS, I need to do something so I was wondering what direction people are going ..

Migrate to NTFS ?
Run Full or Synthetics but stay on ReFS ?

... I cannot continue as my backup merges are killing my server and resulting backups taking days not a day.


Type of storage?
How many synthetic fulls per week ?
Size of VM causing the issue ?
Frequency of backups ?
Last Active full ?
Space available on disk (REPO)
Cluster size ?
How Much ram ?
Which registry keys + driver are you using ?
kubimike
Expert
 
Posts: 305
Liked: 37 times
Joined: Fri Feb 03, 2017 2:34 pm
Full Name: MikeO

Re: REFS issues (server lockups, high CPU, high RAM)

Veeam Logoby operations » Tue Jan 09, 2018 8:33 pm

Type of storage?
IBM SVC with 900GB 10K SAS
How many synthetic fulls per week ?
NONE
Size of VM causing the issue ?
from 127GB to 5TB
Frequency of backups ?
Daily
Last Active full ?
30 days running incr forever no scheduled active fulls hence the merge progress
Space available on disk (REPO)
49Tb free
Cluster size ?
?
How Much ram ?
256GB
Which registry keys + driver are you using ?
No keys + default driver shipped with OS
operations
Service Provider
 
Posts: 12
Liked: never
Joined: Sat Nov 25, 2017 6:49 pm
Full Name: operations

Re: REFS issues (server lockups, high CPU, high RAM)

Veeam Logoby thomas.raabo » Wed Jan 10, 2018 8:04 am

operations wrote:Type of storage?
IBM SVC with 900GB 10K SAS
How many synthetic fulls per week ?
NONE
Size of VM causing the issue ?
from 127GB to 5TB
Frequency of backups ?
Daily
Last Active full ?
30 days running incr forever no scheduled active fulls hence the merge progress
Space available on disk (REPO)
49Tb free
Cluster size ?
?
How Much ram ?
256GB
Which registry keys + driver are you using ?
No keys + default driver shipped with OS


That will not work! contact MS and get them to help you.

You do have a support contract right?
thomas.raabo
Service Provider
 
Posts: 28
Liked: 11 times
Joined: Mon Oct 31, 2016 6:27 pm
Location: infrastructure guy
Full Name: Thomas Raabo

Re: REFS issues (server lockups, high CPU, high RAM)

Veeam Logoby operations » Wed Jan 10, 2018 9:03 am

No like many I have no support contract.

Which why I ask what are people doing that cannot get M$ to fix the issue but still need to run production ?
operations
Service Provider
 
Posts: 12
Liked: never
Joined: Sat Nov 25, 2017 6:49 pm
Full Name: operations

Re: REFS issues (server lockups, high CPU, high RAM)

Veeam Logoby thomas.raabo » Wed Jan 10, 2018 9:54 am

suprnova wrote:Im not saying this is the silver bullet. I do still have freezes but that means its time for an active full. However, when any public release refs driver from msft it freezes no matter what.
Definitely understand, at the moment even running one incremental merge with [fast clone] makes the repository drive become unstable. Everything was fine for awhile, but now it's back to ground zero. I have your same registry keys, but I am running the latest refs.sys driver.


I recommend you to disable fastclone API in veeam

RefsVirtualSyntheticDisabled DWORD = 1


what version of the driver do you have?
thomas.raabo
Service Provider
 
Posts: 28
Liked: 11 times
Joined: Mon Oct 31, 2016 6:27 pm
Location: infrastructure guy
Full Name: Thomas Raabo

Re: REFS issues (server lockups, high CPU, high RAM)

Veeam Logoby suprnova » Wed Jan 10, 2018 2:18 pm

thomas.raabo wrote:
That will not work! contact MS and get them to help you.

You do have a support contract right?


Can you explain why this doesn't work? Thank you!
suprnova
Service Provider
 
Posts: 33
Liked: never
Joined: Fri Apr 08, 2016 5:15 pm

Re: REFS issues (server lockups, high CPU, high RAM)

Veeam Logoby suprnova » Wed Jan 10, 2018 2:59 pm

thomas.raabo wrote:what version of the driver do you have?


10.0.14393.1770
suprnova
Service Provider
 
Posts: 33
Liked: never
Joined: Fri Apr 08, 2016 5:15 pm

PreviousNext

Return to Veeam Backup & Replication



Who is online

Users browsing this forum: Craqcatoa, KevinJ and 38 guests