REFS issues (server lockups, high CPU, high RAM)

[seritlj]

I think I read some where that 9.5 U3 will disable REFS Fast Clone on 2016 1609 - is this correct?

[florian.meier]

seritlj

as i understand, only if you use deduplication on refs.
right veeam?

TY

[thomas.raabo]

quick reminder about the RefsVirtualSyntheticDisabled registry value. if i understand correctly, this disables fast clone functionality on ReFS volumes. I have not tried it myself.
If you like to switch due to fast-clone problems, It could be an option to not re-format a repository to NTFS but just keep data for a while and use ReFS without fastclone by setting this value. Right?
I still have hopes there will be a fix.... eventually...

Yes but it DOES NOT disable general interaction with the API - integrity streams are still getting pushed by veeam

[JaySt]

Yes but it DOES NOT disable general interaction with the API - integrity streams are still getting pushed by veeam

Ok, but are you saying you expect to still see issues when this registry value is used?

[kubimike]

Just an FYI, my freezing issue is back. After one particular job runs an it deletes old retention points the box has to be powered off over ilo. I am going to try an active full see if that fixes it.

[kubimike]

FWIW, active full stopped the freezes. Back to normal.

[thomas.raabo]

Ok, but are you saying you expect to still see issues when this registry value is used?

Yes... been running with the reg since almost day one

and there are still big problems

[dmartenstyn]

Ok this has gone beyond ridiculous. I have been carefully monitoring this thread and associated issues for months now. Personally I think 2 things.

1) Microsoft should not have released ReFS to the general public. In my opinion with the underlying issues and considering it is a file system, you know, a core component of the OS, its still in beta retrospectively speaking.

2) Veeam should pull all mention of using ReFS from their documentation and best practices (as well as advertisements regarding interoperability and its benefits, etc).

I am just about to deploy my 2nd Veeam server and there is absolutely no way I consider ReFS production ready. I will also be re-formatting the initial Veeam server to NTFS (luckily I have space to do this). Microsoft and Veeam both need to pull their respective fingers out and get this sorted, it has quite literally dragged on for months now and frankly is unacceptable. I can only imagine a 'newbie' not doing a search beforehand and finding this thread and going with best practices, etc and using ReFS

[Steven Bricklayer]

I agree with you dmartenstyn

I really think that Veeam don't understand how bad the situation is about ReFs

The big performance problems are a thing, but the real big problem is that 2 customers on that thread lost the Refs Volume with no way to recover it!

That storage contains BACKUPS, it's really critical.

I'm with the Microsoft support since 1 month, on the maximum level with the maximum parternship you can imagine, and they can't recover the data!

So if you get a quick resume :
- Refs perf are bads most of the time
- You can loose all your data because of Refs

Maybe it's a Microsoft issue but who care?

You veeam guys really must STOP advice customers to go Refs, this is to dangerous

[ferrus]

In ReFS' defence ...

For some customers (us included), NTFS is not sufficient for our needs at the moment, and keeping the status quo brings with it as many issues as the ones currently reported with ReFS.
Some of the posts make an assumption that people are migrating from a position of stability and high performance, to something much worse. The opposite can be true.

NTFS is sufficient on four out of five of our five Veeam repositories, but the nature of the VMs being backed up on the fifth, means multiple synthetic/active fulls break the storage capacity for our current RPO strategy, and merge jobs break the backup window.
ReFS/Fast Clone - provides a solution for both of these, and if an Active Full is required occasionally to reset the performance - so be it. It's no worse than our current position, in fact much, much better.

The recent reports of data loss on this thread, are very troubling however. That has me concerned.
We exceed the 3-2-1 rule for backups, but that's to plan against the possibility of data loss - not the expectation.

[kubimike]

This issue with ReFS only comes up if you have a lot of synthetic fulls like myself. I do one per day to keep performance up when doing a restore, else it suffers and you have freezing problems. Other then that I really have no complains. Is it a hassle? Sure but configured probably the benefits for us outweigh switching back to NTFS.

[DaStivi]

as far as i understand right now:

• never go up to more than 64TB per REFS Volume!!!! (issues starts directly -> slow performance etc..)
• only use forward incr. forever (no syn fulls, this is primary the root cause of many problems, refs would make syn fulls "fast" because it just repointers tables... but basically this looks like to be a big issue, because more syn fulls means more and more pointers to be made and taken care of it)
• never let the refs volume get full, i guess leave at least 10, better 15 or 20% of free space
• don't backup very large vms to refs (vms bigger couple of TBs)
• when formating refs make totally sure that you select 64k Block size!! -> 64k was default till server 2012r2, but now with server 2016 "default" settings is 4k be carefull to select 64k there!!!

i've couple of Veeam installations where refs does its job very good, and i've installations where i had big troubles with refs and gone back to ntfs now!

[kubimike]

as far as i understand right now:

• never go up to more than 64TB per REFS Volume!!!! (issues starts directly -> slow performance etc..) True
• only use forward incr. forever (no syn fulls, this is primary the root cause of many problems, refs would make syn fulls "fast" because it just repointers tables... but basically this looks like to be a big issue, because more syn fulls means more and more pointers to be made and taken care of it) Not sure I agree, more chance of corruption not to mention slow restores.
• never let the refs volume get full, i guess leave at least 10, better 15 or 20% of free space I wouldn't let any volume get full
• don't backup very large vms to refs (vms bigger couple of TBs) I have an 8TB job that runs synthetics daily. As long as I run an active full every 2 months no issues.
• when formating refs make totally sure that you select 64k Block size!! -> 64k was default till server 2012r2, but now with server 2016 "default" settings is 4k be carefull to select 64k there!!!

Totally!

i've couple of Veeam installations where refs does its job very good, and i've installations where i had big troubles with refs and gone back to ntfs now!

Get the beta driver, apply all the settings mentioned in this thread.

[Lunatic Magnet]

Get the beta driver, apply all the settings mentioned in this thread.

Based on threads earlier around page 35-39 several people stated that the beta driver was included in the July Cumulative updates. My server was built in September and has the latest updates and it states that KB4025334 isn't applicable. My refs.sys driver version is 10.0.14393.1770.

The MS article says Specify the indicated values in the following subkey: Is that indicating they should already be there? In my case they are not.

[thomas.raabo]

Based on threads earlier around page 35-39 several people stated that the beta driver was included in the July Cumulative updates. My server was built in September and has the latest updates and it states that KB4025334 isn't applicable. My refs.sys driver version is 10.0.14393.1770.

The MS article says Specify the indicated values in the following subkey: Is that indicating they should already be there? In my case they are not.

Hi Guys!

I´ve been working very very hard with Microsoft and all RS4 fixes have been backported to RS1 and i can confirm that it fixes all the problems i have been seeing.

The driver version you need to get from Microsoft is 10.0.14939.1934

You can refer to my Microsoft case :
[REG:117112217202384] ReFS volumes goes offline while Veeam is using BlockCloneAPI

The fixes also includes best practice registrys that does not at all look like the KB on ReFS problems

[rvvliet78]

Hi Guys!

I´ve been working very very hard with Microsoft and all RS4 fixes have been backported to RS1 and i can confirm that it fixes all the problems i have been seeing.

The driver version you need to get from Microsoft is 10.0.14939.1934

You can refer to my Microsoft case :
[REG:117112217202384] ReFS volumes goes offline while Veeam is using BlockCloneAPI

The fixes also includes best practice registrys that does not at all look like the KB on ReFS problems

This sounds good, did they give any indication when this will be released?

[kubimike]

@lunatic magnet
My microsoft rep and veeam rep have told me the opposite. Not to mention back a few months ago I tried the latest refs driver release to the public and the system freezes. I still to this date running the beta refs driver. Its the only thing that keeps my system running.

[kubimike]

The fixes also includes best practice registrys that does not at all look like the KB on ReFS problems

Could you clarify what you're saying here ? Where can we get a copy of the build you're referring to ?

[Lunatic Magnet]

@lunatic magnet
My microsoft rep and veeam rep have told me the opposite. Not to mention back a few months ago I tried the latest refs driver release to the public and the system freezes. I still to this date running the beta refs driver. Its the only thing that keeps my system running.

Sorry, what part is opposite?

[kubimike]

@lunatic magnet - that the beta refs fixes haven't been ported to public release.

[Lunatic Magnet]

@lunatic magnet - that the beta refs fixes haven't been ported to public release.

I just logged a Premiere call with MS and referenced that case. I'll update what the scoop is and my ticket # for reference if useful.

[kubimike]

@lunatic magnet - so you have refs version 10.0.14939.1934 ?

[Lunatic Magnet]

@lunatic magnet - so you have refs version 10.0.14939.1934 ?

No, mine is currently 10.0.14393.1770. Windows 2016 build 1607 (14393.1884) with all latest patches. I'll see what support has to say

[thomas.raabo]

@lunatic magnet - that the beta refs fixes haven't been ported to public release.

Thats just to string you along.
Like i said - It was no easy task getting this update.

I had my case running for 3 weeks at Sev A to get it... And it has only been released to six customers.

[kubimike]

@thomas Well that’s not so cool. Veeam is a Microsoft partner and they don’t even know it exists. Could you clarify your ‘registry’ statement above ?

[thomas.raabo]

@thomas Well that’s not so cool. Veeam is a Microsoft partner and they don’t even know it exists. Could you clarify your ‘registry’ statement above ?

With this new update i was told to remove all ReFS registry I already had.

Along the update came the new recommendation

Windows Registry Editor Version 5.00

[HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\FileSystem]
"RefsProcessedDeleteQueueEntryCountThreshold"=dword:00010000
"RefsEnableLargeWorkingSetTrim"=dword:00000001
"RefsCheckpointSampleInterval"=dword:00000001
"RefsContainerRotationSampleInterval"=dword:00000001
"RefsLogFileFullSampleInterval"=dword:00000001
;"RefsDisableRefCountParallelDelete"=dword:00000001
;"RefsNumberOfChunksToTrim"=dword:0
;"RefsEnableInlineTrim"=dword:00000001
;"RefsDisableCachedPins"=dword:00000001

[pesos]

Since we do on-host, I assume we would need these refs fixes on our hyper-v hosts, not necessarily on the veeam server(s)?

[WimVD]

Just read Anton's digest and finally feeling a bit confident things will get fixed. Thank you all and merry christmas

[nmdange]

Since we do on-host, I assume we would need these refs fixes on our hyper-v hosts, not necessarily on the veeam server(s)?

The ReFS issue applies to backup repositories formatted with ReFS, so it would be whatever Windows Server hosts the backup repository volumes. Your Hyper-V hosts would not need the fix.

[kubimike]

Windows Registry Editor Version 5.00

[HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\FileSystem]
"RefsProcessedDeleteQueueEntryCountThreshold"=dword:00010000
"RefsEnableLargeWorkingSetTrim"=dword:00000001
"RefsCheckpointSampleInterval"=dword:00000001
"RefsContainerRotationSampleInterval"=dword:00000001
"RefsLogFileFullSampleInterval"=dword:00000001
;"RefsDisableRefCountParallelDelete"=dword:00000001
;"RefsNumberOfChunksToTrim"=dword:0
;"RefsEnableInlineTrim"=dword:00000001
;"RefsDisableCachedPins"=dword:00000001

I see new goodies there. Interesting they now have you set '"RefsProcessedDeleteQueueEntryCountThreshold"=dword:00010000' , with the beta driver the only thing that works for me is value 'dword:00000200'

I curious about 'RefsDisableRefCountParallelDelete' I wonder, if this throttles the pointers when deleting multiple cloned files at once. Any reason why they've had you not use RefsDisableCachedPins ?