REFS issues (server lockups, high CPU, high RAM)

[mkretzer]

Could it be the amount of fragmentation thats causing this? I checked our 4K volume before we deleted it and defrag showed 100 % fragmentation...

[Gostev]

All, I have some good news from Microsoft about this whole issue, stay tuned for the update soon!

[kubimike]

What a tease

[j.forsythe]

Hi.
Sorry for the delay in answering
I have changed my setup to NTFS volumes and since then there has been no crash, or freeze.

@Mike Resseler
I have only installed the patches provided by automatic updates, I haven’t downloaded and installed anything. The Veeam support has been very slow at the beginning, took them 48h to respond. He told me the setup is correct, I should use the ReFS volume again to provoke a server frees and create a server memory dump for the support to analyse. He told as well that they may not be able to analyse anything and I end up opening a MS case, which I won't do.

@kubimike
I don't see much besides that the server is freezing. I have some Eventlogs regarding ReFS timeouts...
I have two rather small ReFS repositorys, 15TB at a local SAS attached HP P440ar controller. It is running in HBA mode and I used Storage Spaces to create a parity volume.
The iSCSI ReFS volume is 10TB connected with two 1Gb connections.

Both are 64k and it worked great for about 2 weeks. Now it turns the rest of my hair grey.

John

[kubimike]

@j.forsythe GEN9 box ? See this post about updating the UEFI veeam-backup-replication-f2/server-2016 ... ml#p231993

[j.forsythe]

@kubimike
Yeah, it is an Gen9.
No I have not seen the post. I am downloading at the moment and will give it a try.
Thanks for the hint!

[kubimike]

@j.forsythe awesome, want to see if that solves your issues.

[skumflum]

I have been running 4K REFS on my copy job repository (DR site replication with two restore points) and ran into BSOD and other horrible stuff. I reformatted my volume to 64K a week ago and so far no issue.

The volume is large – 45TB

However, I noticed that the volume is fragmented (88%). Is defragmentation needed on REFS?... I have a feeling that running defragmentation will not go well on 45TB.

[Gostev]

You should treat ReFS volume where block cloning was used as a sort of deduplication storage. You can never have it NOT fragmented due to the nature of the process (different files sharing the same physical block). As such, there's absolutely no point to perform defragmentation - just like there's no point in defragmenting a deduplicating storage. In fact, the only way to truly defragment any given file would require "inflating" one by recreating the file without leveraging block cloning. Does it make sense?

[j.forsythe]

Hello.
I have installed the HP patch from the following link, as mentioned by @kubimike
http://h20564.www2.hpe.com/hpsc/swd/pub ... nvOid=4231

The servers Intelligent Provisioning version is now 2.50.164, but to be honest I haven't checked before the update which version had been installed....
A small test backup didn't cause any problem with the ReFS repository, tonight there will be a bigger one and I will see in the morning if the backup server froze during the night.

[Richardrichard]

What was the MIcrosoft update please Gostev? I'm currently looking at Veeam and need to work out a best fit repository, need to know if ReFS is on the table.

[WimVD]

All, I have some good news from Microsoft about this whole issue, stay tuned for the update soon!

Hi Gostev, any news to share yet? Need to install a ReFS repository this week and it would be nice if I can avoid any pitfalls...

[Mike Resseler]

Wim,

I can be wrong but I believe he shared it in his weekly digest. But I don't think there is actually a fix at this moment

[Gostev]

Sorry folks, I've got into a little trouble for sharing those news too early in the weekly forum digest, so I need to limit the disclosure to that newsletter for now but in short, the issue is now fully understood and will be addressed by Microsoft soon - the official word from the ReFS dev team should be posted any day now.

[graham8]

Regardless of the specifics of when it will be resolved, it's at least good news just to hear that the actual dev team has become aware of the problem. That takes care of my main concern, which is that this might become one of those "unresolved for years" sorts of problems because the development team is simply never made aware of a serious issue by end-users via frontline support.

[kubimike]

I think I missed the e-mail from Gostev, what did it say ? I've been waiting!

[j.forsythe]

Hello.
Just as a quick update.
Since I installed the HP intelligent provisioning update, there has been no issue with accessing the ReFS volume.
I have changed two jobs, one is writing backups to the RefS iSCSI target, the other one to the local ReFS SAS volume.
Reading data for the "to tape job", is no problem as well.
The last time it worked for about two weeks and not all of the jobs are writing to the ReFS volumes.
But, as I wasn't even able to rescan a repository, this is already a progress.
I will change the last two jobs on Friday, so they can create a new full backup during the weekend.
I will keep you guys updated and hope that there is a MS fix soon too.

John

[WimVD]

Also anxiously awaiting the MS update. Just deployed two proxies with ReFS repositories and have migrated a lot of jobs.
For the moment backup speeds are screaming but they haven't built up their configured restore points yet.
When the jobs start their synthetic operations I hope not to run into any issues or have a fix from MS by then.

[kubimike]

@j.forsynthe are you also running verifier ? Sad to say my box froze today, even with verifier on it failed to create a dump file. It was just frozen at the login screen (Press CTRL + ALT + DEL to login).

[mkretzer]

@Gostev: Do you know if the update released today contains REFS fixes?

EDIT: i just saw there is a new REFS.SYS version 10.0.14393.953 from this month!

[kubimike]

Isn't it patch Tuesday ? Where is the goods

[mkretzer]

https://support.microsoft.com/en-us/help/4013429

[kubimike]

Thanks!
Of the fixed listed which addresses our problems with REFS ?
•Addressed issue with multipath I/O failure that can lead to data corruption or application failures.

[mkretzer]

No i think this might fit:

Addressed additional issues with updated time zone information, Internet Explorer, file server and clustering, wireless networking, Map apps, mobile upgrades for IoT, display rendering, USB 2.0 safe removal, multimedia, Direct3D, Microsoft Edge, enterprise security, Windows Server Update Services, storage networking, Remote Desktop, clustering, Windows Hyper-V, and Credential Guard.

As i said above there is a much newer refs.sys in there

[graham8]

Gotta love how Microsoft has offered less and less information about changes they make over the years. 10 years from now their KBs will just contain $null

[Gostev]

Presumably, it fixes one issue around ReFS, but not "our" issue. I am planning to ping ReFS team this week unless they post the blog they were going to soon.

[kubimike]

Oh

[Mike Resseler]

But it does fixes an issue around ReFS that could lead to corrupted metadata so I would say a rather important update, even if it doesn't fix our issue.

[david.buchanan]

I haven't read the whole 6 pages but from the sounds of it I should stick with my current Server 2012 R2 NTFS until these issues are resolved or is Server 2016 ReFS 64K the way to go?

Dave

[Mike Resseler]

David,

The potential of having issues with 64K are much lower then with 4K. But we have already a report here that it can happen with a very large 64K ReFS volume also. On the other hand, there are many that have 64K volumes with no issues. If you have the possibility to wait until MSFT has solved it. Why not, but it would be a shame if that would put your ReFS 2016 plans on hold for a long time since the benefits of block-cloning API are really cool.

My 2 cents