ReFS state post September 2018 Windows Updates

[jaybios]

Interesting, well I got a full backup on the weekend, both to disk and to tape with defragment and compact turned off. I'll check my tapes today to ensure I could restore from those if needed, then i'll be able to change the block size. Before i do that though, i'll turn defragment and compact back on and just see if removing the SCCM client was the difference.

What were the symptoms of the issues people were actually seeing with the SCCM client installed?

[dcampregher]

Hello guys!

I am running many customers with the 2457 version of the refs.sys without any problem, even in small servers. Now after the windows update the driver is up to date 2515 and still running stable.

In you cases/environments is this the situation?

[mkretzer]

What is "stable"? We never had crashes in the past 1 1/2 years, still even now our Windows Server 2019 IO grinds to a halt as soon as there are enough TB beeing deleted and the system is nearly unresponsive. REFS takes alot of RAM in these situations.

Our REPO is 600 TB, perhaps thats the issue. RAM is 768 TB dedicated to Veeam - i really hoped that would be enough but the REFS driver does not seem to really utilize > 100 GB.

We tried disabling per-VM (seems to help somewhat but slows merges and fulls) and have the latest patches...

[dcampregher]

My question is about if the repository with ReFs is running whithout problems for other peoples because for me it is all right from the last year since the 2457 driver.
No problems for me with ReFS, I wish only hear other opinions.

[backupquestions]

I've got a 50TB refs 64k block volume on Server 2016. I took a few backups to it, and then had Veeam delete them. The volume has only about 5gb of files in it if I go to properties on the folders.

When I take properties of the entire volume, I show almost 300gb used.... This might lineup with the previous backups that I have since deleted, but it might not have been even that much. Either way, does ReFS take time to free up space that is deleted?? Or does it just mark those blocks as useable again?

edit: I've tested adding regular files such as an .iso of 4gb.... I remove that iso and the counter increments and then goes back down as it should. Yet the original space usage is not leaving. So it seems like Veeam backup files that it deleted are not being removed from the volume as far as clearing the space... I've enabled hidden files and operating system files and there is nothing on the volume other than a folder and then a .vbm and vbk of 5gb size...

[Gostev]

does ReFS take time to free up space that is deleted?

Yes, releasing disk space occupied by deleted files is a background process in ReFS.

[backupquestions]

It doesn't seem consistent though, and it still hasn't released the 300gb of ghost space. I copied a 4gb iso just now and then deleted it and it frees up the space within a few seconds. Yet the original 300gb is still there in use but 0 files to correlate with that.

[Gostev]

You may want to troubleshoot this with Microsoft Support then. They should be able to parse the file system and explain what exactly is stored in those 300GB.

[backupquestions]

In the end it was just the overhead space needed upon formatting an refs volume. Lol.

[mathieu.pierluigi]

Hello everybody,

I have still issues with backups on ReFS.

Case: 03831434

Veeam B&R
Windows 2016
9.5 Update 4
Proxy/Repository Server
Dell R440
40 CPU core
64GB RAM
Windows 2016 1607- Build 14393.3443
Cumulative updates january 2020.
Refs driver version: 10.0.14393.3297
Broadcom Dual ports PCI 10 Gbps,last driver, plugged on a 1Gbps switch.
Storage
Synology RS18017xs+
Last DSM version.
Dual ports 10 Gbps, plugged on a 10 Gbps switch.
Thin provisionned iSCSI LUN of 50 TB
Networks
Flat network
Jumbo frames on every devices.


Active full backups behaviour of 600GB source data
Thoughputs highs at the begining
Memory increasing quickly on the proxy/repository server until 40GB.
Thoughputs drops to 0 after processed one disk and iSCSI LUN not responsive for few minutes.
Then the process resume and the behaviour is the same for each disk processed.



What I tried
Enable flow control on switchs ports and Windows cards==> NOK
Disable IPv4 Chesum Offload ==>NOK
fsutil behavior set DisableDeleteNotify ReFS 1 ==> NOK
Limit concurrent tasks to 4 ==> NOK
Other NAS ==> NOK.
On thick provisionned LUN with advanced options ==> NOK
On thick provisionned LUN with no advanced options ==> NOK
Other VMS ==> NOK
Force ports to 1 Gbps ==> NOK.
Change refs keys https://support.microsoft.com/en-us/hel ... on-windows ==> NOK
Map the iSCSI target on a Windows 2016 VM (same patches) ==> OK, no drops, memory stays below 10 GB
Plug the proxy/repository server on the same 10 Gbps switch as the NAS ==> OK, no drops, memory stays below 10 GB. But we cannot stay on this configuration, because of company rules.

I don't know what can cause this behaviour. I have already opnened a case to support who asked to change server, and still the same problems.

If you have a suspicions, thank you in advance!

[Gostev]

I agree with the recommendation from support, as based on the information above your issue has nothing to deal with ReFS.

[popjls]

Quick question regarding ReFS and different versions..

I'm looking to move our entire repository Prod/DR to ReFS as the space savings will be beneficial but my DR proxy is Server 2019 and version 3.4 of refs but the main Prod side veeam server/proxy is Server 2016 with refs version 3.1 - Am I better upgrading the Prod Server to Server 2019 to match or does it not really matter? Repo's are around 80TB each so not huge but i'm wary of the performance issues with Refs and I'd like to try and mitigate that. It's refs 10.0.14393.3471 and 10.0.17763.1192 and both fully patched.

[steissier]

Hi Popjls,

ReFS requirement is based on the Windows repository server.
Proxy have repository role too ?
According to me, it's better to have all node at the same version, but it is not mandatory.

Sebastien

[Gostev]

Correct - it's not mandatory, as all ReFS interactions are isolated to the data mover running on the repository server.

The only scenario when you want to use OS version with the latest ReFS version is for your mount servers, and only in case you're using ReFS in your protected machines. Without this, you will have issues with the file level recovery, due to mount servers being unable to "understand" a later ReFS version.

[popjls]

Ok I see. So no ReFS shares yet but there will be soon, so I guess I'll have to update everything. My DR Proxy and Prod Proxy have a direct attached iscsi disk in ReFS for the backups to use as that proxy is closer to each respective repositories.

[scottadamson1231]

So am I to assume that a fully patched server 2019 should not have issues with ReFS ? What I'm seeing is massive slowdowns in read speed and merges. The repositories are iscsi served from qnap NAS boxes (running 15k SAS disks, 10G networking) merges originally took a few minutes now taking more than 2 hours in some cases. Did see some improvement when I turned on defrag & compact but just seems to be getting worse with time. Running two tape drives, never getting more than about 300MB/s combined and most of the time a lot worse, its got to the point where the tape jobs dont finish before the disk backups are scheduled start. Not sure what to do, go back to 2016 ?

[Gostev]

There's the dedicated thread about Server 2019 ReFS around here, make sure you have the patch listed there in the final pages installed, and suggested registry keys enabled. I don't know if the patch has already made to the Windows Update, not sure how it works in Microsoft.