Windows 2019, large REFS and deletes

[Gostev]

In the past we have tested a little bit with synthetic fulls, but we noticed a significant increase of storage usage so returned back to the forward incremental.

Synthetic fulls don't take up any physical disk space on ReFS (and never did) so whatever storage consumption increase you saw was unrelated to the backup mode change.

[ITP-Stan]

With ReFS repo set-up correctly synthetic fulls should not really increase storage usage.
Maybe only a little bit depending on the number of restore points you keep and the closing of the chain.

[jeffshead]

Does anyone know if this issue still exists with Server 2019 LTSC (1809) with the April 9th, 2021 refs.sys version 10.0.17763.1728?

I'm not sure if I should use NTFS or ReFS for the repository.

[mkretzer]

I'll ask Microsoft - i wanted to ask them anyway about 21H1

[Gostev]

Does anyone know if this issue still exists

Can you clarify which issue are you talking about? There haven't really been any around Server 2019 for a long time now.

[jeffshead]

Can you clarify which issue are you talking about? There haven't really been any around Server 2019 for a long time now.

Thanks for chiming in. My inquiry was for any issues listed in this thread or any issues in general. I haven't formatted the repository drive yet. Wanted to have confidence that ReFS is going to be at least as reliable/robust as NTFS on Server 2019 LTSC (1809).

[mkretzer]

How big is your repo? With 2019 issues only started to come up with a repo way over 300 TB in one extend - even that system runs normally without issues for months!
All that with a very old repo hardware (Dell R720).

[jeffshead]

How big is your repo?

36TB on Dell R720XD, Windows Server 2019 LTSC (1809). The main reason for my post was because some posted that there were ReFS driver issues with 1809 but not with newer versions of Windows.

[mkretzer]

I am 98 % certain you will have no issues at that scale. Our 2019 LTSC repo (with newest updates!) is an old R720 with 300 TB. It ran nearly flawless for a year now and only showed issues when very large backups were deleted.

[jbcap]

With 2019 1809, are there currently any recommended registry settings or hotfixes? Or is a vanilla install fine at the moment?
Are there any benefits to moving off the LTSB to Semi-Annual release?

refs.sys is 10.0.17763.1971

Thanks.

[jeffshead]

Or is a vanilla install fine at the moment?

I think these issues have been resolved.

I made sure my 1809 server had the latest updates and I recreated a repository about a month ago -- Switched from NTFS to ReFS and have encountered zero issues. However, my full backup went from taking 4.25 hrs on NTFS to taking over 6 hrs to complete on ReFS. Synthetic fulls take no time to create on ReFS but I'm still considering going back to NTFS and active fulls.

[DerOest]

@jeffshead, from the previous page:

Hi Adrian_C

If you run any cumulative later then apil 19, you should also have the ReFS fix in place. Check the version of your refs.sys driver. Latest LTS version is 10.0.17763.1369

Did you also applied largeworkingsettrim and deletenotify? If not, execute the following two commands in an elevated command prompt:
fsutil behavior set DisableDeleteNotify ReFS 1
REG ADD HKLM\System\CurrentControlSet\Control\FileSystem /v RefsEnableLargeWorkingSetTrim /t REG_DWORD /d 1

[jeffshead]

@jeffshead, from the previous page:

Thanks but I think that suggestion applies only to repositories that are on SSD's, correct? That will not help if the repository is on mechanical drives, will it?

[DerOest]

These options have nothing to do with SSD/HDD, they are ReFS-internal settings and generally valid

[jeffshead]

These options have nothing to do with SSD/HDD, they are ReFS-internal settings and generally valid

I'm not second-guessing you but all of the results turned up from my previous Internet searches state that those settings relate to trim; hence SSDs. So that I have a better understanding, can you explain or point me to any articles that describe how/why those settings improve performance regardless of the hard drive type(s) that are installed in the system?

[Gostev]

Trim is a generic term, not necessarily exclusive to SSD. For example, memory usage can also be "trimmed" as is the case with RefsEnableLargeWorkingSetTrim value, which controls how aggressively the working set of the ReFS metadata management process is reduced.

I don't know if Microsoft has public articles with the level of details you require, as they would have to go deep into ReFS internals to explain "how/why those settings improve performance". Perhaps try contacting Microsoft Support to see if they would be able to provide this information to you.

[jbcap]

So, is the consensus that we should apply the RefsEnableLargeWorkingSetTrim registry setting at the moment with an up-to-date 1809?

[mkretzer]

Last information i have from microsoft is that we should activate it...

[mkretzer]

Does anyone else have bad ReFS write performance with the latest hotfixes (CU6 + Printernightmare fix)? We have much more target bottleneck and our bigger jobs take 2-3 hours longer.
There is a new ReFS Driver in one of these two updates (at least for SAC 2004 and 21H1, checking LTSC currently).

Markus

[poulpreben]

We are on LTSC with latest patches and are currently troubleshooting some performance issues. Throughput goes to near zero as soon as just one job starts merging.

Subscribing for updates…

[mkretzer]

The driver on LTSC is from 12.05.2021... So its unlikely the same happens there...

I asked MS, will keep you updated.

[poulpreben]

Does that mean you installed it on that date, or that the driver got signed there?

Patch Tuesday in May was on 11.05.2021, so if it was signed the following day, it would've been included in 2021-06 CU instead.

[Edit]: I just checked it. It was indeed signed on 12.05.2021, and we did not deploy it until 15.06.2021 (one week after Patch Tuesday according to the patching policy for those systems). I will try to correlate this with our performance regressions.

[mkretzer]

12.05. is the signing date.
MS wants a memory dump. We will try to provide them one ASAP

[Captnspdr]

We are seeing performance issues with the latest 2019 cumulative updates as well. We have a cloud host that ReFS.sys seems to be eating the CPU on the server to the point that it has to be rebooted every 5 days or so.

Right now, one job is writing to the server at about 20 Mbps and ReFS.sys is using between 40 and 60% CPU on a Dell R720. Uptime is a little over 3 days.

We have already applied RefsEnableLargeWorkingSetTrim and DisableDeleteNotify.

[mkretzer]

Its strange - last week our backups were just slower. This week they work nearly normal. I just wanted inform microsoft that its all good when i saw you message Captnspdr - indeed, our CPU usage seems to be 15-20 % higher after the patch. I will try to verify that with the next backup run.

Which process causes the CPU usage in your system?

[Captnspdr]

I had to run Windows Performance Analyzer to find out, but the culprit is ReFS.sys.

[Captnspdr]

Looks like the ReFS driver was updated again in mid-July, but we're still seeing the high CPU usage from ReFS.sys.

[LaurentZ]

We have 2 Veeam v11 repositories, both Windows Server 2019 v1809 and the behavior is totally different.
One with refs.sys 10.0.17763.1971 : work great
One with refs.sys 10.0.17763.2090 : catastrophic behavior. Copy jobs worked great, but CPU at 100% when merge (fast clone). Even with 1 single job...
Server is totally unresponsive after a couple of hours

[eee777]

We moved from Server 2016 / Veeam v10 to Server 2019 build 1809 / Veeam v11 / refs.sys is 10.0.17763.2090. Copy and Tape Jobs are performing much faster now (almost 2x speed) but regular backups got worse. Sometimes there is not throughput for 10-15 mins , then it goes back to normal ... then back to 0. CPU usage is 2-10% without any spikes during the backup / synthetic full jobs. We are using the same storage as we used with Server 2016 , only the standalone backup server was upgraded with 2x more memory - 256GB for single 80TB ReFS repo. The registry fix suggested here is also applied.

[Captnspdr]

@LaurentZ

We are on 2019 LTSC and it looks like our driver version giving us the issue is 10.0.17763.2090.