Windows 2019, large REFS and deletes

[JaySt]

Can someone running 2019 build 1903 post the version of the refs.sys file?

[DonZoomik]

According to patch notes it hasn't been patched so it's probaby 10.0.18362.0
I don't have an 1903 system right now to check but it's an educated guess from patch info's file list CSV.

[mkretzer]

Correct, 10.0.18362.0. Thats the version working perfectly for us. It is working so good i even understand why they never patched anything in that version.

[WorkingHardInIt]

Any one tested 10.0.17763.771 yet on Windows Server 2019 1809?

[dasfliege]

@WorkingHardInIt
Not yet. When and how has it been released?

@mkretzer / @gostev
Would you may share the Microsoft case number, where you already had contact with the ReFS Devs? MS is asking me for it.

[GACcc]

Any one tested 10.0.17763.771 yet on Windows Server 2019 1809?

Yes I have... and its not working for us... its horrible.
The first backups are "pretty" fust but then it takes hours and hours

[JaySt]

I did some testing as well. We had two repository servers with Server 2019- 1809. We do copy jobs from the first to the second one. Only on the second one, the one doing GFS stuff, things went bad with merging times. Way too long. Had to restart everything to get it going again each time.
I read through this thread, wiped some tears when i concluded MS created a seemingly stable refs.sys driver a while back, but not fixing this for 1809 customers, and we decided to rip out a refs.sys file from a Windows 10 1903 patched desktop and place it on the second repository only. Refs.sys version coming from the windows 10 1903 machine was 10.0.18362.1 , modified date 19-3-2019.
I have to see how it goes in the coming days, will report back.

[mkretzer]

@dasfliege I could give you one of our case numbers but i do not know if that would help as we really had direct contact with the devs - without a ticket system inbetween...

[JaySt]

I did some testing as well. We had two repository servers with Server 2019- 1809. We do copy jobs from the first to the second one. Only on the second one, the one doing GFS stuff, things went bad with merging times. Way too long. Had to restart everything to get it going again each time.
I read through this thread, wiped some tears when i concluded MS created a seemingly stable refs.sys driver a while back, but not fixing this for 1809 customers, and we decided to rip out a refs.sys file from a Windows 10 1903 patched desktop and place it on the second repository only. Refs.sys version coming from the windows 10 1903 machine was 10.0.18362.1 , modified date 19-3-2019.
I have to see how it goes in the coming days, will report back.

unfortunately, this did not work. GFS restorepoint creation still running. job takes more than 80 hours now. Strange, i had good hopes replacing refs.sys would help. Either this is a different issue not related to ReFS.sys somehow or just replacing refs.sys is not the complete story to fix these kind of issues.

[steggles]

Hi All,

We have just built 2019 Veeam Repositories using the Long Term Servicing Branch of Windows Server 2019. We have been having a few issues with high CPU on ReFS jobs, eventually they cause the server to lock up, and all the other jobs wait on them. It tends to happen on Health Check and Compact operations on the jobs. But from the thread here it would seem to indicate I need to have disks with size greater than 100TB, is that right? As I only have a number of ~74TB drives as part of a bigger SOBR. I wouldn't have thought I would be impacted by this.

I have opened a ticket with Veeam. Should I upgrade to 1903? or is there likely another cause?

Regards,
Thomas Higgins

[JaySt]

I also have disk sizes below or around 100TB and seeing problems.
you can't upgrade to 1903. It's a new install. it's all just such a sad story.

[dasfliege]

Another week, another try:
After MS analysed the latest memory dumps i've collected, while our ReFS performance was almost down to zero, they came up with another possible solution. I've already applied it and so far it looks quite promising. Block Clone activities are now running since 1-2 hours and disk activity never dropped to 0 so far.

Would be nice if some other guys could test it and give a feedback. All you have to do is creating the following regkey and reboot the server:
Path: HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\FileSystem
REG_DWORD value: "RefsEnableLargeWorkingSetTrim"=1

There are some KB describing the issue for WS2016. But it seems like it has reappeared on WS2019:
https://support.microsoft.com/en-us/hel ... is-running
https://support.microsoft.com/en-us/hel ... windows-10

[TheJourney]

I can confirm REFS on windows 2019 is Junk.

Our Exchange server merges were stalling after 8 hours or so, many times not completing at all. The last successful merge on 2019 took over 17 hours. Rebooting the repo would help for a day or so then the issue would come right back.

I reinstalled with Windows Server 1903, kept the data volume (no ReFs reformat) attached it to the server with the same drive letter keeping all the backup files. Kept the same IP address, server name, etc. Opened up Veeam > Backup Infrastructure > MS Windows > Properties of the existing server object in Veeam. This installs the Transport Agent. Then ran through the properties dialog under backup Repos, this installs the other Veeam components needed for Repos. I was then able to run backup jobs without changing anything else.

Immediately retried a failed Exchange merge and it completed in 30 mins. The next exchange merge took less than 2 hours. All backup jobs are completing normally after the Repo migration. Not sure if this is a supported move, but working great so far. I hope it stays this way!!!

Cheers!

[jayscarff]

Another week, another try:
After MS analysed the latest memory dumps i've collected, while our ReFS performance was almost down to zero, they came up with another possible solution. I've already applied it and so far it looks quite promising. Block Clone activities are now running since 1-2 hours and disk activity never dropped to 0 so far.

Would be nice if some other guys could test it and give a feedback. All you have to do is creating the following regkey and reboot the server:
Path: HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\FileSystem
REG_DWORD value: "RefsEnableLargeWorkingSetTrim"=1

There are some KB describing the issue for WS2016. But it seems like it has reappeared on WS2019:
https://support.microsoft.com/en-us/hel ... is-running
https://support.microsoft.com/en-us/hel ... windows-10

Just seeing all clients useing our Cloud Repo (the refs one) slowing down completely. While NTFS cloud repo has no issues. Only thing that has changed are windows updates. Im applying this reg key now and will reboot.

[dasfliege]

Update from my side:
Since i've applied the regkey two days ago, i have all our Copy Jobs (30 Jobs running continously) enabled again and i had no problems at all. Merges and GFS creations runs faster then they do on our 2016 repo and i had no locks at all. Seems like this really did the trick. Hope that it don't get worse again.

@jayscarff
Please report back your experience after you tried it.

[nikpolini]

Hi All.
Can some one on 1903 check to see if the RefsEnableLargeWorkingSetTrim entry exists by default and its value?

Cheers.

[mkretzer]

RefsEnableLargeWorkingSetTrim is not there in 1903 - why should it, the driver is so efficient now...

[GACcc]

Another week, another try:
After MS analysed the latest memory dumps i've collected, while our ReFS performance was almost down to zero, they came up with another possible solution. I've already applied it and so far it looks quite promising. Block Clone activities are now running since 1-2 hours and disk activity never dropped to 0 so far.

Would be nice if some other guys could test it and give a feedback. All you have to do is creating the following regkey and reboot the server:
Path: HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\FileSystem
REG_DWORD value: "RefsEnableLargeWorkingSetTrim"=1

There are some KB describing the issue for WS2016. But it seems like it has reappeared on WS2019:
https://support.microsoft.com/en-us/hel ... is-running
https://support.microsoft.com/en-us/hel ... windows-10

Just tried it right now. Set the reg key and rebooted. First backup was "fast" (not as back in times). As soon as I start multiple jobs simultanously its slow again... even on the graph it goes down to 0.0 KB/s for xxx minutes.

Server 2019 - 1809 - 17763.775

[JaySt]

i also tried RefsEnableLargeWorkingSetTrim on a Server 2019 1809 system with problems. First copy jobs after the reboot seemed to be fast and completed. When i checked after the weekend, all jobs had problems once again. So it did not help in our case.

[dasfliege]

@GaCcc
@JaySt

Did you also applied the modification i mentioned earlier?

Check if Trim is enabled by firing the command: "fsutil behavior query DisableDeleteNotify"

If DisableDeleteNotifiy is disabled (0) for ReFS on the 2019 server, enable it with the following commands:
"fsutil behavior set DisableDeleteNotify 1"
"fsutil behavior set DisableDeleteNotify ReFS"

I only did these two modifications and i'm on refs.sys 10.0.17763.771. Our Backup are now running in parallel (30 Backup- and 30 Copy Jobs) without any problems since more then a week. Even faster then before on 2016. Sad to hear that it didn't worked out so far for you guys.

[GACcc]

@dasfliege

I did only the part below.
Path: HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\FileSystem
REG_DWORD value: "RefsEnableLargeWorkingSetTrim"=1

I'll test the one with the DeleteNotify right now. PS. there is a missing "1" in your example:

If DisableDeleteNotifiy is disabled (0) for ReFS on the 2019 server, enable it with the following commands:
"fsutil behavior set DisableDeleteNotify 1"
"fsutil behavior set DisableDeleteNotify ReFS 1" <=========

[JaySt]

@dasfliege
same as GACcc, i only set the RefsEnableLargeWorkingSetTrim=1 value
i'll try the other two as well. will report back, might take a few days.

[rhys.hammond]

Just applied the reg key (RefsEnableLargeWorkingSetTrim) and the 'fsutil behavior set DisableDeleteNotify ReFS 1'.
Will report back in a few days.

[JaySt]

@rhis.hammond.
You did not use "fsutil behavior set DisableDeleteNotify 1" ? may i ask why? i had doubts about the need for that one, as i'm using a NTFS bootvolume on SSD on this server as well. But i'm not sure i'm making the right assumption.

[rhys.hammond]

If DisableDeleteNotifiy is disabled (0) for ReFS on the 2019 server, enable it with the following commands:
"fsutil behavior set DisableDeleteNotify 1"
"fsutil behavior set DisableDeleteNotify ReFS"

@JaySt,
Simply because DisableDeleteNotifiy was indeed disabled (0) for ReFS on this machine so I "enabled" it.
I didn't see the benefit in disabling delete notify for NTFS.

Also,our performance has fallen off a cliff, I'm still offloading 100's of TBs worth of VBKs but unfortunately, it seems the capacity tier offload process also triggers the poor ReFS issue.
Currently down from 700TB to 400TB, another 200Tb to offload (The telco stuffed around this customer, increasing their WAN pipes took months longer than anticipated, fortunately, they didn't have to reduce their retention due to benefits of ReFS)

I'm likley just going to wipe 1809 and install 1903, the customer doesn't want SAC but the alternative is downgrading to 2016 and then we'd have to wipe the ReFS volume.

[JaySt]

Tested for some days. Unfortunately no improvements after changing DisableDeleteNotify settings to enabled (1)

[GACcc]

same here, I started a backup for all backupsets yesterday evening, before this whole story it took approx 1.5h and for now its about 4.5 - 5 hours.

[JaySt]

In my case, the task "compacting full backup file (xx% done) [fast clone] " takes too much time. This task runs for each VM in the job after the copy jobs copies the most recent restore point for every VM.
It's not that the task has problems or errors out, it just takes too long. I don't know why. I would assume the fast cloning should be much much faster. I'm curious to know by which aspects of a Veeam configuration(GFS, datasize, restorepoints, #VMs) fast clone performance / run time would be impacted the most.

I've got this job for 100 VMs, now running that compacting fast clone task for 6,5 hours. Data size of the job s 7.7TB.
GFS is 7 daily, 4 weekly, 3 monthly 4 quarterly, 1 yearly.
GFS selection is the same day of the week for all.
I have 5 copyjobs besides this one, so I reduced the repository tasks from 16 to max 3. This way the server is more dedicated to a single job when it runs. Just to rule out some things regarding runngiun
That seems to improve some things regarding stability, but not regarding speed of fast clone actions. I barely seem to make it within a 24 hour interval.

[poulpreben]

I recently remounted ReFS from a Server 2016 to Server 2019 (LTSC), which caused performance to tank. Then I attached the volumes to a Server 1903 Core, and I'm still experiencing periodical slow performance.

@mkretzer: Did you just attach existing volumes to your Server 1903, or did you create new ones? I am wondering if it could be trying to run trim/unmap on the 2016 data or something.

[mkretzer]

We recreated the volume and data. But according to microsoft it should not be necesarry. Is your backend storage trim/unmap capable and did you disable that in windows?