Windows 2019, large REFS and deletes

[mkretzer]

No, ReFs format is upgraded with 2019. 2016 cannot use it without reformat.

[Steve-nIP]

Andrew@MSFT

It's January.. so..?

It'd be really great to have this sorted out. It seems absurd that a downgrade (or should that be upgrade?) to server 2016 is being consistently suggested as a solution to ReFS performance on 2019.

[Markus M.]

I've updated one WS2019 1809 machine with current KB4534273 - there is no change in refs.sys.
I'd expected a change in file version, but its still V 10.0.17763.831

[DonZoomik]

Week B rarely has functional fixes - mostly only security, next week probably will have another update with hopefully relevant ReFS fixes.

[jcollier5hole]

For what is is worth I will put in my 2 cents.

- We are running 2019 LTSC, also the solution cannot be to upgrade to 2019 1809 to 1903 (Microsoft!), we need this to work on a LTSC and not worry about upgrades every 6 months.
- Having the same issue as everyone else!
- We did implement the RAMMap code suggested by someone in this post and have it running every 10 minutes on all repositories, so far two weeks in everything is running ok.
- We did highly limit the number of concurrent jobs and the concurrent tasks on the repositories, not sure everyone could do this and still meet the backup windows.
- We have also increased the memory considerable on one of the repositories (will do the rest), from 64GB to 384GB. This has allowed us to run the RAMMap ~30 minutes and has also helped greatly with the performance of fast clones and deletes. No clear if we still need to run RAMmap utility but I am to cautious to remove it. We have about 300TB of active data on that repository, currently trying to offload some to Azure.

[dasfliege]

Things became worse again for us after applying the November or December cumulatives. I actually don't know which caused the problems exactly, because i've applied them at the same day. But it seems like performance dropped again significally. We also had RamMap running every 30min for some time and were able to complete our backups within the windows this way. But after the updates i have other strange behaviors like active time and disk queue of our ReFS repo beeing at high values all the time, even when all backjobs are paused and resource monitor lists absolutely no file activity on the given drive. Running RAMMap doesn't do anything in this case. I had to reboot the server completely several times now, in order to get any IO out of that drive. I really hope that there will be some ReFS fixes quite soon...

[EdgarRicharte]

If there's a Server 2019 created ReFs volume can this safely be attached to a Server 2016 system (i.e., do we need to format and start over)?

I believe you will not be able to attach the 2019 volume to a 2016. You would need to evacuate the backups. Reformat. Readd as repo again.

[DonZoomik]

Week B rarely has functional fixes - mostly only security, next week probably will have another update with hopefully relevant ReFS fixes.

Tuesday of week C has passed and nothing. I guess it's a slow month due to holiday season, maybe next week or next month.

[Andrew@MSFT]

DISCLAIMER: I work for Microsoft as a Program Manager on the Storage and File Systems Team – specifically the Resilient File System (ReFS).

Great news! The performance enhancements for ReFS Block Cloning released today in KB 4534321.

https://support.microsoft.com/help/4534321

This update includes improvements to the ReFS reference count table’s performance by adding more granular locking. It also adds a hash table based writeback cache in front of the reference count table. This enables modifications to the reference count table be performed completely in cache, and then lazily persisted in the background.

If you are on WS2019, we recommend you apply KB 4534321.

Thank you all for your patience, and please let us know of your experience when applying KB 4534321 by contacting Microsoft Support.

[EdgarRicharte]

I don't have an optional updates area in server 2019. So I just downloaded from windows update catalog.

[Steve-nIP]

OK, neat. Hopefully it works. I've just worked around some issues by formatting a server with NTFS, and on another I created a 1909 core server as a storage host alongside a 2019 VBR server. But there are still other 2019 servers that could use this ReFS fix.

Looking forward to people's feedback.

[dasfliege]

Going to install it right now and will be able to give feedback on monday.

[poulpreben]

We have a single job backing up a fairly large agent based Windows failover cluster with 4 nodes. The VBK file is 39.4 TB in size, and it’s configured for weekly synthetic full backups.

The repository has 24 HDD in RAID-6+0 (8 drives per span), 32 physical cores and 256 GB RAM.

On Windows 2019 with CU 2019-12, the synthetic full backup generation took 55 hours, 38 hours with CU 2020-01, but after reinstalling with 1909 it took 14 hours.

Considering it’s just a metadata operation, I still cannot understand why it has to take so long, but at least it’s taking less than the RPO. 1909 is a significant improvement, but I wouldn’t consider it a silver bullet.

[deevie]

I've installed the update this morning and synthetic full job (1.5TB of datafiles to process) was about 3 times faster. I still find it rather slow since the system was idling (low disk, mem and cpu usage). I still don't get the speed I got with Server 2016, but it's definately much better than last 2 months.

[Gostev]

Considering it’s just a metadata operation, I still cannot understand why it has to take so long

I would guess that at least a part of the issue may be the fact that you're using an agent-based backup job, however Veeam Agent for Windows does not support parallel disk processing in v3 (but this is coming with v4).

[dasfliege]

I've installed KB 4534321 on friday and don't really see an obvious difference in performance. I left the RAMmap job disabled, which has been running every 15min before, but i had to re-enable it on saturday, as all jobs were stuck. I observed regular disk activity drops to 0 again. After re-enabling the Rammap, the normal Backupjobs are running again, but i still have huge problems with the copy jobs. Some of them are running since 3 days now. So at the moment i can't really confirm that the latest patch has solved any of the problems completely.

[Gostev]

If you're not seeing a significant improvement with the patch installed, then it is safe to say the root cause of your issues is different to start with.

Do you have enough RAM to support the large number of jobs you have running? If yes, then I would try a clean Server 2019 install without putting any things in place that mess with RAM (as ReFS is very sensitive to those). Also, this way you can make sure there's no other 3rd party software messing things up. Finally, such install will make it the perfect starting point for further troubleshooting of your case with Microsoft, as they will be very interested to look at your case.

[dasfliege]

As mentioned earlier in this post, this is a fresh installation of Server 2019 and it's only veeam running on it. It's a physical server with 16 cores and 192GB RAM. We've already treated the case with microsoft premier support for over a month and ended up by setting the EnableLargeWorkingSetTrim and DisableDeleteNotify tweaks. It was running fine back then for a while, but started to become worse over time. We then activated the scheduled RAMMap, which made it work fine again until the december updates.
I actually don't know, if the LargeWorkingSetTrim and DisableDeleteNotify tweaks needs to be removed, in order to gain any profit from the recent update? Anyone else already installed it and can reports how it behaves in their environement?

[Gostev]

Yes, I can confirm that ReFS dev team specifically recommends RefsEnableLargeWorkingSetTrim and DisableDeleteNotify for Server 2019.

Code: Select all

HKEY_LOCAL_MACHINE\System\CurrentControlSet\Control\FileSystem
Value Name: RefsEnableLargeWorkingSetTrim
Value Type: REG_DWORD
Value Data: 1

Code: Select all

fsutil behavior set disableDeleteNotify refs 1

For servers that were upgraded from Server 2016, they also recommend checking if RefsDisableAsyncDelete registry value exists, and deleting it if so. Some customers may have it still set from Server 2016 times, as this was the recommendation back then. But it now prevents large files from being deleted in async mode, as they should.

ReFS devs expect that Server 2019 with KB4534321 installed and these settings above in order should be significantly faster than Server 2016.

[dasfliege]

We did not migrate. We installed everything from scratch and set the above mentioned tweaks.
I therefore still think that there are serious problems within ReFS. For example, right now i have a copy job running since 22 hours in "Creating GFS restore point" state. It claims to use fast clone in the joblog and i can see a .temp file in the repository, which is growing by 20-30MB/s. But in the Windows Ressource Monitor i don't see any entry for that file. Looks for me, like windows somehow doesn't even know that there is something going on and that it should apply some ReFS Magic

[Gostev]

To be honest, actual ReFS fast clone magic never had issues in principle... all ReFS problems were solely related to mass deletion of files created with fast cloning, and impact from the corresponding ReFS metadata updates on the OS and all running processes. Which is why I keep suspecting that your issues are totally unrelated... but let's see what Microsoft has to say.

[poulpreben]

Gostev: That is totally not true.

We have had performance issues on the very first generation of ReFS fast clones on brand new file systems where no deletions were happening at all. Honestly, in my experience, it's just broken on 2019, and has been ever since the GA release. On 2016 it mostly works since around CU 2019-04, although ridiculously slowly. Windows 1903 was an improvement, but still couldn't keep up over time for large jobs. Windows 1909 is a decent workaround for us, but it's a severe limitation that we cannot use it for our "all-in-one" deployments due to lack of Desktop Experience.

[EdgarRicharte]

Yes, I can confirm that ReFS dev team specifically recommends RefsEnableLargeWorkingSetTrim and DisableDeleteNotify for Server 2019.

Code: Select all

HKEY_LOCAL_MACHINE\System\CurrentControlSet\Control\FileSystem
Value Name: RefsEnableLargeWorkingSetTrim
Value Type: REG_DWORD
Value Data: 1

Code: Select all

fsutil behavior set disableDeleteNotify refs 1

We set these tweaks up on the cloud connect server only? I have another server that serves as the SOBR for all of our backups. Do I apply this to that server as well? Or only the cloud connect server?

[JPMS]

Yes, I can confirm that ReFS dev team specifically recommends RefsEnableLargeWorkingSetTrim and DisableDeleteNotify for Server 2019.

Does this also apply to 1903 and 1909 or just LTSB 2019? I'm finding it hard to keep up!

[Gostev]

Gostev: That is totally not true.

It's totally true - you're just not following the conversation. You're talking about performance issues, while I was replying on the specific statement where block cloning just did not work at all (see the last sentence of the post that I'm responding to). Such issues definitely never existed in ReFS, no matter what version. Performance issues on the other hand are real, including Server 2019 specific ones.

[Gostev]

@EdgarRicharte @JPMS these recommendations are for all Server 2019 versions and builds. They need to be applied on each ReFS backup repository server.

[dasfliege]

I've created a video/gif of how GFS restorepoint creation behaves right now in our environement. That definately doesn't look to me like it's using fastclone properly. The corresponding job is now running since 62 hours. No chance to keep up with RPO/RTO until this situation is solved. Is there still no one else who can report his experience with the latest update?

[jcollier5hole]

I have installed the latest updates and the recommendations on the reg and file system settings. I won't know until after the weekend as fast clones are all running on Saturday and Sunday.

[EdgarRicharte]

I'm in the same boat as Jim. I applied the fix on friday. VM seized up. Seems to be running better with the registry fixes. Will find out after the weekend since our compact maintenance happens the first Sunday of the month.

[jcollier5hole]

I've created a video/gif of how GFS restorepoint creation behaves right now in our environement. That definately doesn't look to me like it's using fastclone properly. The corresponding job is now running since 62 hours. No chance to keep up with RPO/RTO until this situation is solved. Is there still no one else who can report his experience with the latest update?

One other thing I would note, have you made sure your backup files for the VMs have not been spreadsheet across different volumes. If they have your fast cloning operations we be slow like on NTFS. I have had this happen where there was not enough space on a volume and it place the backups on a different volume and clones ran forever. I think Veeam should have a setting to stop the backup if it cannot meet placement criteria, today it is just a warning.