Windows 2019, large REFS and deletes

[Gostev]

Can you provide this private patch?

Microsoft specifically restricts this. You will need to open a case with Microsoft to get it. It will be a no-cost support case, because it is associated with the bug.

[Andrew@MSFT]

Thanks everyone for your patience here!! Microsoft is working on a resolution and targeting a solution to be available in late March.

Special thanks to those who have validated the private and given us feedback along to the way.

I'll keep everyone posted on the release of the official patch.

If you are currently experience issues, we recommend the following:

• Ensure Trim is disabled

Code: Select all

 fsutil behavior set DisableDeleteNotify ReFS 1

• Set RefsEnableLargeWorkingSetTrim = 1

Code: Select all

HKEY_LOCAL_MACHINE\System\CurrentControlSet\Control\FileSystem
Value Name: RefsEnableLargeWorkingSetTrim
Value Type: REG_DWORD
Value Data: 1

• Create smaller volumes. This can help with the amount of data churn.
• Engage with Microsoft product support. By opening a support case, you get a dedicated resource to help with your specific needs.

[evilaedmin]

Andrew@msft is it correct that calls associated with this bug will be cost-free? Our org has Premiere.

[poulpreben]

Yes, my case was refunded immediately after confirming that my issue was resolved by the hotfix.

[mkretzer]

Ours as well!

[JaySt]

Andrew, do we need to revert the settings after installing the patch when it’s available?
Are these settings recommended post patch install?

[mkretzer]

They told us to keep RefsEnableLargeWorkingSetTrim. But our 1903 Repo runs perfectly even without RefsEnableLargeWorkingSetTrim.

[dasfliege]

I'm still on vacation, but as far as i have been informed by my colleagues, it's still working pretty good with the private hotfix #3. Memory usage is at a minimum, so i may wouldn't even had to double the RAM in our repo server. Hope that the fix will be published to all of you having problems pretty soon, so you can also be in love with ReFS again

[mkretzer]

Here as well - from my contacts with other customers it seems like every customer who got the hotfix is having no more issues!!

[JeremiahDDS]

When is it expected that the private patch will become public?

[Gostev]

There's a message about this directly from the ReFS PM just a few posts above.

[azizm]

Glad I stumbled across this post, we have zero trust in our Veeam server running backups by itself. Windows Server 2019 1809 with 100TB+ ReFS repositories, latest patches applied.

We opened a case with HPE & Veeam support early last year, no solution was found. We spun up multiple virtual proxies and our hangups disappeared by utilizing these proxies. If we do not utilize virtual proxies, our Veeam server will hang (no RDP, high CPU/mem, no ping).

Experienced this issue on Windows Server 2016 as well as Windows Server 2019 1809 with ReFS repos. I'll ask Microsoft Premier for that fix.

Since opening that case last year, I did increase the amount of RAM to more than 1GB per TB of data. This entire thread seems awful familiar, hope it works out.

Edit:

Hmmm, might just wait for the official release.

[Andrew@MSFT]

It's here! Performance enhancements for ReFS released today in KB 4531331.

https://support.microsoft.com/en-us/hel ... -kb4541331

This update includes all the changes in the private fixes validated by customers in this thread.

If you are on WS2019, we recommend you apply KB 4541331.

Thank you all for your patience, and please let us know of your experiences!

[FrancWest]

Thanks for the update. Do we still need to apply the registry settings? Or should it work fine with the defaults?

[Andrew@MSFT]

Yes. The registry tweaks are still recommended.

• Ensure Trim is disabled

Code: Select all

 fsutil behavior set DisableDeleteNotify ReFS 1

• Set RefsEnableLargeWorkingSetTrim = 1

Code: Select all

HKEY_LOCAL_MACHINE\System\CurrentControlSet\Control\FileSystem
Value Name: RefsEnableLargeWorkingSetTrim
Value Type: REG_DWORD
Value Data: 1

[PeterC]

Thank you for the info, have been waiting for this one!!!

[poulpreben]

Andrew, can you please confirm if the final hotfix is still on track for release by the end of this month?

[Gostev]

@poulpreben can you clarify what do you mean by the "final hotfix"? KB4531331 seems pretty final to me

[poulpreben]

I need new glasses. D'oh. Sorry

[bbuchan]

Little back story, We recently stepped into the ReFS world after watching for a while and waiting for 2019 to become stable. We skipped 2016 because we require Dedup AND ReFS. During my initial POC testing I saw no issues with GFS restore points or performance. Since then, we officially added an Archival solution to our catalog and sold it to a customer, and also pointed about half of our Cloud Connect traffic to the extents running ReFS. Once all the retention built up and we started seeing merges running we started seeing a massive performance decrease. Our dedup jobs were barely progressing and many of our merges were taking more than 24 hours our hanging up all together.

I have been following this thread for a while and trying the fixes and not seeing much improvement but chalking that up to the fact that no one else here has mentioned they are also running dedup. In addition, once our first archival customer's jobs attempted to generate a GFS restore point we started getting an error as soon as it would start the fast clone:

Failed to merge full backup file Error: Agent: Failed to process method {Transform.CompileFIB}: The request is not supported.
Failed to duplicate extent. Target file: RelativePath:\...

I created support case 03914893 to troubleshoot the issue. We never reached a solution, although still working.
03/18 I installed KB4531331. Immediately noticed a significant decrease in merge times. All of our local and remote merges completed overnight. Many of them still took 5 - 10 hours (@Andrew@MSFT, do you think that is because of the dedup overhead? At this point I would call this "fast-enough clone", but no where near the sub 5min block cloning should be capable of.) I almost feel like with Dedup mixed with ReFS, block cloning should be even faster, it should be virtually 100% pointer manipulation. Thoughts anyone?

[Andrew@MSFT]

This is indeed dedup overhead. In short, if you try to clone a deduped file, dedup will inline-rehydate the file before forwarding the cloning api. This can be expensive, and slow depending on the system...

Also, deduping a cloned file, would increase storage footprint, at least initially, till all the clones have been dehydrated by dedup and their chunks deduplicated.

[bbuchan]

Andrew@MSFT,

Are there any plans for performance improvements or tighter integration between ReFS and Dedup? Is it possible to ever get to the point where it is purely pointer manipulation? A cloned file essentially results in the same result as two identical (or mostly identical) files that have been fully deduped correct?

[Andrew@MSFT]

Thanks for your feedback! I can't comment on future plans here unfortunately.

[PeterC]

We installed the KB4531331 and had high hopes that we finally would resolve some performance issues we are having with our Server 2019 repository server. But it looks like it is not the resolution for us. It has improved a little bit but surely that what we hoped it would bring.
We are still experimenting with the task limits settings on this Apollo 4200, but at the moment it still looks like when a few merges start during the regular backups the performance decreases dramatically.

The throughput of network traffic is constant during the backups, but when merges start the traffic gets very irregular and drops to 0 randomly. Depending on the number of jobs we have seen throughput as much as 19 Gbps(!) but as soon as the merges start it will drop to Kbps and never spike above 200 Mbps anymore. For longer periods it will even be 0 Kbps. When merges are done it will get better again.

We still have some registry settings in place;

ReFS DisableDeleteNotify = 1
RefsDisableDeleteNotification = 1
RefsDisableLastAccessUpdate = 1
RefsEnableLargeWorkingSetTrim = 1
RefsNumberOfChunksToTrim = 128

Can someone confirm if we have to change or remove some of these settings after the last update?
Any suggestion would be very much appreciated!

[nmdange]

I just applied the update and can report that merge times are significantly improved now! It did seem like the first merge after the update was still slow for a couple of jobs, but right now the latest merge on all jobs was fast.

The throughput of network traffic is constant during the backups, but when merges start the traffic gets very irregular and drops to 0 randomly. Depending on the number of jobs we have seen throughput as much as 19 Gbps(!) but as soon as the merges start it will drop to Kbps and never spike above 200 Mbps anymore

Merge is only working locally on the repository, there is no data to transfer to a different server, so network traffic isn't really a good indication of there being a problem or not. I only have the two changes Andrew mentioned, not the other ones. Might be worth removing them to see if that helps.

[PeterC]

The throughput of network traffic is constant during the backups, but when merges start the traffic gets very irregular and drops to 0 randomly. Depending on the number of jobs we have seen throughput as much as 19 Gbps(!) but as soon as the merges start it will drop to Kbps and never spike above 200 Mbps anymore. For longer periods it will even be 0 Kbps. When merges are done it will get better again.

I mean the throughput of the other running backupjobs that are still transferring data to the repository at the same time other jobs start merging.

What we also notice that during the backups/mergers the ReFS volume is very slow in the Windows explorer, deleting files or (empty) folders takes several minutes. I will see if i can remove the other registry settings. But i would be surprised if this is going to change a lot.

[poulpreben]

One of the most impressive results we have seen after applying the hotfix:

• 60 VM job with only SQL Servers
• 104 TB source data
• 60 TB VBKs
• Synthetic full with fast clone: 1 hour 8 minutes.

Prior to applying the hotfix, this job would never finish.

[mkretzer]

The difference in synthetic backup hours on our weekend synthetics between 1903 and 1909 is also nice:

1903: 10:25:35
1909: 3:49:47

[JeremiahDDS]

I can confirm that the hotfix fixes major issues with ReFS on Windows Server 2019 1809. I was generally sitting at 50+ jobs in progress (because they were taking so long to merge), 128GB+ of RAM and disk queue was high. After hotfix and reboot with no other changes RAM is generally at 64GB, disk queue low and now sitting at 10- jobs generally running.

[PeterC]

We finally got to the sweet spot of the best setting for the task limit on our Server 2019 repository. Now we can also see that the hotfix makes a huge difference.

Normally we would have around 40 jobs still running at 09:00 am. Around 14:00 they would be finished. Since the hotfix and correct task limit setting the jobs are all done at 05:30 - 06:00 am.

For now we still have the extra registry settings in place, first we want to be sure this better performance continuous. After that maybe we can revert some settings, but for now we are very happy to have the backup window back to what it used to be.