Server 2025 - high CPU and RAM

[alexandre.nakagawa]

Hello gostev and Johan
I installed a new Veeam server and physical proxies (256GB RAM) with Windows 2025.
I mapped the LUNs from the old servers (Windows 2022 REFS) to the new server.
We migrated on Friday, and early this morning, the server started freezing, with high CPU and memory usage.
I have repositories of 300TB each, with approximately 1PB of data in one of the repositories.
I applied the FileCacheLimitPercent registry keys, but the server remains unstable after 30 minutes of uptime.
Is there any Microsoft fix I can request when opening a support case with Microsoft?
Would you be able to provide this fix if I open a P0 support case with Veeam

[karsten123]

why do you not just wait for the official fix?

[Gostev]

Unfortunately we're never able to provide private fixes we receive from Microsoft for validation, as it's an intellectual property that does not belong to us and therefore we cannot redistribute it.

[DaStivi]

Can you post some reference number to help with a case at Microsoft to get the private hotfix without any more hassle ?

Waiting another month isn't a option for many if the server just stops responding without being able to create backups... And no none of us is willing to install another Server any maybe loose some weeks or months of backups ..

[alexandre.nakagawa]

why do you not just wait for the official fix?

Because my server is unresponsive and I missing almost 1PB of backups... the server run only for about 4 hours and freeze, it has 256GB o ram used.

and the sql databases are becoming unavailable per lack of space on transaction logs directory since the backup logs are not running, (our databases have 200TB each) and we do 30 min log backups.

[karsten123]

BEFORE jumping no scope on the 2025 train

… at least, its a valuable lesson

[alexandre.nakagawa]

I have other 3 veeam repositories that was migrated to 2025 and was running fine for a long time. this is our only server that had this bug, problably because the size of backups and repositories, small repo servers is working fine

[johan.h]

Understand that this is very frustrating. We were given KB9178699 at the time, private fixes should be requested from Microsoft (its their code after all). If that doesn't work use the private message feature to reach out to me and maybe I can help you push a little.

Separately, Alexandre, I've worked with your Veeam Support engineer on this one and I believe they have a plan in place for you.

[alexandre.nakagawa]

Ok, thanks johan, i received the update from veeam eng.

[BenM2024]

The ReFS memory exhaustion fix is expected from Microsoft in July for HCI23H2 and Windows Server 2025.

That's good news - if it works

FWIW I still can't run a backup (to an NTFS drive) becasue the ReFS bug hangs the server. It seems the 5ish Gb of ram left after ReFS takes it's bit (even when Veeam is disabled)
is not enough memory for a backup to a local NTFS volume.
I might get the opportunity to rob some RAM form another server next week and see if a few more Gb provides enough space for backups to run - it does seem that the ReFS bug maxes out at 90Gbish RAM in my use case. Its worth the downtime on the donor server I feel.

[BenM2024]

I might get the opportunity to rob some RAM form another server next week and see if a few more Gb provides enough space for backups to run

Well that went as expected - even with 160Gb RAM in the server it dies whilst doing the backup job. All other jobs disabled. Roll on July Patch Tuesday (8th isn't it?)

[johan.h]

From what I heard from Microsoft, I would expect a little later in July. But yes, not so far away now.

[alexandre.nakagawa]

My workaround was to mount the iSCSI disk on another machine with 512GB of RAM (a 300TB ReFS volume).
After many hours, I was finally able to access the disk directories.
I then copied the VIB/BKP files to another NTFS disk.
The server crashed multiple times even with 512GB of RAM, but I gradually managed to copy each file.
I lost several backups, but they were weekly backups that would have been deleted soon due to retention, so I only copied the critical ones.
I was able to copy approximately 150TB.

[DaStivi]

i've got a privat fix from microsoft that looks like to fixed the issues... i only had about 10 or 15mintues until the server freezed...
ms-support told me the fix is planned to become publicly available in August ...

[Gostev]

From what I heard, the fix requires changing one registry setting in addition to deploying actual binaries. Did you do both things?

[DaStivi]

No just to install the fix because it's a test-signed file...
But I had refs memory reg keys in place before to try to work around the issue, but none worked...

https://learn.microsoft.com/en-us/troub ... usage-refs

[halllowien]

if you don't "have enough time" for the fix to install or problems with backups on ntfs while refs is mounted, you can maybe remove the refs disk.
for me this solves the hanging problem of Windows to have enough time to do something.

[JeremiahDDS]

Having the same issue, does anyone have this private patch?

[JTT]

Latest response from my MS support case:
I just wanted to give you a quick update on the ongoing ReFS issue. Our Product Group (PG) team is actively working on the fix, and it looks like it might be scheduled for release at the end of July.
I'll be sure to share any further updates as soon as I have it.

[JeremiahDDS]

I was on Windows Server 2022 and it was very fast, upgraded to Windows Server 2025 my 7 ReFS volumes are crawling even ones that are SSD. 100% disk activity, lots of RAM usage even when the server isn't doing much. If the final patch isnt going to be out till end of July, what does anyone recommend? I have some very large HDD volumes that I had to disconnect because when they go to 100% it makes the server non respond to, I had to disconnect them. I can't just format the volumes it's too much data. I tried the ReFS tweaks and the private patch.

[robg]

I've probably been reading about ReFS problems here for at least 5 years now. Is it still bleeding edge? Why don't you just abandon it for more traditional methods, and simplify your life?

[mkretzer]

"Why don't you just abandon it for more traditional methods, and simplify your life?" -> Is XFS traditional now?
Because XFS is the same concept but (until now) with 0 Bugs....

[Gostev]

Is it still bleeding edge?

Indeed, and the only issue is people jumping Microsoft's "bleeding edge" instead of steering clear out of it. It's like 40 years was not enough to learn that every major Microsoft release brings a bunch of regressions and therefore best avoided for the first year or so.

[Gostev]

Is it still bleeding edge?

Indeed, I would say the very issue here is that people keep jumping on Microsoft's "bleeding edge" instead of steering clear of it. It's like 40+ years was not enough to learn that every major Microsoft release brings a bunch of regressions and therefore best avoided for the first year or so.

[MirkoKiel123]

We've been using Server 2025 on a bare metal server for Veeam since April and haven't had any issues. However, we haven't migrated a repository; we've created a new one.

I've probably been reading about ReFS problems here for at least 5 years now. Is it still bleeding edge? Why don't you just abandon it for more traditional methods, and simplify your life?

I've never had any issues with ReFS in the past eight years in various customer environments.

[JeremiahDDS]

Further testing I can confirm the private patch and the ReFS registry changes mostly resolve the issue. In my case, it was volumes with capacities of 500TB+, and we were using block cloning for GFS.

[wcmichael]

Anything you can share Jeremiah? I have an active MS support case and they haven't been able to help so far. If you have a case # you could share or reference to the fix I'd be forever grateful. This bug has been driving me nuts for months.

[dr.action]

I am running into the same issue i guess.
Could anyone provide an URL, Info or something howto receive this private patch?
That would be very kind....

[JeremiahDDS]

I made these changes, along with the patch.

$regPath = "HKLM:\SYSTEM\CurrentControlSet\Control\FileSystem"

New-ItemProperty -Path $regPath -Name RefsEnableLargeWorkingSetTrim -PropertyType DWord -Value 1 -Force
New-ItemProperty -Path $regPath -Name RefsNumberOfChunksToTrim -PropertyType DWord -Value 32 -Force
New-ItemProperty -Path $regPath -Name RefsDisableCachedPins -PropertyType DWord -Value 1 -Force
New-ItemProperty -Path $regPath -Name RefsProcessedDeleteQueueEntryCountThreshold -PropertyType DWord -Value 512 -Force

fsutil behavior set DisableDeleteNotify ReFS 1

New-ItemProperty -Path "HKLM:\SOFTWARE\Veeam\Veeam Backup and Replication" `
-Name FileCacheLimitPercent -PropertyType DWord -Value 0 -Force

[JTT]

Well i applyd latest patches and those registry changes, but still after restart i still see high disk usage.