Server 2025 - high CPU and RAM

[alexandre.nakagawa]

Hello gostev and Johan
I installed a new Veeam server and physical proxies (256GB RAM) with Windows 2025.
I mapped the LUNs from the old servers (Windows 2022 REFS) to the new server.
We migrated on Friday, and early this morning, the server started freezing, with high CPU and memory usage.
I have repositories of 300TB each, with approximately 1PB of data in one of the repositories.
I applied the FileCacheLimitPercent registry keys, but the server remains unstable after 30 minutes of uptime.
Is there any Microsoft fix I can request when opening a support case with Microsoft?
Would you be able to provide this fix if I open a P0 support case with Veeam

[karsten123]

why do you not just wait for the official fix?

[Gostev]

Unfortunately we're never able to provide private fixes we receive from Microsoft for validation, as it's an intellectual property that does not belong to us and therefore we cannot redistribute it.

[DaStivi]

Can you post some reference number to help with a case at Microsoft to get the private hotfix without any more hassle ?

Waiting another month isn't a option for many if the server just stops responding without being able to create backups... And no none of us is willing to install another Server any maybe loose some weeks or months of backups ..

[alexandre.nakagawa]

why do you not just wait for the official fix?

Because my server is unresponsive and I missing almost 1PB of backups... the server run only for about 4 hours and freeze, it has 256GB o ram used.

and the sql databases are becoming unavailable per lack of space on transaction logs directory since the backup logs are not running, (our databases have 200TB each) and we do 30 min log backups.

[karsten123]

BEFORE jumping no scope on the 2025 train

… at least, its a valuable lesson

[alexandre.nakagawa]

I have other 3 veeam repositories that was migrated to 2025 and was running fine for a long time. this is our only server that had this bug, problably because the size of backups and repositories, small repo servers is working fine

[johan.h]

Understand that this is very frustrating. We were given KB9178699 at the time, private fixes should be requested from Microsoft (its their code after all). If that doesn't work use the private message feature to reach out to me and maybe I can help you push a little.

Separately, Alexandre, I've worked with your Veeam Support engineer on this one and I believe they have a plan in place for you.

[alexandre.nakagawa]

Ok, thanks johan, i received the update from veeam eng.

[BenM2024]

The ReFS memory exhaustion fix is expected from Microsoft in July for HCI23H2 and Windows Server 2025.

That's good news - if it works

FWIW I still can't run a backup (to an NTFS drive) becasue the ReFS bug hangs the server. It seems the 5ish Gb of ram left after ReFS takes it's bit (even when Veeam is disabled)
is not enough memory for a backup to a local NTFS volume.
I might get the opportunity to rob some RAM form another server next week and see if a few more Gb provides enough space for backups to run - it does seem that the ReFS bug maxes out at 90Gbish RAM in my use case. Its worth the downtime on the donor server I feel.

[BenM2024]

I might get the opportunity to rob some RAM form another server next week and see if a few more Gb provides enough space for backups to run

Well that went as expected - even with 160Gb RAM in the server it dies whilst doing the backup job. All other jobs disabled. Roll on July Patch Tuesday (8th isn't it?)

[johan.h]

From what I heard from Microsoft, I would expect a little later in July. But yes, not so far away now.

[alexandre.nakagawa]

My workaround was to mount the iSCSI disk on another machine with 512GB of RAM (a 300TB ReFS volume).
After many hours, I was finally able to access the disk directories.
I then copied the VIB/BKP files to another NTFS disk.
The server crashed multiple times even with 512GB of RAM, but I gradually managed to copy each file.
I lost several backups, but they were weekly backups that would have been deleted soon due to retention, so I only copied the critical ones.
I was able to copy approximately 150TB.

[DaStivi]

i've got a privat fix from microsoft that looks like to fixed the issues... i only had about 10 or 15mintues until the server freezed...
ms-support told me the fix is planned to become publicly available in August ...

[Gostev]

From what I heard, the fix requires changing one registry setting in addition to deploying actual binaries. Did you do both things?