Server 2025 - high CPU and RAM

[JTT]

I tried it, didnt help at all. After reconnectiong old iSCSI disk, the RAM ja DISK usage goes up, even without creating any new backups on it.

[Gostev]

But are there still OS lockups? This is what we're fighting here, not the fact that ReFS needs to do some work on its metadata after seeing a new volume attached.
Also, do you have a case open with Microsoft on this? If yes, please share the Microsoft case ID.

[JTT]

After of couple of hours, servers is working, but i´ll update the status tomorrow. But still iscsi disk usage is almost 100%, full gigabit link is used, so how to solve the Refs problem? I cannot even copy backups off it.

[Gostev]

That's why I asked about Microsoft case as they would be able to tell what exactly ReFS is doing at the moment, whether it is normal (for example, ReFS format upgrade) and if so, when it's going to finish.

[FrancWest]

I’ve implemented the registry key because the last 2 days I have lockups again. I only have a 1TB repository in my lab. The system was upgraded to Windows 2025 5 weeks ago so any ReFS background activity should have finished now.

I’ll monitor the system with the registry key in place.

[Gostev]

We've also been testing this registry mod internally and it seems to effectively "exchange" high CPU load to high RAM load. So clearly there's some underlying issue with certain ReFS activity causing this compute spike. We've shared this with our ReFS dev contacts at Microsoft and found out that they we've already been actively researching the issue based on a support case one of our customers has opened with them a while ago (THANK YOU!)

[JTT]

I have added the same registry key. After of couple of days, creating some regular incrementals and synth full, everything seems normal.

Also I created a support case at Microsoft, 19.12. But they wanted to kind of blame iSCSI storage vendor
Support request number: 2412190050002026

[Gostev]

Microsoft has informed us that they've confirmed the related ReFS bug and are working towards resolution. We will try to get clarity on the fix timelines and what OS versions the fixes will be available for at the next scheduled call between our teams.

[johan.h]

We currently believe this might be an issue in builds of Server 23H2, Win 11 23H2, and AZSHCI 23H2 and higher. Once fix validation takes place, we'll be able to learn what the release cadence of these fixes will be.

[JTT]

Any ETA for the fix? My Microsoft case is silent

[simcfc73]

I'd be interested in an ETA for the fix also, Mine was locking up after a few days for the past few weeks but now it's locking up 5 minutes after a reboot.

Haven't tried the reg fix yet but my old server only has 8GB ram.. I have ordered more.

[marksf]

For those who have been having this issue, is it just the Veeam Backup & Replication that has high CPU/RAM /locks up or is it the Hypervisor as well if using Hyper-V 2025?

[ericf312]

In my case, the hypervisor is fine- it's just the guest B&R machine that's affected.

[DaStivi]

Server 2025 brought a new dedup feature for ReFS that's based on the block cloning feature veeam also leverages.... You can only set this with wax and the get/set-refsdedup Powershell cmdlets... Don't use the server manager and dedup role from fileserver role! Even more you should not mix these! The old/traditional dedup stores the chunklets in the system volume Information folder .. the new refs-dedup has a different logic!

Maybe , if you use this new refs-dedup you might run into the same issue!?

[Ken.Ellis]

We are experiencing this issue on a brand new clean-installed fully updated Windows Server 2025 physical server running Veeam B&R 12 and a 140 TB repository volume formatted REFS + 64k. The repository volume is on local RAID50 disks inside the server, no iSCSI involved at all. The server was working fine for the first couple weeks of active full and some incremental backups. Shortly after the first synthetic full or other REFS block-cloning operations started though the server ground to a halt and quickly became completely unresponsive with just a black screen showing on the iDRAC and only hard power cycling to get it started again. The server has 64 GB of RAM but even after rebooting in Safe Mode (no VEEAM services at all) Windows would completely fill all 64 GB within a minute or two of booting and go unresponsive again. We tried the FileCacheLimitPercent = 0 registry setting described here which did seem to cause it to behave slightly differently with the CPU/RAM usage but it still went completely unresponsive within minutes of startup. Eventually I had to boot to safe mode with CMD prompt and use DISKPART to force delete the REFS volume and even that I only barely managed to do quickly enough before the server hung entirely. As soon as the REFS volume was gone everything went back to normal. For now we have reformatted the repository NTFS and everything is working but it's not a good solution since synthetic fulls take days to complete and will rapidly fill the repository compared to REFS.

We tried contacting Microsoft support but they refused to talk to us or let us open a case without paying for per-incident. I'd just like to find out is Microsoft is in fact actively working on a patch for this and if there's any timeline for when they might release it?

[masterchief07]

Check HKEY_LOCAL_MACHINE\SYSTEM\Software\Microsoft\TIP\TestResults\ to see if there are literally hundreds of thousands of records.

This was a known bug in 2022 and I'd be curious to see if its plaguing 2025 as well.

I had to set up a script to auto clear out that reg entry regularly to prevent build up and eventual crashing.

Interestingly enough, I am running into another major PIA with 2025 coming from 2022 w/ regard to RDS. There is a major memory leak with the svchost.exe for terminal services. Still fighting it.

Anyways, hopefully the TIPResults thing is your silver bullet!!

[Ken.Ellis]

I checked that TIP\TestResults registry key but it's empty, no entries. The only thing that "helped" in my case was completely deleting the REFS volume -- everything instantly went back to normal as soon as it was gone, and has been working as expected with NTFS thus far but I can't keep it like this since the synthetic full operations take far too long to complete and we don't have disk capacity to keep enough history without the block cloning features of REFS. I was really hoping we'd get *some* kind of guidance from either VEEAM support or Microsoft about when they expect the patch for this to be ready, but I can't get any sort of answers out of anyone. I can't even find any official acknowledgement from Microsoft that they're *actually* aware of this issue and working on it, they don't list it as a known issue anywhere I can find. I'm about to give up and wipe/rebuild the entire thing with Server 2022.

[Ken.Ellis]

Just a quick update for everyone here -- after failing to get any sort of useful response from Microsoft Support, I went ahead and wiped & rebuilt our backup server with Windows Server 2022 instead of 2025, and reformatted the repository back to REFS / 64k. As I described in my previous posts it had become completely unusable due to this apparent bug with REFS on 2025. So far we've done a couple synthetic full (which seemed to be the thing that triggered the problem via REFS block cloning) backups and everything appears to be working smoothly on Server 2022, no high CPU or RAM issues at all.

[ITP-Stan]

I came to that conclusion over a year ago when running in to this problem with Windows 11.

[johan.h]

Just to keep this thread updated. We were given an initial fix to try, but it didn't quite work. But will let you know when we receive a working one from Microsoft.

[JTT]

Well my MS support case gets only a regular response: Our Product Group team is actively working on this case, and we will update you as soon as we receive any updates from our team.

[Gostev]

Good news, Microsoft rootcaused this to a small coding mistake with a simple and low risk fix. We're testing the new hotfix already and it looks promising.

[johan.h]

After some further testing there were some additional issues found with the new hotfix, we're now waiting for an update from Microsoft.

[Gostev]

Specifically the fix has failed long-term stability test, happy these are standard in our QA as the fix was looking very good otherwise.

[wallstrum]

@gostev - is this a hotfix for the Veeam side or the Microsoft side?

[Gostev]

Microsoft, this is related to ReFS metadata caching logic.

[wallstrum]

@gostev - thank you!

[william.scholes]

Any ETA from Microsoft on this hotfix? will it be a download on request or a GA patch for Server 2025?
I'm about to put a new 1PB backup server into production in 3 days and I can't afford problems

[Gostev]

Then you shouldn't use Server 2025 at this time, as whenever it comes I can't see it fully tested in 3 days.

[karsten123]

then better go the XFS route