Server 2025 - high CPU and RAM

[jokoenen]

Hi all!
Today I tested the Windows version 26100.6584 on both, W11 and Server 2025.
I started with the W11 machine 32GB Ram, Intel core ultra 7 155U with 12 cores, 14 logical CPUs. On a ReFS volume with veeam backups - full and incremental - I deleted one folder with 9 TB size.
It took 76 minutes to free up the space of the deleted files. During this time period the CPU load was between 13% and 40%, memory usage between 60% and 86%, writing speed approx 96% or 90 MB/s, which is near the maximum and reading speed 2-4 MB/s during all the time it took.

So far so good, but the long time it took, might cause some problems when space is needed or if speed slows down due to e.g. parallel backup writing.

Afterwards I tried it on a Server 2025 deleting 6.5TB of veeam backup files stored in a different directory.
Intel Xeon E-2224 CPU 4 cores, 32GB RAM. It started quite ok with 34% CPU and 60 % RAM. The free space increased slowly over time similar to the W11 machine, but then after 10 minutes it happened:
The remote console became non responsive. I waited for another 40 minutes - no real change, only the time display in the bottom line changed a little bit was far - 26 minutes - behind reality.
I decided to unplug the HD and connected it back to my W11 PC. Here the cleanup process continued immediately and finaly the space became fre, but 93GB stayed reserved and for System.
In the meantime the Server 25 recovered by itself and I could check the eventlog. But nothing meaningful has been in there, only, that the timelimit 30000ms of a transactionresponse has been reached.

So in summary, there is some progress how deleting of huge files is processed, but the long time and relatively high loads it takes to free up the space is not acceptable. The freezing of the Server 2025 is definetly beyond acceptance. I did not any tuning in the registry, so this might reduce but ybe not solve the problems.

So my advice regarding ReFS volumes:
Avoid deleting of huge files - mine have not been really huge since still below the 4TB border.
So reverse incremetal backups and full backups with incremetals following are ok. But don't delete a full backup to built up a new one. Don't replace an old backup with a new one on the same device. This only works after formatting it.

Hope this helps.
Jo

[ChristophINV]

@jokoenen

Thanks, had nothing to delete yet and as I'm running on Reverse-Incremental right now, that's helpful to have in mind!

So I "just" finished the 69 TB-Job (Full, 40 hours) and one Reverse-Incremental step on Top (Server 24H2 Build 6584 as well) without any problems. System peaked at 24 GB of RAM, went back to 14 GB once done. Guess I can downsize it back again from its current 64 GB. Reporting CPU-Load during operation is pointless for me, as it is where it is supposed to be at Max-Compression (28 Threads Xeon Gold 6132) as I'm somewhat Storage-Performance-Limited anyway (300 MB/s Limit on the ReFS-Repo, but that's going constantly). System did not crash, that's a good start. As an Increment is done within three hours, I'm scheduling that thing for the upcoming week and monitor how it goes.

Seems like KB5065426 did something. Time will tell. Hopefully Microsoft will improve things before someone needs to delete something.

[Flole]

From my experience deleting works but it requires that the OS is restarted a few times in between to clean up the memory. I deleted 5 TB of data and rebooted the machine 3-4 times while it was reclaiming space. No registry tweaks, just rebooting, and eventually the deleted files space was fully reclaimed.

[Heimer-BEAS]

I thought the "private" fix was supposed to go live in August?

[HenrikS.]

Seems we hit this behaviour with large memory usage, 100% CPU and system lock.

New AIO server with Windows Server 2025, set up a couple of weeks ago, 2xReFS volumes of 250TB each, no initial tuning and September 2025 patches installed.
Problems started yesterday after having changed retention period for a large job, resulting in deletion of about 20TB of data.
Server froze some hours later. Forcefully rebooted, would then start working on freeing up space again, but would lock up after about 30 minutes with memory usage of 190GB and 100% CPU on 24 cores.

Checked some of the registry tweaks,
RefsEnableLargeWorkingSetTrim=1 (This was set by default it seems, it was already there and I cannot remember to have set it earlier.. ?)
RefsEnableInlineTrim=1 (I set this after a couple of forceful reboots)

RefsEnableInlineTrim seemed to do the trick, after setting this, the server would not lock up after a reboot, but continued to work on freeing up the space. Memory usage was still <190GB, but CPU usage was way lower and it did not lock up.
After it finished some more hours later, the memory usage dropped to 43GB while idling. It'll be interesting to follow the next run..

[m.novelli]

I'm wondering how Microsoft cannot manage to solve what looks like a commodity: delete a huge amount of data from a filesystem

Marco

[jokoenen]

Hi!
First I want to confirm, that the registry entry RefsEnableInlineTrim=1 is working an prevents the server from freezing.

Nevertheless deleting huge files takes a lot of time. On my small veeam server approximately 1.1-1.2 TB are set free per hour. That means after deleting the space occupied by the deleted file is marked as "system reserved" in the advanced properties of the drive. During all this time the system is permanetly writing on the drive at maximum speed. The total amount written is aproximately 33% of the file size of the deleted file.

I think this is caused by the CoW policy:

"Copy-On-Write (COW) is an update policy that may be used to alter data on a storage medium. A COW
update policy makes sure that data is never updated data in place. Whenever the contents of a block should
be altered, the block is read into memory, modified and its contents are written to an alternate location on the
storage medium." (from Technical Reports / ISSN 2191-5008 https://d-nb.info/1201551625/34)

Since the meta data of the file consists of B+-trees, which has a lot of leaves if it's a big file, the system writes the whole new tree to disk for every cutted leaf. So this data is scaling with (number of leaves) times (number of leaves).

I also tried to reduce the number of leaves by using explorer's copy function to copy a huge fie. The required space on the destination disk is immediately occupied. But the effect on the duration of the deleion and free up space process is very limited (10%).

So any backup process requiring deletion of huge files is affected an will need much more time e.g. several hours depending on filesize.

MS has to solve this! Can anybody put some pressure on them?

Jo

[jokoenen]

Sorry, I have to correct myself: during the time 'I wrote my previous post, the sever became frozen after 3 hours working on freeing up space form a 7.3 TB file
Jo

[JaySt]

@Gostev , please help me understand:
i think i'm running into this issue (again... flashbacks to the server 2016 thread...) . This seem to be a known issue for quite some time for 2025?
Isn't this an issue "bad" enough to at least put something about this in the helpcenter system requirements for this OS for the time being? word of caution perhaps? Or something in the known issues list on the forum?

[m.novelli]

Indeed...

[BenM2024]

OK, Im trying to follow Ben (not on the Registry-Tweaks yet)....

I wouldn't

The server is misbehaving again. I did get a couple of backups in and, when it started showing symptoms I disabled the backup jobs, but now it stays responsive for barely 30 minutes after a hard reboot.

Currently trying to get it stable enough to install October's patch Tuesday patches just in case something is fixed... as someone upthread says its amazing that Microsoft can't get a File system to cleanly delete files - I expect the name the picked for it didn't help (Re in ReFS is Reliable IIRC).

As others have mentioned its definitely related to the clean up after the expiry of retention periods (hence deletion of huge chunks of data).

ETA: I last rebooted it yesterday morning. It has been doing something ever since - the disks are showing activity so I am reluctant to hard reset it; i will check again in the morning. Normally it gets so bogged down that the drive activity lights stop blinking....

[JaySt]

so, meanwhile, Server 2022 would be the go-to most stable version if Windows is a must?

[jokoenen]

Unfortunately there is no way back, because you can not read data from any ReFS disk Server 2025 has been mounting using Server 2022.

[JaySt]

true, but we have an option to evacuate a SOBR extent from a 2025 to a 2022 server perhaps.

[JaySt]

@Gostev , please help me understand:
i think i'm running into this issue (again... flashbacks to the server 2016 thread...) . This seem to be a known issue for quite some time for 2025?
Isn't this an issue "bad" enough to at least put something about this in the helpcenter system requirements for this OS for the time being? word of caution perhaps? Or something in the known issues list on the forum?

i stand corrected for most of this. There is a KB on it: https://www.veeam.com/kb2792
could use a bit of an update perhaps and a link to it on the sys requirements for repositories though...
Seems like "new-install only server 2022" is the current go-to, if you must...

[Flole]

Re in ReFS is Reliable IIRC

It's resilient, not reliable. So it can stay broken until December, nobody said it would be reliable

[BenM2024]

>>So it can stay broken until December, nobody said it would be reliable


And December is only a NFB (not fixed before) date...

[jokoenen]

December which year?
resilent regarding improvements and not reliable - that's true

[ChristophINV]

Backup-Server (2025) is still behaving since applying the Sep-Patchday. No Registry-Edits. Re-Enabled Defender with ATP again last week (disabled that for Troubleshooting), added a NAS as 2nd Repo, Secondary-Backup-Chain, Replication of the first Backup-Chain (ReFS) to NAS and a Cloud-Repository. Everything is working as expected. Restore-Point-Max reached. In Reverse-Incremental at least, don't have space due to "reasons" for doing it like I should.