Server 2025 - high CPU and RAM

[JTT]

Well i also applied the latest patches, but do not see any difference. RAM usage goes up and also disk usage have some constant traffic going on, even without any backup running.

[Gostev]

KB5058411 does not help in our labs either.

[BenM2024]

Yeh guys - I jumped the gun on that one despite being careful to make sure it works properly, rather than just assumng it would. I retract what I said about the LCU making a difference

Monday morning RAM use hadn't moved (but no backup jobs were enabled). I enabled them and re-ran the weekend's backups. RAM crept up somewhat but seemed reasonable. Yesterday it went to 90Gb again after doing the backup jobs.

It still takes ages to restart in the high RAM use situation - yesterday I started the reboot at lunchtime and the machine actually restarted (as in power off and on again) about 2.5 hours later. So I am guessing its file cache flushing, or something similar, slowing things down.

At the minute it seems the only work around high RAM demand is to reboot the backup server each day 3h before the backups are due to start.

Out of curiosity I have removed the FileCacheLimitPercent reg DWORD and am waiting for the machine to restart once more.

[BenM2024]

So the machine has restarted, and in a quiescent state the ram use increases about 1Gb every 5 minutes or fewer.

Looking at the Resource Monitor it appears that it is Veeam.Backup.RestAPIService.exe and Veeam.Backup.Service.exe that have the highest commit charges. 2.5 hours after the reboot the system is starting to hard fault but not much.

The system isn't doing anything (other than running Veeam services - no backup jobs are scheduled - and sending RDP data to my workstation) there are sporadic burst of the system image writing quite a lot of data (upto 90MB/s for a few seconds) to the iSCSI host. The host concerned serves the locations for the backup copy jobs (4 iSCSI targets) which are all ReFS. The 'Master' backup is made to an NTFS volume which needless to say, is not seeing traffic.

[lreed]

I am having similar issues with a new VBR (built as a VM) on Server 2025 with a new ReFS volume. Everything ran fine for about 2-3 weeks and then last weekend it started having issues after the weekly synth full job ran. No changes/windows updates on the VBR etc at that time. No iSCSI or "converting" of the ReFS volume from older to newer version as with some people.

One job in particular seems to triggers things more than the other 2 jobs (either local or the backup copy job pointed at it). Unfortunately that's the big main job we care about the most. As with many others here so long as the jobs are left disabled the VM keeps running fine. I am in the middle of building out a new VM with 2022 and copying the backup chains over.

For what ever its worth we have one other 2025 VBR for a client (all new setup) that has been running fine for a few months with no issues. Its only backing up a single VM tho and only a couple hundred gigs which may be something to do with it behaving (?). Or maybe we are living on borrowed time.

[ChrisAnderson_2019]

Case # 07710245 - ReFS WS2025 BUG

Hi All, I came across this thread after two days of troubleshooting in-place upgrade to Server 2025 from 2022.
(Went down a rabbit hole of clean build, reviewing / updating drivers etc...)

Was at a point where repo server would go unresponsive as soon as it was even re-scanned by VBR.

I attempted the registry key workaround, but it wasn't initially successful.
Following several further reboots (hard stop via iDRAC), the OS remained stable for sometime.

Backups are now processing with success.
CPU and memory are still high, so I've reduced max concurrent tasks to alleviate resource constraints.

Early doors, but I'd certainly recommend adding the "FileCacheLimitPercent" (DWORD, 0) registry key, if you haven't already!!

Cheers and thanks to Veeam support team for their really fast response and escalation with this today.

[BenM2024]

Ok So at the risk of boring you good folk.... it seems to me that the issue really hits when Veeam is doing full backups... what evidence do I have?

Well yesterday I checked 'used' RAM over the day, every hour, and it sat at 38.5Gb. The system has had the registry setting removed and been rebooted prior to the monitoring.
This morning, after incrementals and associated data ciopies to the ReFS volumes over night, it had risen to 40Gb. The last time it used the whole 90Gb was over last weekend - coinciding with the full backups last Friday.

Tonight the full backups are scheduled.

I am back in on Tuesday next week (national holiday on Monday)- will have a look then and restart the server if necessary.
At least now with enough RAM available to actually run the system after a full backup (when I expect the 'Used' RAM to go up to 90Gbish) I can log in to reboot it cleanly rather than waiting for it to bugcheck when Windows decides to try swapping out essential parts of the OS (like VMS used to... many many years ago... told you I was old ).

[jokoenen]

Proceed with fingers crossed!
From my experience with Veeam and ReFS the main problem, which poped up already for several years with Windows Server 2019, was the delete of lage files (several TB). It often happens, that after you or veeam has deleted a huge file, the space is not free, but somehow lost. Not occupied by any files, not in the recycle bin, just not available. On server 2019 it has been typically the size of the files above 4TB (64kB Block size) - so a 6 TB file produced 2TB loss - and I found no way to gain it back but reformat the drive. On Server 2025 it's even worse. after in our case e.g. 16 hours after deleting if we do it manually nothing happens, but space ist lost. Otherwise when veeam is starting the job you don't have this time gap. Nevertheless at some point veeam using the volume triggers ReFS to do something and it ends with a totally blocked computer after a few hours. I'm not familiar with the ReFS details regarding deleting files, but from my experience with HPFS and NTFS it should be doable. But since ReFS is somehow transactional even whe deleting files, therr must be a loop(hole) or something built in and all the occupation of the to be deleted files on the volume is first moved to another directory place to be save on completing of the transactions and then in a second step these "new" direcory entries are removed and the space is marked free in the free space bitmaps. shouldn't be an unsolvable problem, but Microsoft ....
Just my thought on that
Jo

[BenM2024]

Well here I am again, whilst waiting for new copiers to arrive I am poking about with the backup server.

Since Friday the machine has:-

• Performed a full backup
• An incremental
• It is still creating a synthetic full... this is abnormal but it did have issues with one VM that failed on the first attempt to backup but succeded on the retry; this failure to back up first go often happens
• copied the data from am iSCSI ntfs backup repsoitory to the ReFS location (and a VHR - doesn't count cos ZFS in a hardened Linux box).
• system memory used has gone from ~40Gb to ~48Gb.

Hopefully I will still be here when the Synth full has completed so I can reboot the machine and reset the system memory use - it will then last till next week when the school is back in.

[CBHBVT]

The FileCacheLimitPercent registry fix seems to have worked for us. Our daily backups and test restores have worked fine for a week now. No more run away resource usage. This is on Windows 11 PRO 24h2 with REFS.

[BenM2024]

OK Yet more random scribblings

The server failed to survive last week on its own, I rebooted it on Monday with a clean reboot; it took over 8 hours to restart. The machine was doing 'something' all day yesterday but this morning was copletely unresponsive - physical console or remote desktop session so I hard reset it.

The machine came back ok and Veeam started to play nicely. So I re-tried the jobs that failed yesterday.

When the jobs got to making the Synhetic Full Backups on the iSCSI ReFS volumes for the 4Tb main file server VM, they reached 89/90% and stopped responding - it was doing two synths at the same time, to two different ReFS Volumes and two different iSCSI Targets on the same physical NAS box (10Gbps back to back fibre connection).
The machine may well still be doing the jobs (there is disk activty, unlike this morning when there was none) so unless the disks quiesce before end of shift I will leave the machine up tonight and reboot again in the morning.

Hey ho - Patch tuesday is next Tuesday right? I wonder if the ReFS patch will be included....

[m.novelli]

Strange this bug wasn't found during Veeam testing of Windows Server 2025

Marco

[Gostev]

Probably because most systems in our QA lab are pretty beefy in terms of RAM. I know our QA uses some "special lab" where the issue reproduces well to test private fixes from Microsoft.

[adb.scott]

I wish I would have known this issue exists or that Veeam would have warned of it! I spent 6 days building out our new VBR environment and just ran into Server 2025 with ReFS issues on a 100TB datastore! Everything was going wonderfully and all the backups were running (86 of them)! Then all of a sudden this past Sunday everything failed and I have been fighting this issue until I came across this thread! I now have over 65TB of backups that cannot run in anyway! I have applied the registry hack and that seemed to work for about 2 days and then once again I am here! Microsoft punts the issue yet I read on this forum they are aware! I cannot even move backups to another server being the vol is 2025 ReFS 3.14 so I cannot just attach it to a server 2022! This has really borked our business! Before anyone attacks me on testing prior; we did test prior and it ran 15 days in our lab without issues! It seems after day 22 in our production the S hit the fan!

[jokoenen]

Hi Scott!
Just for curiosity: have there been deleted large files around day 22? Like certain full backups after expiration of keeping period.
Jo

[william.scholes]

I would like to share my experience here with Server 2025, maybe someone else can benefit.

I setup a physical HPE server with Server 2025;
4*370TB ReFS repos (2 repos per 800TB RAID 60)
512GB RAM
2*24Core (48 thread) CPUs, for a total of 96 Thread
Initially it was running OK for about 3 weeks... then it started locking up randomly, iLO showed the CPU at 100% and the only way to recovery was to hard reset the server from iLO.
The firmware is latest as of May 2025, with the BIOS on n-1 because the latest BIOS won't install...

At the time it was running Crowdstrike, with all the recommended AV exclusions.
When reviewing the many event logs before the unresponsive issue, I thought it could be related to defrag, so I disabled defrag on all disks, only to find that Server 2025 still defrags the disks even when they are set to not run defrag, so I just disabled the degfrag service.
Well this didn't resolve the issue, but, after uninstalling Crowdstrike, the problem was resolved.

After a couple weeks of stable operation, with no AV installed, I installed Microsoft Defender for Business on the server, and implemented all recommended AV exclusions, so far, its been 4 days, and the locking up issue has not returned.

We don't run any custom reg keys on the server other than S3GovernanceImmutabilityMode.

So maybe it is just AV is the problem?

Also, when I followed the AV exclusions list; https://www.veeam.com/kb1999
I used *:\repo location for the extension exclusions, because I have 4 repos.

For Veeam guys - the Postgres AV exclusion document linked from above, https://wiki.postgresql.org/wiki/Runnin ... s_software
It says to exclude the data dirs and postgresql.exe, however, Veeam's installation of Postgres doesn't have postgresql.exe, it runs from postgres.exe as far as I can tell, can anyone comment on what's going on with that, and what the recommended exclusions for Postgres are, in a default bundled Postgres installation?

Code: Select all

Microsoft Windows [Version 10.0.26100.4061]
(c) Microsoft Corporation. All rights reserved.

C:\Windows\System32>where /r C:\ postgresql.exe
INFO: Could not find files for the given pattern(s).

C:\Windows\System32>where /r C:\ postgres.exe
C:\Program Files\PostgreSQL\15\bin\postgres.exe

[william.scholes]

Well i also applied the latest patches, but do not see any difference. RAM usage goes up and also disk usage have some constant traffic going on, even without any backup running.

I noticed the same strange thing with disk usage but no backups running, I think this is the Windows Defrag service...

[JTT]

Thanks, I´ll try it out!

[adb.scott]

I have all defrag turned off.. I have all AV turned off.. everything was running smoothly for about 3 weeks and then the HIGH CPU and Memory issue landed! I bumped the memory and CPU and everything is consumed no matter how many times I bump it! I'm at 32CPU and 128GB of RAM and still the server freezes after about 30 mins on a hard stop then power back up! This has brought our production backup to a halt and the issue is being avoided by Microsoft!

Has anyone received a HOTFIX from MS or Veeam? I have 170TB of production data that was moved (AFTER LAB) and we are absolutely stuck!

Veeam should ABSOULTEY have clear logic that sees if a backup location is ReFS and running Windows 2025 and HALT all installs with a warning!

[JTT]

Well i reported the issue now almost 6 months ago, but still Microsoft support only sends me a standard response:

I hope this message finds you well. I wanted to provide you with an update regarding the ongoing issue. Our engineer has received information from the Product Group team, and they are still diligently working on resolving it. Unfortunately, there is no estimated time of arrival for the fix at this moment.

Given the current situation, would it be possible for us to temporarily archive the case until we receive further updates from the Product Group team? This will allow us to manage the case more effectively and ensure that nothing falls through the cracks.

Thank you for your understanding and patience as we work towards a resolution. If you have any concerns or need further assistance, please feel free to reach out.

[m.novelli]

Ridiculous Microsoft

Marco

[Gostev]

Since most new posters usually don't read the topic before commenting, I just want to highlight that above there are a few reports of FileCacheLimitPercent registry value helping other users to improve the situation, even if it is a temporary workaround until Microsoft issues an update with the real fix.

[adb.scott]

@gostev I have applied the FileCacheLimitPercent and all worked for 2-3 days and then went right back to the same issue.

[Gostev]

As I explained when I first shared this value, it essentially trades high CPU usage (which causes OS to lock up) to high RAM usage. So there has to be a decent amount of RAM to actually make the trade.

[adb.scott]

@gostev I'm at 128GB of RAM up from the 32GB that worked just fine for 4 almost 5 weeks and was barely utilized even at PEAK.. I have about 30 mins of uptime before the server becomes unusable and Microsoft is of absolute zero help.

[johan.h]

The ReFS memory exhaustion fix is expected from Microsoft in July for HCI23H2 and Windows Server 2025.

[DaStivi]

The temporary fix with that regkey can be done while booting into safemode? or how to apply the regkey if the server is hanging instantly after reboot? disable the data volume in Raidcontroller?

[DaStivi]

ok, i've disable the PCI-Device/Slot with the RAID Controller an was able to boot normally then applied latest windows updates and created the regkey..

[DaStivi]

is there and fixnumber or Case to reference for when contacting Microsoft to aquire the a privat hotfix for this??

workaround with regkey does not here, and not having a backup for another month (at best) isn't a option!

[jokoenen]

Just my observations which might help you.
Until now we didn't had problems with reverse incremental backups on ReFS, since here in our case no huge files are being deleted and the smaller ones don't show these issues.
So we will avoid any deletion of huge files until these MS ReFS bugs are fixed for Win11 and Server 2025.
Our normal backups with regulary full backups to be deleted after a period of time show exactly the described issues with freezing machines after a huge backup file has been deleted.
So we store these kind of backups on a external NAS device.