Server 2025 - high CPU and RAM

[m.novelli]

What is the trigger to this issue? iSCSI repository? or very huge repository?

Marco

[Gostev]

Neither. The working theory is that some improvements in metadata handling logic in Server 2025 made ReFS caching so much metadata that it's causing system to go out of memory. ReFS does have a cache throttle, but certain code path is apparently ignoring it, thus the issue.

[m.novelli]

So far I have installed two or three (little) Veeam Backup Server with Dell hardware, PowerEdge T360, and I din't incur in the issue

Finger crossed

Marco

[debouy007]

Hi,
is there any update or fix regarding this issue? We are also experiencing this.

[Gostev]

See my post a few hours ago

[SpikeAUS]

Any update on this?

Having the same issues, VM locking on backup, typically 4/5th backup. Have to reset it via ESXi.

CPU at 100% when looking at it on ESXi.

Windows 11 24H2 26100.3775

REFS Volume Version : 3.14

[Gostev]

Updates will be posted as soon as we have them. Why would we start keeping them secret all of a sudden?

[wallstrum]

Hi,
is there any update or fix regarding this issue? We are also experiencing this.

This (I'll call it a workaround until the official hotfix comes out) worked for me on a Server 2025 setup that was experiencing the same freezing issues posted by others as well:

post536519.html#p536519

Wally

[SpikeAUS]

Thanks, Wally, will give this a go. Anyone else really not like REFS? I've only had issues with it, it's been out for a while now, but still feels like a beta product.

[mkretzer]

@SpikeAUS At the current rate of issues i would rather host a Linux virtual machine with XFS on the Veeam Server (under Hyper-V) than use ReFS directly...

[debouy007]

This (I'll call it a workaround until the official hotfix comes out) worked for me on a Server 2025 setup that was experiencing the same freezing issues posted by others as well:

post536519.html#p536519

Wally

Thanks Wally.
We are using Server 2019. Will this also work?

[ITP-Stan]

Server 2019 should not be affected by the same issue that we're talking about here (only Server 2025 & Windows 11)

[jokoenen]

Hi all!
Mid March 2025 we had upgraded our veeam server to Windows Server 25, since the old 2019 Server was not able to read the ReFS filesystems from backed up Server 2022 machines.
Afterwards we noticed also these freezes related to ReFS.
Does anybody know, if these issues had been fixed in this update I installed recently?
"2025-04 Kumulatives Update für Microsoft server operating system version 24H2 für x64-basierte Systeme (KB5055523)"
Thanks
jokoenen

[Gostev]

Hi, Server 2019 being unable to read Server 2022 ReFS volumes is not an issue but rather the expected behavior: ReFS driver in Server 2019 simply does not have any knowledge of newer ReFS versions. Thanks

[jokoenen]

Hi everyone!
Looking bachwards, the incompatibility of the different ReFS versions is a feature, not an issue. What was surprising, that filesystems of backups of 2022 ReFS done with den 2019 server have not been readable with the 2025 server. So the requrements of ReFS are very strict. So everything has to fit: the creator of the filesystem, the backup maker and the backup reader. Luckily NTFS seeams to be compatible between different OS version.
LG
Jokoenen

[karsten123]

so what is the actual recommendation for new deployments right now? go with 2022 for ReFS repositories and all other infrastructure components?

[Gostev]

Correct.

[jokoenen]

On our Windows 25 serer veeam has to delete some files (6TB) from a ReFS volume befor doing a full backup. This hasn't worked well, since afterwards 4TB showed as system/reserved. After 1 hour backuping the CPU load increases up to 100%, but storage still ok with 72% usage.Computer was still up somehow and taskmgr was able to updates every few minutes. Eventlog-ReFS noticed waiting time more than 200s. After 3 hours eventlog noticed latency of 9186s for cleanup, DASD and file open. After that a lot of messages occured about timeouts of services, the last one afre 8 hours has been:
Anwendungspopup: svchost.exe - Systemfehler: Das System hat in dieser Anwendung den Überlauf eines stapelbasierten Puffers ermittelt. Dieser Überlauf könnte einem bösartigen Benutzer ermöglichen, die Steuerung der Anwendung zu übernehmen.
Application popup: svchost.exe - System error: The system has detected the overflow of a stack-based buffer in this application. This overflow could allow a malicious user to take control of the application.

Had to reformat the disk using an older W10 machine not aware of ReFS 3.14, since W11 machines also died with this now damaged RefS HD.

So MS still has to to something about ReFS and Server 2025.
LG
jokoenen

[BenM2024]

For what it's worth - I have a DL380 Gen 10 running Veeam B&R accessing ReFS volumes via iSCSI that now reboots itself every few hours. Each reboot I get a few minutes to log in and see whats going on, then things go rappidly downhill as the machine starts swapping everything to disk. Eventually the Kernel gets fedup and bugchecks.

I have disabled all Veeam backup jobs, in one of the server's more lucid moments, and had hoped that would sort things out somewhat, however that isn't the case.
I am hoping to install the server tool that allows you to restrict the working set of all processes so I can, hopefully actually log in and see what's what; I don't care if things take a long time, I just want the server to be available!

Like another thread contributor, I cannot - owing to lack of other storage - copy data off the ReFS volumes and re-install WS2022, reformatting them with the old version of the OS. Anyone got a spare 32Tb disk array they want to get rid of?

I am hoping that Microsoft fix this next week (May "Patch Tuesday")... otherwise backups are going to be err.... interesting.

[jokoenen]

Maybe the rebooting is caused by a watchdog, since I had always to press the reset button to get an unfrozen machine.
Jokoenen

[vmtech123]

Happy I stayed on 2022. I'm a bit shocked by the amount of people updating their critical backup infrastructure to 2025 so soon.
Remember this when the next version comes out... 1 behind is a wonderful spot to be for stability, firmware, OS, feature updates. Unless a CVE of 10 comes out this is the way, Even still, if it's isolated enough, don't be first.

[BenM2024]

>>Maybe the rebooting is caused by a watchdog,
Yeh - I suspect it is the server completely running out of RAM and starting to swap rather critical services to disk which then didn't complete in time - it may even have tried to swap the swapper (that was something VMS used to do if it really got out of memory)

I have increased the pysical RAM from 32Gb to 96Gb and, after 30 minutes, or so the used memory is at 45Gb and climbing slowly. After boot, RAM use started at about 6Gb and then as Veeam services (specifically Veeam.RestAPI Service) started the RAM demand increased fairly rappidly. I will enable the backup jobs again, do a clean restart at the end of the day and hopefully there will be eough RAM to enable the backups to complete. Hopefully Postgres doesn;t object too much tot he reboots - it is all on local storage so as long as that survives I should have some backups tonight - win

>>Remember this when the next version comes out...
Heh - one would have thought that I would remember it - having done more OS upgrades than enough over the very many years I have been managing servers.

I thought "how bad can it be? if it fails then just roll back" however I reckoned without Microsoft breaking RefS and also making ReFS dependant on OS version... once you have upgraded the OS (and hence ReFS) you can't go back. The lack of downgradability Its a serious issue that Microsoft won't address because the FS is behaving exactly as advertised in that respect - outwith their inability to update the internals without breaking stuff.

I could have set up a test rig and seen how things went - no, wait... that would have required upgrading ReFS so rolling back the OS was imposible.
I could have left off upgrading to 2025 - fair call. However internally getting one physical server on 2025 was a bit of a win and had to be done sooner or later.

[karsten123]

just wait at least 1 year. after that you can evaluate a upgrade. no rocket science

[BenM2024]

>>just wait at least 1 year. after that you can evaluate a upgrade. no rocket science
Look sonny, I have probably had more breakfasts than you by a long way don't talk to me about waiting a year, I may not have that long!
as for rocket science - yeh, but that sort of falls into the shoulda woulda coulda bucket and we are, as they say, where we are.

Anyhow

Server was restarted using the windows GUI at 1520 yesterday with memory use at around 45Gb and climbing gently. It had not restarted by end of shift (15:30) and nor had it restarted by the scheduled backup start times. It had restarted by 2200 when a Surebackup job was scheduled start; that job promptly got upset becasue things weren't as it expected.

I checked the machine this morning and memory use is constant at around 90Gb "used" and 5 Gb "free" - quoted because those concepts are a bit vague when it comes to memory management in Windows. The machine is, at least, usable and will happily retry backup jobs. Yes there are issues (not being able to publish malware/indexes to the catalog - known feature) but it works after a fashion.

Memory demand doesn't appear to change significantly during backup operations so, for my use case, there is a finite amount of memory required to run Veeam with Server 2025 and ReFS file systems presented, to the server, using iSCSI.

I will leave it be until it runs the scheduled tasks over the weekend.