Server 2025 - high CPU and RAM

[HenrikS.]

After 1 months without issues, we had the freezing starting again after another round of deletion of a large set of restore points.
I'll give the 8 feature overrides regfix a try.

[BenM2024]

Hey guys,

After the last weekend's backup runs/backup copies/VHR copies & SureBackup tests, everything is still stable in Veeam Server/ReFS land.
I guess I will have to give it a few weeks now to see what happens at weekly/monthly retention rollover times.

Regards

[JaySt]

I've seen 18 days of stable processing right now after the overrides. Looking good so far.

[1dna]

we just encountered this today

Server: HPE Alletra 4140
OS: Windows 2025 - Latest Updates

We had latest firmwares and everything.

We ran this:

Code: Select all

# --- REFS Registry Tweaks ---
$fsPath = "HKLM:\SYSTEM\CurrentControlSet\Control\FileSystem"
 
New-Item -Path $fsPath -Force | Out-Null
 
Set-ItemProperty -Path $fsPath -Name "RefsEnableLargeWorkingSetTrim" -Type DWord -Value 1
Set-ItemProperty -Path $fsPath -Name "RefsDisableCachedPins" -Type DWord -Value 1
Set-ItemProperty -Path $fsPath -Name "RefsEnableInlineTrim" -Type DWord -Value 1
 
 
# --- Feature Management Overrides ---
$fmPath = "HKLM:\SYSTEM\CurrentControlSet\Policies\Microsoft\FeatureManagement\Overrides"
 
New-Item -Path $fmPath -Force | Out-Null
 
$values = @(
    "593004686",
    "815149711",
    "861440142",
    "882258575",
    "1150694031",
    "1230538894",
    "1398311054",
    "1633192078"
)
 
foreach ($val in $values) {
    Set-ItemProperty -Path $fmPath -Name $val -Type DWord -Value 1
}
 

Run and reboot - our main alletra went from 150gb+ Memeory to around 15-20GBs

I hope this helps someone.

Note: These Tweaks is collected through the forums so not sure if all is needed but we were out off options once this kicks in.

[karsten123]

@veeam official kb?

[Gostev]

This is anything but official so no... not until Microsoft is confident enough about these behavior mods to include them in the official update by default.

[karsten123]

sad. but i understand. its not veeams fault

[JaySt]

This is anything but official so no... not until Microsoft is confident enough about these behavior mods to include them in the official update by default.

Honestly, still dont fully understand why you support server 2025 (as in: listed as supported os in the docs) under these circumstances.
With v13, you had the chance to delist it (if it wasnt possible for whatever historical reason in v12).

[Gostev]

Why wouldn't we support it? We have current issues with ReFS on 2025 documented but it's not the only file system one can use on it, nor repository is the only possible role for it.

[iDeNt_5]

I agree with JaySt, or at least I would say it should be better to add a "*" after the OS WS 2025 version to add a statement about this point, in order to inform the people about this.

[JaySt]

Why wouldn't we support it? We have current issues with ReFS on 2025 documented but it's not the only file system one can use on it, nor repository is the only possible role for it.

when looking at the system requirements for a repository, can we say Veeam is doing the best it can to prevent the deployment of ReFS on Windows 2025? because that's what should be done here. I don't think so.
Yes you document a few issues in a KB, but it's not there right in your eye when checking the docs.

[Gostev]

add a "*" after the OS WS 2025 version to add a statement about this point, in order to inform the people about this.

Personally I don't dislike this idea, why not add this or any other relevant information - it certainly would not hurt to have it. You can use Send Feedback at the bottom of the relevant manual page (where you believe this information fits best) and hopefully our technical writing team will be able to add the KB link there.

[Gostev]

can we say Veeam is doing the best it can to prevent the deployment of ReFS on Windows 2025? because that's what should be done here. I don't think so.

I'm guessing you're assuming this issue plagues every Server 2025 deployment of ReFS, however this is simply not the case. There're seemingly a number of variables in play. Of our own QA labs, we have only one where the issue is consistently reproducible and we cherish it as the apple of one's eye as to be able to validate fixes from Microsoft (luckily it's a VM).

[JaySt]

That’s the same reasoning as when one would assume all 2025 deployed systems with issues are known to Veeam.

We have got 9 pages of forum posts through quite some time and heavy troubleshooting including microsoft involvement. Running into these issues in all cases is a real pain in the ass. And its frustrating for the ones that have the “if i had known…” feeling after coming here.

All this, imho, would justify Veeam doing more than it does now to inform customers planning deployments.

Given the trackrecord of ReFS stability, Microsoft with ReFS should not deserve Veeam’s support without remarks/footnotes for the repository role.

[m.novelli]

Personally I don't dislike this idea, why not add this or any other relevant information - it certainly would not hurt to have it. You can use Send Feedback at the bottom of the relevant manual page (where you believe this information fits best) and hopefully our technical writing team will be able to add the KB link there.

The warning should be in the GUI when creating a new ReFS repository, not in the manual...

Marco

[Gostev]

Warnings in GUI are for permanent situations only, as they can't be easily updated or removed.

[m.novelli]

Well Windows Server 2025 is out since one year... quite permanent, yes

[Gostev]

We're in the same position as everyone who opened a support case on this issue i.e. we're also told at any point in time that "the next month's Windows update resolves the issue".

[JaySt]

unfortunately, today we had to hard reset the repository again. So despite the registry settings , including the ones for the overrides, still no joy.
Probably due to the GFS rollup for last month.

[Martin@SHH]

Hi all,

Same experience as MILJW002!
Every Saturday night between 00:20 and 00:40 AM the veeam cleaning task kills our Windows Server 2025, only ping is functional.
Server is physical with a QNAP-ISCSI Repository and Refs.
Problems occur not with the backup job nor NAS neither LTO (SAS), but with clean up task !
Tried also some disable Teaming 10Gbit-Broadcom Network adapters, changes in netzwork drivers with offloading, deactivating System Raid on SSD's.
Nothing helped!

Is there a possbility to make the clean up task less aggressive or to slow down?

LG
Martin

Hi,

I was having the experience of when there was a cleanup task to remove old backup chains the server would become unresponsive. Still respond to PING, but unable to RDP, all remote services would be accessible, the remote management agents wouldn’t report, and if I tried to login to the console a black screen.

If I left if for enough hours it would usually come back. This was only related to backup jobs using ReFS volumes. So, I’m thinking this experience is an example of the ReFS code regression in Windows 2025. If I looked at the Event Log when I could eventually get in there was a large gap in the logs (covered the unresponsive period).

I have tried usual monthly updates, and currently running with some of the recommended registry entries (which I’ll call workarounds). I also removed some older backup chains a month or so ago to try and reduce the situation that triggers the ReFS bug.

I just checked the Event Log on that server and don’t see any examples of that error message you’ve listed.

[Martin@SHH]

PS: Was someone experimenting with reducing max. concurrent tasks in the backup repository or lower retention time in backup jobs? What was the result?
LG
Martin

[DaStivi]

btw. you can experience some similiar issue when using Hyper-V S2D and REFS and using REFSDedup...
after a while when the refs dedup runs, memory starts to rise (from the ReFsDedupSvc.exe) and after it hit 100% ram utilization the dedup jobs errors and fails..
the s2d volume becomes shortly unavailable, Vms might start to hang...

have this experienced now on 2 different systems and also found a blog post on technet: https://learn.microsoft.com/en-us/answe ... memory-and

[BenM2024]

It been OK here since I added the overrides... also server patched with November's Patch Tuesday patches.
Just to jinx it, I upgraded the infrastructure to Veeam B&R 13 today, as over the weekend may retention deletes were carried out and the server still behaves as expected.

[DaStivi]

we just encountered this today

Server: HPE Alletra 4140
OS: Windows 2025 - Latest Updates

We had latest firmwares and everything.

We ran this:

Code: Select all

# --- REFS Registry Tweaks ---
$fsPath = "HKLM:\SYSTEM\CurrentControlSet\Control\FileSystem"
 ..........

thx for the "script"

BUT !!!! be carefull, or please someone from veeam, correct the scritp! there is a major issue in it!

the command

New-Item -Path $fsPath -Force

creates a new reg-key ("folder) and deletes the existing one! ("HKLM:\SYSTEM\CurrentControlSet\Control\FileSystem")

and there are some preset regkeys init, that are gone after running this command!

the "-force" parameter should be removed (results in a warning if the key allready exist), or even the complete "new-item for the filesystem" key shouldn't be run at all!