Comprehensive data protection for all workloads
JTT
Service Provider
Posts: 125
Liked: 5 times
Joined: Jan 02, 2017 7:31 am
Full Name: JTT
Contact:

Server 2025 - high CPU and RAM

Post by JTT »

Hi

I did a fresh Veeam VM install with 2025, installed also 12.3, migrated old config and attached backup data disk over iSCSI, with Refs on it. Old server was 2022, everything was working.
Now there are random crashes, ram usage goes up, CPU is 100% by system proccess. As veeam support suggested, i stopped all veeam services, reconnected iSCSI disk and then CPU went back to 100%. Also when iSCSI REFS backup disk was not connect, with veeam services working,it seemd stable.
Has anybody encountered something like this?

Because of holidays are coming i am thinking of reinstalling server with 2022, but i would prefer not to do it.

Case # 07548410
Steve-nIP
Service Provider
Posts: 132
Liked: 62 times
Joined: Feb 06, 2018 10:08 am
Full Name: Steve
Contact:

Re: Server 20225 - high CPU and RAM

Post by Steve-nIP » 1 person likes this post

Although I haven't tested this exact scenario, it's very unlikely you'll be able to go back to 2022 anyway, the ReFS volume will have been upgraded to a newer ReFS version as soon as it was mounted on 2025, which I highy doubt will work on 2022.
m.novelli
Veeam ProPartner
Posts: 589
Liked: 112 times
Joined: Dec 29, 2009 12:48 pm
Full Name: Marco Novelli
Location: Asti - Italy
Contact:

Re: Server 20225 - high CPU and RAM

Post by m.novelli »

I'm not planning to try Windows Server 2025 until second half of 2025...

Marco
Ciao,

Marco
Gostev
Chief Product Officer
Posts: 32217
Liked: 7583 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Server 20225 - high CPU and RAM

Post by Gostev »

Perhaps it's updating ReFS format? Give it a few days (assuming the volume is quite big).
JTT
Service Provider
Posts: 125
Liked: 5 times
Joined: Jan 02, 2017 7:31 am
Full Name: JTT
Contact:

Re: Server 20225 - high CPU and RAM

Post by JTT »

Well the install was done beginning of december, there has been random crashes, but now almost constanlty the iSCSI disk usage is 100%
I started anther case in Microsoft also
mkretzer
Veteran
Posts: 1253
Liked: 443 times
Joined: Dec 17, 2015 7:17 am
Contact:

Re: Server 20225 - high CPU and RAM

Post by mkretzer »

Is the ReFS upgrade in backround now? Normally it happens at mount time and can take hours...
FrancWest
Veteran
Posts: 540
Liked: 112 times
Joined: Sep 17, 2017 3:20 am
Full Name: Franc
Contact:

Re: Server 20225 - high CPU and RAM

Post by FrancWest »

I had the same issue in my lab. I couldn’t even logon to the server anymore due to the high cpu usage, so I couldn’t see which process had the high cpu. I ended up reformatting the ReFS partition (I used the existing one from Windows 2022) and after that the system is stable again.
JTT
Service Provider
Posts: 125
Liked: 5 times
Joined: Jan 02, 2017 7:31 am
Full Name: JTT
Contact:

Re: Server 20225 - high CPU and RAM

Post by JTT »

Gostev wrote: Dec 19, 2024 3:58 pm Perhaps it's updating ReFS format? Give it a few days (assuming the volume is quite big).
Any suggestion how to check it? FSUTIL showed 3.14 refs version.
ericf312
Novice
Posts: 3
Liked: 1 time
Joined: Dec 21, 2024 11:22 am
Full Name: Eric F.
Contact:

Re: Server 20225 - high CPU and RAM

Post by ericf312 »

I have a similar issue, also on a lab machine. I have a fresh Windows 2025 Hyper-V host with Veeam 12.3 running on a Windows 2025 Guest VM, backing up about 9TiB to a new ReFS 3.14 volume (VHDX). After a few hours to a few days, the CPU usage spikes within the guest and requires a hard reset to get back up.

About a week ago, I created an identical VM with Veeam installed (with no backups configured), and it remains up.
TechMinerUK
Novice
Posts: 7
Liked: 3 times
Joined: Dec 22, 2024 4:42 pm
Contact:

Re: Server 2025 - high CPU and RAM

Post by TechMinerUK » 1 person likes this post

Had a similar issue this weekend in my homelab, installed a Server 2025 server about a month ago and ran Veeam 12.1 and then Veeam 12.3 without any issues. On Friday the server did a Windows Update and since then the RAM has been locked at 90% but CPU is normal.

Anytime Veeam attempts to run a backup the system locks and has to be force reset.

Tried to rebuild the server with a new copy of Windows Server 2025 and the issue occurs before Veeam is installed but as soon as the ReFS volume is connected so I think this may be a Microsoft bug
pesos
Expert
Posts: 227
Liked: 27 times
Joined: Nov 12, 2014 9:40 am
Full Name: John Johnson
Contact:

Re: Server 2025 - high CPU and RAM

Post by pesos »

Interesting. Definitely sounds like an MS bug. FWIW I have multiple hosts and guests now on 2025 and 12.3 with refs repositories but no iscsi - all shared SAS or local storage with no issues.
Andreas Neufert
VP, Product Management
Posts: 7200
Liked: 1547 times
Joined: May 04, 2011 8:36 am
Full Name: Andreas Neufert
Location: Germany
Contact:

Re: Server 2025 - high CPU and RAM

Post by Andreas Neufert »

Just some ideas to isolate the issue and really find root cause:
You can check maybe what is going on by the following:

Even Viewer:
Applications and Services Logs > Microsoft > Windows > ReFS > Operational
Relevant Event IDs:
1101: ReFS initialization
1102: Metadata operations
1103: Integrity scrub or other background tasks

Resource Monitor (resmon).
Navigate to the Performance or Disk tab to observe high activity associated with the target volume. Look for elevated I/O operations involving the ReFS volume or its partition.

PowerShell to get some status and the current load:
# Check disk usage
Get-Process | Where-Object { $_.Name -match "System|refs" } | Select-Object Name, CPU, IO

# Query active volumes for operations
Get-Volume | Where-Object FileSystem -eq "ReFS"
M-M-C
Novice
Posts: 9
Liked: 1 time
Joined: Jul 29, 2019 9:12 am
Contact:

Re: Server 2025 - high CPU and RAM

Post by M-M-C »

I just add my own home-lab findings:
In my home-lab Server 2025 gets stuck after fast cloning on ReFS 3.14. Fresh install, new and empty ReFS partition, local HDD (not iSCSI or anything).
The inital full active backup works fine until the next backup with a fast clone operation. Then it gets stuck.
@Andreas Neufert
Once the server gets stuck there is no way to enter any commands, even the remote desktop session disconnects. Only a hard reset revives the server.
But because my server is virtual I can check the hypervisor for any load. Beside some single bytes there is no disk activity once the server gets stuck (or any other activity besides 100% CPU).
Image
(the red arrow marks the spot when the server gets stuck. Ignore that "Server 2022" text, that's just a typo because I used an old template)

That's pretty much the same error we already discussed back in june:
post521147.html#p521147

Server 2025 (the 24H2 base in general) with ReFS newer than 3.7 is completely unusable right now and has been that way since summer. I had to learn that the hard way, Server 2025 automatically upgraded my ReFS repo after some hours and everything was broken.
I really thought that bug was fixed, it was reported back in june. And the release notes stated that Server 2025 is fully supported...
To be honest I'm quite disappointed of your QA right now because this was a known bug for months....
william.bishop
Enthusiast
Posts: 25
Liked: 6 times
Joined: Aug 03, 2023 8:46 pm
Full Name: william bishop
Contact:

Re: Server 2025 - high CPU and RAM

Post by william.bishop » 1 person likes this post

I can pretty much tell you that you need to go back to 2022 for a while. It will take some time to work out all of the little bugs and gotcha's. I really, really wanted 2025 to be road ready, but in every testing scenario so far, I've gone back to 2022, which runs like a ferrari compared to the new release. I'd recommend not going with 2025 until August at least.
Gostev
Chief Product Officer
Posts: 32217
Liked: 7583 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Server 2025 - high CPU and RAM

Post by Gostev » 2 people like this post

@M-M-C let's run this through support to collect additional performance metrics as actually I feel our QA spent too much time testing ReFS on Server 2025 and yet have never observed anything like this.

The reason they spent more time than usual with this new ReFS version is first testing our support for Azure Stack HCI (where this ReFS version was first introduced), and then later playing with new ReFS dedupe & compression capabilities on preview builds too see if they have any use for us.

In fact, since your test above is extremely basic, they've just done exactly the same test in their lab and did not see any VM lockups either.
Andreas Neufert
VP, Product Management
Posts: 7200
Liked: 1547 times
Joined: May 04, 2011 8:36 am
Full Name: Andreas Neufert
Location: Germany
Contact:

Re: Server 2025 - high CPU and RAM

Post by Andreas Neufert »

Hi Ralf (MMC),

do you have an antivirus/ransomware detection installed on this system and did you set https://www.veeam.com/kb1999 exclusions?
M-M-C
Novice
Posts: 9
Liked: 1 time
Joined: Jul 29, 2019 9:12 am
Contact:

Re: Server 2025 - high CPU and RAM

Post by M-M-C »

No antivirus/ransomware detection installed, not even MS Defender. All my lab tests are "naked".
Lab setup: Proxmox (newest non-substription patch level), unmodified Windows with newest versions of ms visual c++ and .Net (unmodified official installers), VEEAM comunity edition with newest 15.X PostgreSQL. The HDD is made available via sata-controller pass-through, Windows has direct access to spinning drive. It's a 4TB disk, freshly re-partitioned and formated with ReFS, cluster size 64kb.
I can reproduce the "lock up" with every 24H2-Version out there since this summer, Windows 2025 or Windows 11 Enterprise doesn't matter. Windows starts fast cloning on ReFS, the system locks up.
/edit
Just in case somebody was wondering, I even deleted Proxmox and installed Windows on bare metal, same result.
Andreas Neufert
VP, Product Management
Posts: 7200
Liked: 1547 times
Joined: May 04, 2011 8:36 am
Full Name: Andreas Neufert
Location: Germany
Contact:

Re: Server 2025 - high CPU and RAM

Post by Andreas Neufert »

Can you please share server model, controller model and disk type? Is there any raid involved?
M-M-C
Novice
Posts: 9
Liked: 1 time
Joined: Jul 29, 2019 9:12 am
Contact:

Re: Server 2025 - high CPU and RAM

Post by M-M-C »

It's a home lab, the "server" is an old HP ProDesk 600 G2 SFF, consumer grade SATA controller (Q170/Q150/B150/H170/H110/Z170/CM236 Chipset SATA Controller).
The system is on a NVME drive, the larger backup HDD is a Seagate BarraCuda ST4000DM004.
No raid or anything, somewhat old but good for simple testing.

I'm just "glad" to hear that others are having the exact same problem as me, I already started to doubt my reasoning.
SKratzTS
Enthusiast
Posts: 72
Liked: 16 times
Joined: Jan 06, 2017 7:23 pm
Full Name: Steve Kratz
Contact:

Re: Server 2025 - high CPU and RAM

Post by SKratzTS »

At least now I know I’m not going crazy.

I’ve had a 2022-> 2025 server install that was working fine with a SAN refs repository just completely lock up twice now in the last week.

Only things on it were dns and dhcp roles, and a UniFi controller.
ITP-Stan
Expert
Posts: 226
Liked: 65 times
Joined: Feb 18, 2013 10:45 am
Full Name: Stan G
Contact:

Re: Server 2025 - high CPU and RAM

Post by ITP-Stan »

I've reported what I think is the same issue with Windows 11 some time ago.
I'm guessing Windows 2025 uses the same ReFS version as Windows 11 23H2.
veeam-backup-replication-f2/b-r-on-wind ... 95062.html
FrancWest
Veteran
Posts: 540
Liked: 112 times
Joined: Sep 17, 2017 3:20 am
Full Name: Franc
Contact:

Re: Server 2025 - high CPU and RAM

Post by FrancWest » 2 people like this post

As mentioned in my earlier post I was also experiencing this after a fresh install of server 2025 and an existing 2022 refs repository. After formatting the repository on server 2025 everything is stable now for 2 weeks. Fastclone also works fine. To me it looks like the issue only occurs after an existing 2022 refs repository has been converted to the 2025 version.
M-M-C
Novice
Posts: 9
Liked: 1 time
Joined: Jul 29, 2019 9:12 am
Contact:

Re: Server 2025 - high CPU and RAM

Post by M-M-C » 1 person likes this post

@FrankWest
Nope, my ReFS repo was brand new, not converted. In my home lab I always test things from scratch.
I even wiped everything and tried a complete reinstall. Server, repo, Veeam config, everything from scratch. I even recreated the partition table for my backup disk, because I didn't trust a simple re-format.
Let's try and find similarities that could cause the problem, shall we?
My (non-working) repo ist ReFS 3.14 (check with "fsutil fsinfo refsinfo d:"), 4TB spinning disk, cluster size 64kb.
One of the jobs that causes the lock up: agent based backup of a physical workstation, around 1.3TB data, maximum compression and 4mb block size. No anti virus, malware detection, application aware backup, indexing or anything.
Could it be the block size? I don't have time for any more tests, it's christmas, and because I had to quit working because of health issues I don't have access to "real" servers and storages any more.
Also, what kind of CPU are you all using? I'm starting to think it might be an issue with older Intel CPUs, although my CPU is still officially supported (https://learn.microsoft.com/en-us/windo ... erver-2025 ; https://learn.microsoft.com/en-us/windo ... quirements ; https://learn.microsoft.com/de-de/sysin ... s/coreinfo )
ericf312
Novice
Posts: 3
Liked: 1 time
Joined: Dec 21, 2024 11:22 am
Full Name: Eric F.
Contact:

Re: Server 2025 - high CPU and RAM

Post by ericf312 » 1 person likes this post

My machine's specs: Windows 2025 Hyper-V role (in-place upgraded from 2022) on a Supermicro X13SAE-F (BIOS 4.1 with CPU microcode 0x12b) with an i9-13900K, 128GB ECC RAM, OS boot drive is onboard Intel RST VMD SATA RAID1 (2x 480GB SM863A), VMs (including Veeam backup server OS drive, 220GB expanding VHDX) reside on a 20TB LSI 9271-8i RAID60 volume with 16x 1.92TB SM863 SATA SSDs. Repository is on an expanding 40TB VHDX with 32TB partitioned, stored on a 43TB Windows Storage Space mirror (originally created on Server 2022, backed with 4x 24TB WD DC HC580 [onboard] SATA hard drives). Both the host storage space volume and the repo's VHDX are newly formatted from 2025 ReFS 3.14 with 64K clusters and otherwise default settings (e.g. no integrity streams or host dedupe).

The Veeam installations are completely fresh- at first, the virtual machine itself (8 vCPU, ~16+ GB RAM, static vs dynamic didn't make a difference) was repurposed from the old version's backup server, but on my second attempt, I deleted and recreated the VM. Left everything on the install at defaults, only changing the IR cache and guest catalog back to the SSD backed OS volume instead of the larger HDD one. Backup encryption on/off did not matter.

There aren't any meaningful ReFS events logged that appear to correlate with the lockups. I only have event IDs of 4 (successfully mounted), 142 (summary of disk space usage, since last event), and 170 (IO latency summary) on both the Hyper-V host and guest VM.
FrancWest
Veteran
Posts: 540
Liked: 112 times
Joined: Sep 17, 2017 3:20 am
Full Name: Franc
Contact:

Re: Server 2025 - high CPU and RAM

Post by FrancWest »

My lab setup:

- Virtual Veeam server running on VMware ESXi, 8.0.3, 24414501
- Intel(R) Core(TM) i9-9900 CPU @ 3.10GHz
- VM has 2 vCPU's and 36GB assigned
- VM HW Version 21
- OS on SSD, Backup disk 16TB

Guest OS:
- Windows Server 2025 build 26100.2605
- ReFS driver version 10.0.26100.2605
- ReFS filesystem version 3.14, block size 64Kb
- vDisk 14TB
- Windows Defender active and every other security options
JTT
Service Provider
Posts: 125
Liked: 5 times
Joined: Jan 02, 2017 7:31 am
Full Name: JTT
Contact:

Re: Server 2025 - high CPU and RAM

Post by JTT »

So after connecting the iSCSI disk to another server, at some point, the disk usage went back to normal. I tried connecting it back to Veeam server, did a couple of backups and now the disk usage is up again.
SKratzTS
Enthusiast
Posts: 72
Liked: 16 times
Joined: Jan 06, 2017 7:23 pm
Full Name: Steve Kratz
Contact:

Re: Server 2025 - high CPU and RAM

Post by SKratzTS »

I "punted" in my test lab. The Veeam app got its own 2022 VM and storage was wiped to get it back to the "old" ReFS. At least this showed up before I had any management yelling about upgrading the backup server(s) to 2025.
MIDAC
Influencer
Posts: 13
Liked: never
Joined: Jan 01, 2006 1:01 am
Contact:

Re: Server 2025 - high CPU and RAM

Post by MIDAC »

Gostev wrote: Dec 23, 2024 1:32 pm @M-M-C let's run this through support to collect additional performance metrics as actually I feel our QA spent too much time testing ReFS on Server 2025 and yet have never observed anything like this.

The reason they spent more time than usual with this new ReFS version is first testing our support for Azure Stack HCI (where this ReFS version was first introduced), and then later playing with new ReFS dedupe & compression capabilities on preview builds too see if they have any use for us.

In fact, since your test above is extremely basic, they've just done exactly the same test in their lab and did not see any VM lockups either.
Perhaps only an issue when upgrading / with upgraded ReFS volumes? If creating a new ReFS volume using 2025 perhaps there are no issues?
MIDAC
Influencer
Posts: 13
Liked: never
Joined: Jan 01, 2006 1:01 am
Contact:

Re: Server 2025 - high CPU and RAM

Post by MIDAC »

M-M-C wrote: Dec 24, 2024 8:49 am @FrankWest
Nope, my ReFS repo was brand new, not converted. In my home lab I always test things from scratch.
I even wiped everything and tried a complete reinstall. Server, repo, Veeam config, everything from scratch. I even recreated the partition table for my backup disk, because I didn't trust a simple re-format.
Let's try and find similarities that could cause the problem, shall we?
My (non-working) repo ist ReFS 3.14 (check with "fsutil fsinfo refsinfo d:"), 4TB spinning disk, cluster size 64kb.
One of the jobs that causes the lock up: agent based backup of a physical workstation, around 1.3TB data, maximum compression and 4mb block size. No anti virus, malware detection, application aware backup, indexing or anything.
Could it be the block size? I don't have time for any more tests, it's christmas, and because I had to quit working because of health issues I don't have access to "real" servers and storages any more.
Also, what kind of CPU are you all using? I'm starting to think it might be an issue with older Intel CPUs, although my CPU is still officially supported (https://learn.microsoft.com/en-us/windo ... erver-2025 ; https://learn.microsoft.com/en-us/windo ... quirements ; https://learn.microsoft.com/de-de/sysin ... s/coreinfo )
Ah so much for my upgraded ReFS theory. I’m having same lockup issues on a i5-13600k processor and single SATA drive running Windows 11 Enterprise 24H2 installed on a NVMe. System was fine before it auto-updated itself from 23H2 to 24H2. I did a fresh install of 24H2 and Veeam 12.3 but the lockup issues soon started again after a few backups. For now have installed a different HDD formatted as NTFS and unplugged the ReFS drive and the system is stable.
NightBird
Veteran
Posts: 260
Liked: 59 times
Joined: Apr 28, 2009 8:33 am
Location: Strasbourg, FRANCE
Contact:

Re: Server 2025 - high CPU and RAM

Post by NightBird »

Post Reply

Who is online

Users browsing this forum: ldumaine, smarivoet and 360 guests