B&R on Windows 11 VM - freeze/lockup caused by ReFS?

[ITP-Stan]

I have a customer that has a Windows 11 Pro virtual machine running on ESXi v8 where Veeam B&R is installed.
Backup storage is mounted in Windows using iSCSI. There are 2 volumes.
- One with NTFS used as repository for the VM Backup Job.
- One with ReFS 64K used as repository for Backup Copy Job with GFS virtual fulls.

The VM would lock up completely, can't RDP in to it, VM console shows a login screen that is frozen showing a date/time from when it froze.
ESXi performance monitor will show problem with the VM, constant 100% virtual cpu usage, vmtools not responding.

Windows event logs show nothing that stands out leading up to the freeze.

I have a suspicion that high I/O load on the ReFS volume is causing the freeze.
I was just wondering if anyone else has experienced this?

I know that ReFS on iSCSI is probably not supported and not the best idea
I know that Windows 11 can't officially format a volume in ReFS but there workarounds for that, once the volume is there it works.

Thanks for your time!

PS: The customer did not renew their socket based licenses in the past, we have taken this customer on recently and are in the process of acquiring Veeam VUL licenses.

[david.domask]

Hi Stan,

Since ReFS isn't really natively supported on Windows 11 Pro, I'm afraid that likely Veeam Support won't be able to assist much on such a configuration as if the issue is related specifically to this configuration, we will be at a dead end.

I cannot say I've heard of this specifically with ReFS volumes, but my guess is that it is related to the configuration, but cannot tell you what to check -- is there any chance to re-implement the ReFS repository as a simple linux repository with an XFS volume? That would give you Fast Clone and any supported distribution should be able to serve as a Fast Clone capable repository, which avoids this Windows 11 ReFS hack.

Perhaps others can comment on the specific issue if they've experienced it, but I think the best answer is just to get a supported configuration, and you should be able to save on license costs with a Linux Repository; this also opens up the possibility of a Hardened Repository, which means immutability for the backups.

[doktornotor]

I know that Windows 11 can't officially format a volume in ReFS but there workarounds for that, once the volume is there it works.

Well, actually you can...Officially.

https://learn.microsoft.com/en-us/windows/dev-drive/

[ITP-Stan]

I have turned off the copy job last week and the freezes no longer happen.
So I'm pretty sure it's somehow ReFS related.

[marcos0317]

Hi,

I am encountering exactly the same issue as Stan is mentioning.
The difference is that my setup is a physical backup repository with a Dell Optiplex 7010.
The hardware configuration is the following :
CPU : I5 13500
RAM : 64 GB
OS Storage NVME SSD 256 GB
Backup Storage is an ISCSI target formated as REFS with 64k cluster, volume size 40 TB

Software :
Windows 11 Enterprise (REFS is natively supported on it)
Veeam components version : 12.2.0.334
Veeam Roles installed : Backup Repository, backup gateway, backup proxy
REFS version 3.10
REFS Registry keys :
RefsDisableDeleteNotification=0x1
RefsDisableVolumeUpgrade=0x1
RefsEnableLargeWorkingSetTrim=0x1

A single backup task is saving data on this backup repository with a GFS retention 3years, 7 months, 5 weeks, using fast clone.
The solution has been working fine during almost 2 months, but since last windows cumulative updates of september and october we have encountered issue.
We also have an equivalent solution but on Windows 10 Enterprise which is working perfectly fine, now since almost 2 years for the same customer.

The symptoms are the following after a reboot of the computer :
- Windows 11 boots fine
- The iscsi refs volume is correctly mounted
- Then Windows is doing some big reads on the iscsi target (around 50 MB/s)
- The memory is filled with REFS metadata (checked with RamMap from sysinternals)
- Then system is writing REFS metadata at maximum target storage speed (around 220 MB/s)
- After a while (somewhere between 20 minutes and an hour) the system freeze
- The ram is not exhausted before the freeze usage is between 25 GB and 45 GB
- A few times I noticed a CPU usage of 100% just before the freeze, so hard to detect
- The Windows event log doesn't report pertinent information
- If the Iscsi volume is disconnected, the computer doesn't freeze.

Other information of interest :
The windows 11 refs.sys driver is updated in the october cumulative updates from version 10.0.22621.4111 to version 10.0.22621.4317
Probably to correct a vulnerability in refs : https://msrc.microsoft.com/update-guide ... 2024-43500
ISCSI driver has also been updated by Microsoft in cumulative update of July also to correct a vulnerability : https://msrc.microsoft.com/update-guide ... 2024-35270

I already tried to reinstall windows 11 directly using Microsoft iso september cumulative state, to avoid Dell Bloatware, this has lead to no change in the behaviour.
I have disabled Windows automatic updates, I have blocked REFS version upgrade in the regsitry

I have a case opened with Veeam, case number : 07473723
So far I didn't receive any help, just a few advice on some questions I Asked.
That it's not recommended to upgrade to windows 11 24H2, not supported by Veeam yet.

So any help or advice would be appreciated

[ITP-Stan]

It's always with ReFS on ISCSI and always Windows 11.

The first time I noticed this problem was august 2023, with Windows 11 Pro.
That time I downgraded to Windows 10 Pro (had to reformat the ReFS volume).
Last time I saw this (post above) I recognized this must be a problem with the specific version of ReFS in Windows 11.
That time we used Server 2022 instead (had to reformat the ReFS volume).

I wonder if this problem will pop-up in Server 2025 now.

[MIDAC]

We are seeing this same issue now too on a physical PC with a single HDD formatted as REFS. Tried a full reinstall of Windows 11 24H2 along with Veeam 12.3 and all seemed well for a few days until the issue started occurring again. Has there been any progress on what is causing the issue and what can be done to resolve? Apparently ReFS on Windows 11 24H2 got the new Block Cloning feature which was already present in Server versions since 2016. Is this what is causing lockups when Veeam is using the drive?

[M-M-C]

It's still completely broken: veeam-backup-replication-f2/server-2022 ... 96912.html
I just gave up and rolled everything back to Server 2022 and ReFS Version 3.7. Everything newer is affected by the lock ups.

[MIDAC]

I ended up installing a new HDD formatted as NTFS and moved the jobs to it, then unplugged the ReFS drive and so far so good, no lockups. So is this a Microsoft issue or a Veeam issue or just a deadly combo? Is Veeam aware of the issue and working on a fix or at least pressuring Microsoft to fix their drivers?

[Gostev]

We have not yet observed any issues with the latest ReFS version in our QA labs however we're testing ReFS on Server 2025. I can say that from your description it sounds very much like a Microsoft issue that is completely identical to the one from early days of ReFS.

Our QA engineers have been actively trying to reproduce the issue in the last few days by repeating the same steps affected customers are performing but so far no luck.

[Gostev]

Happy New Year all !!

We have an idea on what might be causing this and it is directly connected to the amount of physical memory on the system, which would explain why it affects some systems but not other. Specifically, it could be a side effect of us limiting the system file cache size.

The reason we've been limiting file cache size all these years is to work around some Memory Manager bugs that caused excessive memory consumption/leaks. However there's a chance this is no longer needed with Server 2025 and Windows 11 24H2 due to on-going bug fix and optimizations by Microsoft (we have not specifically tested this yet).

To disable this functionality, create FileCacheLimitPercent (DWORD, 0) registry value under the HKLM\SOFTWARE\Veeam\Veeam
Backup and Replication key on the backup server, restart the affected backup repositories and see if this helps.

[marcos0317]

New updates
I am still on Windows Enterprise 23H2. REFS 3.10

One month ago, I disabled some useless services on Windows, then I rebooted the computer.
All backup tasks toward this backup repository were already disabled, since the issue appeared I didn't run any backup related tasks.
The symptoms listed above where still occurring, but somehow the computer managed to run during maybe 10-12 hours without a freeze.
The ram usage was high around 60 GB, the read and write on the REFS ISCSI target were still present (why? no backup task is running), some high CPU usage also occurred, and it ended up after whose 12 hours by BSOD.
After this last BSOD, the computer rebooted and the symptoms didn't occur anymore, no read or write to the storage, no high ram usage or high CPU usage.
It's like the issue never existed, the system was now running perfectly.
So I re enabled the backup task. It ran fine, the incremental were running fine (daily), the weekly synthetic full (with fast clone), monthly backup health check was fine too.
So it ran fine during more than a month since the issue stopped, but it was to good to be true, on the 5th of January at around 11 AM the system started to experience exactly the same symptoms again, so I disabled the backup task. Same as before if the ISCSI REFS target is disconnected, the system is perfectly stable.

It's truly driving me crazy

[MavhRik]

Happy New Year all !!

We have an idea on what might be causing this and it is directly connected to the amount of physical memory on the system, which would explain why it affects some systems but not other. Specifically, it could be a side effect of us limiting the system file cache size.

The reason we've been limiting file cache size all these years is to work around some Memory Manager bugs that caused excessive memory consumption/leaks. However there's a chance this is no longer needed with Server 2025 and Windows 11 24H2 due to on-going bug fix and optimizations by Microsoft (we have not specifically tested this yet).

To disable this functionality, create FileCacheLimitPercent (DWORD, 0) registry value under the HKLM\SOFTWARE\Veeam\Veeam
Backup and Replication key on the backup server, restart the affected backup repositories and see if this helps.

My Windows 24H2 with B&R and ReFS on VMware ESXi was unusable, but this seems to have fixed it!

[Gostev]

It's a workaround rather than a fix, see my last few posts here for where the real issue is.