B&R on Windows 11 VM - freeze/lockup caused by ReFS?

[ITP-Stan]

I have a customer that has a Windows 11 Pro virtual machine running on ESXi v8 where Veeam B&R is installed.
Backup storage is mounted in Windows using iSCSI. There are 2 volumes.
- One with NTFS used as repository for the VM Backup Job.
- One with ReFS 64K used as repository for Backup Copy Job with GFS virtual fulls.

The VM would lock up completely, can't RDP in to it, VM console shows a login screen that is frozen showing a date/time from when it froze.
ESXi performance monitor will show problem with the VM, constant 100% virtual cpu usage, vmtools not responding.

Windows event logs show nothing that stands out leading up to the freeze.

I have a suspicion that high I/O load on the ReFS volume is causing the freeze.
I was just wondering if anyone else has experienced this?

I know that ReFS on iSCSI is probably not supported and not the best idea
I know that Windows 11 can't officially format a volume in ReFS but there workarounds for that, once the volume is there it works.

Thanks for your time!

PS: The customer did not renew their socket based licenses in the past, we have taken this customer on recently and are in the process of acquiring Veeam VUL licenses.

[david.domask]

Hi Stan,

Since ReFS isn't really natively supported on Windows 11 Pro, I'm afraid that likely Veeam Support won't be able to assist much on such a configuration as if the issue is related specifically to this configuration, we will be at a dead end.

I cannot say I've heard of this specifically with ReFS volumes, but my guess is that it is related to the configuration, but cannot tell you what to check -- is there any chance to re-implement the ReFS repository as a simple linux repository with an XFS volume? That would give you Fast Clone and any supported distribution should be able to serve as a Fast Clone capable repository, which avoids this Windows 11 ReFS hack.

Perhaps others can comment on the specific issue if they've experienced it, but I think the best answer is just to get a supported configuration, and you should be able to save on license costs with a Linux Repository; this also opens up the possibility of a Hardened Repository, which means immutability for the backups.

[doktornotor]

I know that Windows 11 can't officially format a volume in ReFS but there workarounds for that, once the volume is there it works.

Well, actually you can...Officially.

https://learn.microsoft.com/en-us/windows/dev-drive/

[ITP-Stan]

I have turned off the copy job last week and the freezes no longer happen.
So I'm pretty sure it's somehow ReFS related.

[marcos0317]

Hi,

I am encountering exactly the same issue as Stan is mentioning.
The difference is that my setup is a physical backup repository with a Dell Optiplex 7010.
The hardware configuration is the following :
CPU : I5 13500
RAM : 64 GB
OS Storage NVME SSD 256 GB
Backup Storage is an ISCSI target formated as REFS with 64k cluster, volume size 40 TB

Software :
Windows 11 Enterprise (REFS is natively supported on it)
Veeam components version : 12.2.0.334
Veeam Roles installed : Backup Repository, backup gateway, backup proxy
REFS version 3.10
REFS Registry keys :
RefsDisableDeleteNotification=0x1
RefsDisableVolumeUpgrade=0x1
RefsEnableLargeWorkingSetTrim=0x1

A single backup task is saving data on this backup repository with a GFS retention 3years, 7 months, 5 weeks, using fast clone.
The solution has been working fine during almost 2 months, but since last windows cumulative updates of september and october we have encountered issue.
We also have an equivalent solution but on Windows 10 Enterprise which is working perfectly fine, now since almost 2 years for the same customer.

The symptoms are the following after a reboot of the computer :
- Windows 11 boots fine
- The iscsi refs volume is correctly mounted
- Then Windows is doing some big reads on the iscsi target (around 50 MB/s)
- The memory is filled with REFS metadata (checked with RamMap from sysinternals)
- Then system is writing REFS metadata at maximum target storage speed (around 220 MB/s)
- After a while (somewhere between 20 minutes and an hour) the system freeze
- The ram is not exhausted before the freeze usage is between 25 GB and 45 GB
- A few times I noticed a CPU usage of 100% just before the freeze, so hard to detect
- The Windows event log doesn't report pertinent information
- If the Iscsi volume is disconnected, the computer doesn't freeze.

Other information of interest :
The windows 11 refs.sys driver is updated in the october cumulative updates from version 10.0.22621.4111 to version 10.0.22621.4317
Probably to correct a vulnerability in refs : https://msrc.microsoft.com/update-guide ... 2024-43500
ISCSI driver has also been updated by Microsoft in cumulative update of July also to correct a vulnerability : https://msrc.microsoft.com/update-guide ... 2024-35270

I already tried to reinstall windows 11 directly using Microsoft iso september cumulative state, to avoid Dell Bloatware, this has lead to no change in the behaviour.
I have disabled Windows automatic updates, I have blocked REFS version upgrade in the regsitry

I have a case opened with Veeam, case number : 07473723
So far I didn't receive any help, just a few advice on some questions I Asked.
That it's not recommended to upgrade to windows 11 24H2, not supported by Veeam yet.

So any help or advice would be appreciated

[ITP-Stan]

It's always with ReFS on ISCSI and always Windows 11.

The first time I noticed this problem was august 2023, with Windows 11 Pro.
That time I downgraded to Windows 10 Pro (had to reformat the ReFS volume).
Last time I saw this (post above) I recognized this must be a problem with the specific version of ReFS in Windows 11.
That time we used Server 2022 instead (had to reformat the ReFS volume).

I wonder if this problem will pop-up in Server 2025 now.