Windows Server 2022 ReFS: in-place upgrade issue (BSOD boot loop)

[ffwo]

Could someone tell me please, how to fix the Problem?

[Gostev]

Read posts from Andrew@MSFT on the previous page.

[darklogix]

I just came across this thread while searching for a fix for a 2016 server that I attempted to upgrade to 2022 and my account is to new to PM Andrew@MSFT yet.

What happened to me is that I have a server 2016 physical server with 4x 10TB drives in Windows Raid 5 formatted with ReFS and attempted to upgrade to 2022 (I do have a backup but it'll take a long time to restore that way).

The 2022 upgrade failed and after the ReFS volume is now marked as Raw I'm thinking as it was mentioned that ReFS volumes are upgraded to a new ver that may have rendered 2016 unable to recognize the volume.
The backup is a cloud backup so it'll take a long time to download multiple TB of data for recovery.

[Stahlreck]

Anything new on this issue? There's not a lot of information on this one around and I guess there's nothing that can be done without an official fix from Microsoft. Or has anyone managed to get a workaround privately from Andrew?

[RGijsen]

Because with HPE Gen9 servers there are no update packs running Server 2022. All drivers are for Server 2016 or 2019. And rewriting the xml files for all drivers and firmware updates to run under Server 2022 was no option. So we decided to go the inplace upgrade way.

Offtopic but it might help others as not everyone has shiney new Gen10 Plus v2 backup hardware: I've installed a few test machines yesterday with 2022, Gen8, Gen9 and Gen10. Gen 8 and 9 are not officially supported by HPe but SPP installs the drivers just fine. No issues with either server. No need to to in-place upgrades which I personally wouldn't even consider on server systems. See https://www.hpe.com/psnow/doc/a50003760enw, section INSTALLING THE HPE WINDOWS SERVER 2022 OS SUPPORT PACK. SPP is supported with Server 2022, and for our Gen8, 9 and 10 servers it installs all drivers perfectly fine.

[stanthewizzard]

Subsribed

[chris.childerhose]

Veeam 11a is out with support for Windows Server 2022 but does not fix ReFS if upgraded - https://www.veeam.com/kb4215

[Gostev]

Yes, please only use Server 2022 for newly deployed backup repositories for now.

[Andrew@MSFT]

Good news! The fix is now publicly available in KB5005619.

Thanks everyone for your patience. Feel free to let me know of any issues after installing the update.

[Gostev]

@Andrew@MSFT what is the recommended approach of upgrading to Server 2022 in light of this bug?

What you shared seems to be a patch to be installed on top of Windows Server 2022.
But servers get into a boot loop the moment Server 2022 upgrade is completed.
Would not this prevent users from installing the patch in the first place?

[Andrew@MSFT]

Thanks for asking Gostev!

Broad steps are:
1. Disable (read-only) the ReFS volume/disk.
2. Apply the KB.
3. Re-enable the ReFS volume/disk.

I'm working to get the steps documented into an associated KB article and will post back when that becomes available.

[DonZoomik]

Alternatives would probably pre-patching install media (integrate patches into WIM with DISM) or apply patch offline in WinPE with DISM (if in trouble after the upgrade).

[COCONET]

Good news! The fix is now publicly available in KB5005619.

Thanks everyone for your patience. Feel free to let me know of any issues after installing the update.

I finally updatet all our servers with the latest patch and re-mapped the storage LUNs to the cluster. Everything fine until now.
So no bluescreen or even a bootloop. But when it comes to reading the disks: file system: RAW.

So the ReFS volume is not readable. Windows does recognizes the volumes as ReFS, but gives an error:
Volume "\\?\Volume{684d1176-3d96-4c94-9326-3d2ba0b89587}" ist als ReFS formatiert, es kann jedoch von ReFS nicht eingebunden werden. Von ReFS wurde der Status "{Gerät ausgelastet}
Das Gerät ist momentan ausgelastet." erkannt.

I am still unable to read the cloud backup files located on the LUNS.

Any suggestions @Andrew@MSFT?

Thanks!

[Gostev]

This may indicate that the patch is unable to recover volumes which were hit by the bug in the ReFS format upgrade logic.
It would appear that in your case at least, the bug left metadata of impacted volumes in an inconsistent state.

Try this tool and see if you can still salvage some data:
https://docs.microsoft.com/en-us/window ... s/refsutil

[COCONET]

Update:
Our bluescreen bootloop happened when I assigned the disks to the cluster, and not while mapping the fc storage to the physical nodes. So therefore I unmapped the disk again because removing from the cluster wouldn't be possible when you stuck in a bootloop. After unmapping the disks, the cluster complained about the missing disks but to keep everything in the same condition I haven't removed the disks out of the cluster. Installing the patch from Microsoft was the first step, afterwards I remapped the disks and at least no bluescreen shown up again. But in the diskmanager the "refresh disks" told me all disks are available again but in RAW. I haven't tried to change anything about the disks at this time so I thought the patch is not working.
The happy part now is: Taking the disks offline, removing them out of the cluster and retake them online on only one node made my day! Windows regained access to the disks with file system shown as ReFS. Upgrading the complete Veeam infrastructure to v11a reattaching the disks to the VMs and everything works fine now.

Thanks to Andrew@MSFT and Gostev for supporting here!

[Gostev]

Phew! I'm very glad to hear that the original bug does not seem to cause a data loss... that would be really bad.

[hACK]

Got RAW error after upgrading WS2019 to WS2022.
Provided version of refsutil doesn't support ReFS 3.7 Any way to get updated version of this tool?

Code: Select all

refsutil salvage -D D: C:\salvage -x -v
Microsoft ReFS Salvage [Version 10.0.11070]
Copyright (c) 2015 Microsoft Corp.

Local time: 9/30/2021 16:45:29

Option(s) specified: -v -x

ReFS version: 3.7
Error: The volume is an unsupported ReFS version. This utility supports versions up to 3.5. Volume is 3.7.
Error: Command failed.
Error: The volume does not contain a recognized file system. Please make sure that all required file system drivers are loaded and that the volume is not corrupt.

Run time = 2 seconds.

###

[System.Environment]::OSVersion.Version

Major Minor Build Revision
----- ----- ----- --------
10 0 20348 0

[mkretzer]

"This comes in Windows 10 in the %SystemRoot%\Windows\System32 folder or in Windows Server in the %SystemRoot%\System32 folder."

It should be preinstalled....

[FrancWest]

Got RAW error after upgrading WS2019 to WS2022.
Provided version of refsutil doesn't support ReFS 3.7 Any way to get updated version of this tool?

Did the upgrade complete successfully? Are you running this tool from WS2022? I also had the situation once when the upgrade failed, so it reverted back to WS2019, but it had upgraded the ReFS volume to the latest version already which WS2019 was unable to mount. After I ran the upgrade again and WS2022 was successfully installed, I was able to access the reFS volume again.

[hACK]

Upgrade passed successfully, all other things works ok. Preinstalled refsutil running from WS2022.

[datacomm]

Thanks for asking Gostev!

Broad steps are:
1. Disable (read-only) the ReFS volume/disk.
2. Apply the KB.
3. Re-enable the ReFS volume/disk.

I'm working to get the steps documented into an associated KB article and will post back when that becomes available.

@Andrew@MSFT did you ever get a KB article posted? Would like something official from MS before we attempt this.

[schewee]

I've got KB5005619 installed and now I'm back to the RAM starvation issues ReFS used to have where doing certain operations, like making large synthetic backups, basically bogs down the system and makes the Veeam console unusable until the synthetic job finishes. Wonder if Microsoft also re-introduced that bug? Anyone else running into this problem too?

[Gostev]

No, we're not seeing such issues with 2022 so far. However, it was not a "bug" that can be easily "re-introduced" to start with... rather, it was just sub-optimal engine of working with ReFS metadata that received a massive re-write in a few stages: first in 2016, then in 2019 and final polish in 2019 SAC releases, changes from which were rolled into 2022.

@All please stay on topic going forward and don't discuss any issues in this thread which are not directly related to the original BSOD boot loop issue.

[ksl28]

Hi,

Are there any update to this matter?
When looking over the latest comments, the upgrade approach from MS dosent seem to stable.
Also when looking at the release notes for the update https://support.microsoft.com/en-us/top ... 995515c791 ReFS is not mentioned anywhere.

[Gostev]

Hi, I don't expect any further updates on this matter since the above-mentioned update has resolved the issue discussed in this topic. Thanks!

[hACK]

ReFS is damaged after upgrade. Then new version of refsutil will be released?

[Gostev]

refsutil is provided by Microsoft so you should check with them. Although in your situation I would rather open a case with Microsoft Support, instead of waiting for some tool to be released.

[ksl28]

Hi, I don't expect any further updates on this matter since the above-mentioned update has resolved the issue discussed in this topic. Thanks!

So even that the update is still an preview, and the KB from microsoft does not contain one word about ReFS - you can confirm this update fixes the boot loop?
And are these steps still required to be performed?
1. Disable (read-only) the ReFS volume/disk.
2. Apply the KB.
3. Re-enable the ReFS volume/disk.

@Andrew@MSFT

[Gostev]

Sure it does. If it did not fix the issue we would for sure hear about it too many times by now, instead of zero posts since November.

Gotta love the law of large numbers... it is actually one thing that makes Veeam super awesome: with over 1 million of active backup servers, any real issue immediately results in many similar posts and support cases. So when you run into an actual bug, you're rarely alone.

[doktornotor]

Ok, just to be really clear. Using this ISO for in-place upgrade, there should not be any issues with this?

Code: Select all

en-us_windows_server_2022_updated_feb_2022_x64_dvd_d4a089c1.iso