BEWARE: possible RAW ReFS volumes after installing January updates

[FrancWest]

Hi,

Today I noticed that after installing the January Windows Updates released yesterday, ReFS volumes came up as RAW. This luckily it wasn't on our Veeam backup server since that one hasn't installed the updates yet. On Reddit I read reports from users that experience the same issue.

https://www.reddit.com/r/exchangeserver ... _database/

The above post is about Windows 2012 R2, but I had the issue on a Windows 2022 server.

Uninstalling the update (KB5009555 in case of Windows 2022) brings the ReFS volumes back up again.

So be cautious not to install these updates on your Veeam server until further notice.

[SKratzTS]

Microsoft has pulled a good one this month. It appears that ReFS drives break after applying the patches released yesterday. I had two USB ReFS backup destinations change to RAW.

Reddit has a number of posts regarding this, and other major breakage going on: https://www.reddit.com/r/sysadmin/comme ... _20220112/ You can remove the bad update and get ReFS drives back. Hopefully you see this and don't do something destructive trying to repair any backup drives you have!

[Gostev]

Franc, do you also use a USB hard drive for ReFS repository?

[FrancWest]

No, we don’t. I had this issue on 2 servers using ReFS. One fileserver running Windows Server 2022 containing the FSLogix user profiles and one test server running Windows Server 2022. Both servers have fixed drives, so no USB.

Our Veeam server wasn’t affected sine I unapproved the update for installing using our WSUS server after I noticed the issue.

I just thought to give a heads-up since ReFS is being used heavily in combination with Veeam.

[HannesK]

just as a note: I installed the KB5009555 update and my REFS volume is still accessible. But my lab is relatively empty, nothing special installed there.

[Gostev]

KB5009555 has a bunch of ReFS CVEs and ALL of them require "physical" attack vector (meaning direct storage manipulation). This makes external drives the best exploit tool for these vulnerabilities: you just bring a ReFS-formatted USB drive containing necessary "tweaks", plug it into the system and take over it. So Microsoft could close all 7 vulnerabilities (and probably many other similar ones yet to be discovered) by simply restricting ReFS driver to fixed media only.

This does not explain though why Franc (and some other folks on Reddit) also have issues with fixed drives.

[Gustav]

We run a handful of physical servers, 2019 and 2022. All updated yesterday.
Each server hosts an internal and an external VBR repository (USB harddrive). All repositories are ReFS formatted.

The internal drives are not affected by the update.
Two servers have "lost" the ReFS partition on the external USB drive. They now appear as RAW. These two servers are of different brands.

Would there be an official link to the documentation of the buggy update? Indeed when to expect it to be resolved?

[Mildur]

Would there be an official link to the documentation of the buggy update? Indeed when to expect it to be resolved?

Best to ask in a Microsoft forum.

They have pulled now the faulty updates:
https://www.bleepingcomputer.com/news/m ... -bugs/amp/

[FrancWest]

KB5009555 for Windows 2022 doesn’t seem to have been pulled. My test server still downloads this update from Windows Update. However, this is the faulty update which causes the raw ReFS issue.

[Mildur]

Yes, you are right, looks like they pulled only the other one.

[candersoncch]

So, Microsoft has a KB article up about this, which states that ReFS is not supported on external drives:

https://support.microsoft.com/en-us/top ... 9b0ee86c65

I hope that they intend to fix this. Despite it being "unsupported," (something I didn't know, and I'm not sure how I could have been expected to know since it's not like Windows warned you if you tried to format an external drive with ReFS), it has worked perfectly fine for years. I have a bunch of USB external drives we rotate through as backup repositories and to move all of the data, reformat to NTFS, and copy it back would be extremely time consuming.

[FrancWest]

The issue did occur on 2 fixed drives for us, not external USB drives. So there must be some sort of situation where it also occurs on fixed (internal) drives.

[popjls]

I have KB5009557 installed on my 2019 ReFS Veeam server and it is all working fine. I did install it on the 13th AUS time so it might have been re-released and then picked up in my time zone. Unsure whether to remove it or not now.

[dkvello]

I had an SQL server (last week, running on WS2019) at a customer site that lost all its ReFS drives after MS "fixed" things. They where all "local" disks under vSphere, nu external stuff.
What puzzled me was that when I tried instant recovery (this server is backed up every other hour), the instantly recovered server had all ReFS drives come up as Raw.
But, Veeam could easily access and restore all the files from the same recovery-point when using file-level restore.

I tested adding a new virtual disk and formatting it as ReFS and that disk worked just fine. So, since nothing had been added the databases since the last recovery-point I just removed the old ReFS disks from the running SQL server and created new ReFS disks. Then it was just a few minutes to "Restore files to original location" in Veeam and everything was up and running again.

After that I went hunting for "why" this happened.

[FrancWest]

My experience is that it doesn't happen with every ReFS volume. There must be some special circumstances why the fixed ReFS volumes come up as RAW after installing the update. Today I also noticed that another lab machine running Windows Server 2019 experiences the same issue. After uninstalling KB5009557 the ReFS volumes where accessible again.

The reason Veeam is able to recover files from it is probably because of the fact that the proxy which is used to mount the volume isn't affected by the ReFS issue.

[Gostev]

Exactly. More specifically, it's the mount server associated with the backup repository.

[m.novelli]

So, Microsoft has a KB article up about this, which states that ReFS is not supported on external drives:

https://support.microsoft.com/en-us/top ... 9b0ee86c65

I hope that they intend to fix this. Despite it being "unsupported," (something I didn't know, and I'm not sure how I could have been expected to know since it's not like Windows warned you if you tried to format an external drive with ReFS), it has worked perfectly fine for years. I have a bunch of USB external drives we rotate through as backup repositories and to move all of the data, reformat to NTFS, and copy it back would be extremely time consuming.

Yes, me too I have a dozen customers with RDX drives formatted ReFS 64K to speed up Veeam Backup Copy Job

Marco

[m.novelli]

BTW , my company manage hundreds of VM and physical servers for many customers. I had 4 Domain Controller with the endless reboot for 2 different customers , VM in vSphere , nothing special installed, just DC. Luckily no issues with ReFS RAW volumes

I still wonder why in 2022 Windows Update doesn't offer two basic settings

- wait xxx days before installing updates
- install updates at this specific hour and reboot immediately

Marco

[GMGinger]

To possibly explain the difference between some ReFS volumes becoming unavailable and others working fine - there were comments on Reddit that suggested that it was only upsetting servers which had ReFS filesystems that had been created on older versions of Windows. I'm sure Windows will upgrade the ReFS version for a filesystem when you first mount it, but perhaps there's some left over artifact from the previous ReFS version that catches the updated ReFS driver out.

[ryan_gcits]

Hello all,

Looks like MS has release an out-of-band patch to resolve the bug with REFS formatted removable media.

https://support.microsoft.com/en-au/top ... 6a9ca52da7

[bytewiseits]

Had a smaller affected Windows 10 21H2 Repo with an internal disk and USB disk, both with ReFS. USB drive had showed as raw until uninstalling the 2022-01 CU a few days ago.
Installed the 2022-01 CU (KB5009543) again and confirmed USB drive RAW again, then installed OOB update KB5010793 and USB drive is now accessible again. Looks like they have fixed it for Win10 which is promising at least.

[FrancWest]

This OOB patch doesn’t solve the RAW ReFS issue with fixed drives we are having. Volume is still shown as RAW after installing KB5009555 and KB5010796.

Reverting back to working state without KB5009555 and KB5010796.

[corbitech]

Per message posted on 1/17/2022 from Windows Message Center, Microsoft's OOB patch currently applies to most but not all versions of Windows 10 and Windows Server. The complete list of supported OS is available under "Recent announcements":

https://docs.microsoft.com/en-us/window ... age-center

Unfortunately, Windows 10 version 1809 and Windows Server 2019 currently are NOT on the list of OS included in the OOB patches.

My Veeam backup host is running Windows 10 Enterprise LTSC (version 1809), so I'm out of luck at the moment. After the January Windows Update was installed on my Veeam backup host, my removable hard drive (originally formatted with REFS) is showing up as "RAW". I had been using the removable hard drive in a set of rotated drives for Backup Copy Jobs (for offsite archival).

I hope Microsoft releases an OOB for Windows v1809 soon, or I will have to ditch REFS and use NTFS for removable drive. When I set up the rotated drive repository for the Backup Copy Jobs, the Veeam application recommended REFS with 64k allocation unit size instead of NTFS because REFS supported "fast cloning". I really don't want to give up REFS for my rotated drives because the "fast cloning" capability would be beneficial per Veeam recommendation.

[popjls]

Unfortunately, Windows 10 version 1809 and Windows Server 2019 currently are NOT on the list of OS included in the OOB patches.

Hmm I'm running Server 2019 version 1809 and mine are still working fine - Even created a brand new ReFS iscsi repository today to test. At least if it stops working removing this update should fix it.

[PerHolm]

This have just been released from Microsoft:
January 17, 2022 Non-security update (KB5010796) Out-of-band

[candersoncch]

Per message posted on 1/17/2022 from Windows Message Center, Microsoft's OOB patch currently applies to most but not all versions of Windows 10 and Windows Server. The complete list of supported OS is available under "Recent announcements":

https://docs.microsoft.com/en-us/window ... age-center

Unfortunately, Windows 10 version 1809 and Windows Server 2019 currently are NOT on the list of OS included in the OOB patches.

My Veeam backup host is running Windows 10 Enterprise LTSC (version 1809), so I'm out of luck at the moment. After the January Windows Update was installed on my Veeam backup host, my removable hard drive (originally formatted with REFS) is showing up as "RAW". I had been using the removable hard drive in a set of rotated drives for Backup Copy Jobs (for offsite archival).

I hope Microsoft releases an OOB for Windows v1809 soon, or I will have to ditch REFS and use NTFS for removable drive. When I set up the rotated drive repository for the Backup Copy Jobs, the Veeam application recommended REFS with 64k allocation unit size instead of NTFS because REFS supported "fast cloning". I really don't want to give up REFS for my rotated drives because the "fast cloning" capability would be beneficial per Veeam recommendation.

I'd like to ask Gostev, or anyone from Veeam, is the guidance for ReFS correct for USB-attached drives? Veeam's pop-up suggesting that I use ReFS with 64k block size is why I did so on our rotated hard drives. I didn't see anything anywhere stating that this only applied to fixed disks, since Microsoft is suggesting that ReFS is not supported on "removable media."

[FrancWest]

I opened a case with Microsoft Support. They confirmed the RAW ReFS issue is also an issue with fixed drives, but the patch from yesterday is only for external drives. They are still working on a patch for fixed drives and he will inform me once the patch is available. He couldn't explain though why Microsoft doesn't mention the issue for fixed drives anywhere, but only for external drives. He confirmed he already had multiple cases where other customers also experience the issue with fixed drives.

[Gostev]

Thanks for the update Franc!

Chris, sounds like you have some wrong expectation about this pop-up. Its goal was to drive all customers to use ReFS for their backup repository, so it merely checks file system type for the selected volume and recommends using ReFS.

It was never a goal for this functionality to be a validator whether storage hardware meets the ReFS system requirements. The latter is up to customers to ensure, as it is simply impossible to validate programmatically. For example, the most important system requirement for ReFS is that the storage hardware behind is on Microsoft HCL (there is actual technical reason for this, having to deal with the reliable FLUSH command processing). Another consideration is the most current Microsoft recommendations, which are not set in stone and are constantly evolving. In fact, this pop-up was added back when Microsoft did not officially support ReFS on ANYTHING other than Storage Spaces it was much later when they relaxed their position based on the feedback from the field and started to officially support ReFS first on basic disks, then later on RAID volumes etc.

[FrancWest]

Hi Anton,

were you able to contact the ReFS team as you mentioned in the digest from last Sunday about this issue?

[Gostev]

I've been in touch with them since the day you started the topic I like to think KB5010796 is a result of this communication!