Comprehensive data protection for all workloads
FrancWest
Expert
Posts: 408
Liked: 76 times
Joined: Sep 17, 2017 3:20 am
Full Name: Franc
Contact:

BEWARE: possible RAW ReFS volumes after installing January updates

Post by FrancWest » 7 people like this post

Hi,

Today I noticed that after installing the January Windows Updates released yesterday, ReFS volumes came up as RAW. This luckily it wasn't on our Veeam backup server since that one hasn't installed the updates yet. On Reddit I read reports from users that experience the same issue.

https://www.reddit.com/r/exchangeserver ... _database/

The above post is about Windows 2012 R2, but I had the issue on a Windows 2022 server.

Uninstalling the update (KB5009555 in case of Windows 2022) brings the ReFS volumes back up again.

So be cautious not to install these updates on your Veeam server until further notice.

SKratzTS
Enthusiast
Posts: 69
Liked: 14 times
Joined: Jan 06, 2017 7:23 pm
Full Name: Steve Kratz
Contact:

[MERGED] ReFS drives break after Jan 2022 MS update!

Post by SKratzTS » 1 person likes this post

Microsoft has pulled a good one this month. It appears that ReFS drives break after applying the patches released yesterday. I had two USB ReFS backup destinations change to RAW.

Reddit has a number of posts regarding this, and other major breakage going on: https://www.reddit.com/r/sysadmin/comme ... _20220112/ You can remove the bad update and get ReFS drives back. Hopefully you see this and don't do something destructive trying to repair any backup drives you have!

Gostev
SVP, Product Management
Posts: 29947
Liked: 5873 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: BEWARE: possible RAW ReFS volumes after installing January updates

Post by Gostev »

Franc, do you also use a USB hard drive for ReFS repository?

FrancWest
Expert
Posts: 408
Liked: 76 times
Joined: Sep 17, 2017 3:20 am
Full Name: Franc
Contact:

Re: BEWARE: possible RAW ReFS volumes after installing January updates

Post by FrancWest » 1 person likes this post

No, we don’t. I had this issue on 2 servers using ReFS. One fileserver running Windows Server 2022 containing the FSLogix user profiles and one test server running Windows Server 2022. Both servers have fixed drives, so no USB.

Our Veeam server wasn’t affected sine I unapproved the update for installing using our WSUS server after I noticed the issue.

I just thought to give a heads-up since ReFS is being used heavily in combination with Veeam.

HannesK
Veeam Software
Posts: 11557
Liked: 2209 times
Joined: Sep 01, 2014 11:46 am
Full Name: Hannes Kasparick
Location: Austria
Contact:

Re: BEWARE: possible RAW ReFS volumes after installing January updates

Post by HannesK »

just as a note: I installed the KB5009555 update and my REFS volume is still accessible. But my lab is relatively empty, nothing special installed there.

Image

Gostev
SVP, Product Management
Posts: 29947
Liked: 5873 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: BEWARE: possible RAW ReFS volumes after installing January updates

Post by Gostev »

KB5009555 has a bunch of ReFS CVEs and ALL of them require "physical" attack vector (meaning direct storage manipulation). This makes external drives the best exploit tool for these vulnerabilities: you just bring a ReFS-formatted USB drive containing necessary "tweaks", plug it into the system and take over it. So Microsoft could close all 7 vulnerabilities (and probably many other similar ones yet to be discovered) by simply restricting ReFS driver to fixed media only.

This does not explain though why Franc (and some other folks on Reddit) also have issues with fixed drives.

Gustav
Enthusiast
Posts: 34
Liked: 36 times
Joined: May 29, 2020 2:12 pm
Full Name: Gustav Brock
Contact:

Re: BEWARE: possible RAW ReFS volumes after installing January updates

Post by Gustav »

We run a handful of physical servers, 2019 and 2022. All updated yesterday.
Each server hosts an internal and an external VBR repository (USB harddrive). All repositories are ReFS formatted.

The internal drives are not affected by the update.
Two servers have "lost" the ReFS partition on the external USB drive. They now appear as RAW. These two servers are of different brands.

Would there be an official link to the documentation of the buggy update? Indeed when to expect it to be resolved?

Mildur
Veeam Software
Posts: 3868
Liked: 1249 times
Joined: May 13, 2017 4:51 pm
Full Name: Fabian K.
Location: St. Gallen, Switzerland
Contact:

Re: BEWARE: possible RAW ReFS volumes after installing January updates

Post by Mildur »

Would there be an official link to the documentation of the buggy update? Indeed when to expect it to be resolved?
Best to ask in a Microsoft forum.

They have pulled now the faulty updates:
https://www.bleepingcomputer.com/news/m ... -bugs/amp/
Product Management Analyst @ Veeam Software

FrancWest
Expert
Posts: 408
Liked: 76 times
Joined: Sep 17, 2017 3:20 am
Full Name: Franc
Contact:

Re: BEWARE: possible RAW ReFS volumes after installing January updates

Post by FrancWest »

KB5009555 for Windows 2022 doesn’t seem to have been pulled. My test server still downloads this update from Windows Update. However, this is the faulty update which causes the raw ReFS issue.

Mildur
Veeam Software
Posts: 3868
Liked: 1249 times
Joined: May 13, 2017 4:51 pm
Full Name: Fabian K.
Location: St. Gallen, Switzerland
Contact:

Re: BEWARE: possible RAW ReFS volumes after installing January updates

Post by Mildur »

Yes, you are right, looks like they pulled only the other one.
Product Management Analyst @ Veeam Software

candersoncch
Novice
Posts: 7
Liked: 4 times
Joined: Jul 13, 2021 1:38 pm
Full Name: Chris Anderson
Contact:

Re: BEWARE: possible RAW ReFS volumes after installing January updates

Post by candersoncch » 1 person likes this post

So, Microsoft has a KB article up about this, which states that ReFS is not supported on external drives:

https://support.microsoft.com/en-us/top ... 9b0ee86c65

I hope that they intend to fix this. Despite it being "unsupported," (something I didn't know, and I'm not sure how I could have been expected to know since it's not like Windows warned you if you tried to format an external drive with ReFS), it has worked perfectly fine for years. I have a bunch of USB external drives we rotate through as backup repositories and to move all of the data, reformat to NTFS, and copy it back would be extremely time consuming.

FrancWest
Expert
Posts: 408
Liked: 76 times
Joined: Sep 17, 2017 3:20 am
Full Name: Franc
Contact:

Re: BEWARE: possible RAW ReFS volumes after installing January updates

Post by FrancWest »

The issue did occur on 2 fixed drives for us, not external USB drives. So there must be some sort of situation where it also occurs on fixed (internal) drives.

popjls
Enthusiast
Posts: 39
Liked: 4 times
Joined: Jun 25, 2018 3:41 am
Contact:

Re: BEWARE: possible RAW ReFS volumes after installing January updates

Post by popjls »

I have KB5009557 installed on my 2019 ReFS Veeam server and it is all working fine. I did install it on the 13th AUS time so it might have been re-released and then picked up in my time zone. Unsure whether to remove it or not now.

dkvello
Service Provider
Posts: 103
Liked: 12 times
Joined: Jan 01, 2006 1:01 am
Full Name: Dag Kvello
Location: Oslo, Norway
Contact:

Re: BEWARE: possible RAW ReFS volumes after installing January updates

Post by dkvello »

I had an SQL server (last week, running on WS2019) at a customer site that lost all its ReFS drives after MS "fixed" things. They where all "local" disks under vSphere, nu external stuff.
What puzzled me was that when I tried instant recovery (this server is backed up every other hour), the instantly recovered server had all ReFS drives come up as Raw.
But, Veeam could easily access and restore all the files from the same recovery-point when using file-level restore.

I tested adding a new virtual disk and formatting it as ReFS and that disk worked just fine. So, since nothing had been added the databases since the last recovery-point I just removed the old ReFS disks from the running SQL server and created new ReFS disks. Then it was just a few minutes to "Restore files to original location" in Veeam and everything was up and running again.

After that I went hunting for "why" this happened.

FrancWest
Expert
Posts: 408
Liked: 76 times
Joined: Sep 17, 2017 3:20 am
Full Name: Franc
Contact:

Re: BEWARE: possible RAW ReFS volumes after installing January updates

Post by FrancWest » 1 person likes this post

My experience is that it doesn't happen with every ReFS volume. There must be some special circumstances why the fixed ReFS volumes come up as RAW after installing the update. Today I also noticed that another lab machine running Windows Server 2019 experiences the same issue. After uninstalling KB5009557 the ReFS volumes where accessible again.

The reason Veeam is able to recover files from it is probably because of the fact that the proxy which is used to mount the volume isn't affected by the ReFS issue.

Gostev
SVP, Product Management
Posts: 29947
Liked: 5873 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: BEWARE: possible RAW ReFS volumes after installing January updates

Post by Gostev »

Exactly. More specifically, it's the mount server associated with the backup repository.

m.novelli
Veeam ProPartner
Posts: 442
Liked: 61 times
Joined: Dec 29, 2009 12:48 pm
Full Name: Marco Novelli
Location: Asti - Italy
Contact:

Re: BEWARE: possible RAW ReFS volumes after installing January updates

Post by m.novelli »

candersoncch wrote: Jan 16, 2022 5:15 pm So, Microsoft has a KB article up about this, which states that ReFS is not supported on external drives:

https://support.microsoft.com/en-us/top ... 9b0ee86c65

I hope that they intend to fix this. Despite it being "unsupported," (something I didn't know, and I'm not sure how I could have been expected to know since it's not like Windows warned you if you tried to format an external drive with ReFS), it has worked perfectly fine for years. I have a bunch of USB external drives we rotate through as backup repositories and to move all of the data, reformat to NTFS, and copy it back would be extremely time consuming.
Yes, me too I have a dozen customers with RDX drives formatted ReFS 64K to speed up Veeam Backup Copy Job

Marco

m.novelli
Veeam ProPartner
Posts: 442
Liked: 61 times
Joined: Dec 29, 2009 12:48 pm
Full Name: Marco Novelli
Location: Asti - Italy
Contact:

Re: BEWARE: possible RAW ReFS volumes after installing January updates

Post by m.novelli » 2 people like this post

BTW , my company manage hundreds of VM and physical servers for many customers. I had 4 Domain Controller with the endless reboot for 2 different customers , VM in vSphere , nothing special installed, just DC. Luckily no issues with ReFS RAW volumes

I still wonder why in 2022 Windows Update doesn't offer two basic settings

- wait xxx days before installing updates
- install updates at this specific hour and reboot immediately

Marco

GMGinger
Lurker
Posts: 1
Liked: never
Joined: May 12, 2020 4:19 am
Full Name: Gary Mellor
Contact:

Re: BEWARE: possible RAW ReFS volumes after installing January updates

Post by GMGinger »

To possibly explain the difference between some ReFS volumes becoming unavailable and others working fine - there were comments on Reddit that suggested that it was only upsetting servers which had ReFS filesystems that had been created on older versions of Windows. I'm sure Windows will upgrade the ReFS version for a filesystem when you first mount it, but perhaps there's some left over artifact from the previous ReFS version that catches the updated ReFS driver out.

ryan_gcits
Novice
Posts: 8
Liked: 1 time
Joined: Feb 27, 2017 10:02 pm
Full Name: Ryan Colombo
Contact:

Re: BEWARE: possible RAW ReFS volumes after installing January updates

Post by ryan_gcits » 1 person likes this post

Hello all,

Looks like MS has release an out-of-band patch to resolve the bug with REFS formatted removable media.

https://support.microsoft.com/en-au/top ... 6a9ca52da7

bytewiseits
Service Provider
Posts: 43
Liked: 18 times
Joined: Nov 23, 2018 12:23 am
Full Name: Dion Norman
Contact:

Re: BEWARE: possible RAW ReFS volumes after installing January updates

Post by bytewiseits »

Had a smaller affected Windows 10 21H2 Repo with an internal disk and USB disk, both with ReFS. USB drive had showed as raw until uninstalling the 2022-01 CU a few days ago.
Installed the 2022-01 CU (KB5009543) again and confirmed USB drive RAW again, then installed OOB update KB5010793 and USB drive is now accessible again. Looks like they have fixed it for Win10 which is promising at least.

FrancWest
Expert
Posts: 408
Liked: 76 times
Joined: Sep 17, 2017 3:20 am
Full Name: Franc
Contact:

Re: BEWARE: possible RAW ReFS volumes after installing January updates

Post by FrancWest »

This OOB patch doesn’t solve the RAW ReFS issue with fixed drives we are having. Volume is still shown as RAW after installing KB5009555 and KB5010796. :(

Reverting back to working state without KB5009555 and KB5010796.

corbitech
Influencer
Posts: 19
Liked: 4 times
Joined: Nov 28, 2016 4:31 am
Full Name: Ashok Kumar
Contact:

Re: BEWARE: possible RAW ReFS volumes after installing January updates

Post by corbitech » 1 person likes this post

Per message posted on 1/17/2022 from Windows Message Center, Microsoft's OOB patch currently applies to most but not all versions of Windows 10 and Windows Server. The complete list of supported OS is available under "Recent announcements":

https://docs.microsoft.com/en-us/window ... age-center

Unfortunately, Windows 10 version 1809 and Windows Server 2019 currently are NOT on the list of OS included in the OOB patches.

My Veeam backup host is running Windows 10 Enterprise LTSC (version 1809), so I'm out of luck at the moment. After the January Windows Update was installed on my Veeam backup host, my removable hard drive (originally formatted with REFS) is showing up as "RAW". I had been using the removable hard drive in a set of rotated drives for Backup Copy Jobs (for offsite archival).

I hope Microsoft releases an OOB for Windows v1809 soon, or I will have to ditch REFS and use NTFS for removable drive. When I set up the rotated drive repository for the Backup Copy Jobs, the Veeam application recommended REFS with 64k allocation unit size instead of NTFS because REFS supported "fast cloning". I really don't want to give up REFS for my rotated drives because the "fast cloning" capability would be beneficial per Veeam recommendation.

popjls
Enthusiast
Posts: 39
Liked: 4 times
Joined: Jun 25, 2018 3:41 am
Contact:

Re: BEWARE: possible RAW ReFS volumes after installing January updates

Post by popjls »

Unfortunately, Windows 10 version 1809 and Windows Server 2019 currently are NOT on the list of OS included in the OOB patches.
Hmm I'm running Server 2019 version 1809 and mine are still working fine - Even created a brand new ReFS iscsi repository today to test. At least if it stops working removing this update should fix it.

PerHolm
Lurker
Posts: 1
Liked: never
Joined: Nov 02, 2021 7:03 pm
Full Name: Per Holm
Contact:

Re: BEWARE: possible RAW ReFS volumes after installing January updates

Post by PerHolm »

This have just been released from Microsoft:
January 17, 2022 Non-security update (KB5010796) Out-of-band

candersoncch
Novice
Posts: 7
Liked: 4 times
Joined: Jul 13, 2021 1:38 pm
Full Name: Chris Anderson
Contact:

Re: BEWARE: possible RAW ReFS volumes after installing January updates

Post by candersoncch »

corbitech wrote: Jan 18, 2022 1:02 am Per message posted on 1/17/2022 from Windows Message Center, Microsoft's OOB patch currently applies to most but not all versions of Windows 10 and Windows Server. The complete list of supported OS is available under "Recent announcements":

https://docs.microsoft.com/en-us/window ... age-center

Unfortunately, Windows 10 version 1809 and Windows Server 2019 currently are NOT on the list of OS included in the OOB patches.

My Veeam backup host is running Windows 10 Enterprise LTSC (version 1809), so I'm out of luck at the moment. After the January Windows Update was installed on my Veeam backup host, my removable hard drive (originally formatted with REFS) is showing up as "RAW". I had been using the removable hard drive in a set of rotated drives for Backup Copy Jobs (for offsite archival).

I hope Microsoft releases an OOB for Windows v1809 soon, or I will have to ditch REFS and use NTFS for removable drive. When I set up the rotated drive repository for the Backup Copy Jobs, the Veeam application recommended REFS with 64k allocation unit size instead of NTFS because REFS supported "fast cloning". I really don't want to give up REFS for my rotated drives because the "fast cloning" capability would be beneficial per Veeam recommendation.
I'd like to ask Gostev, or anyone from Veeam, is the guidance for ReFS correct for USB-attached drives? Veeam's pop-up suggesting that I use ReFS with 64k block size is why I did so on our rotated hard drives. I didn't see anything anywhere stating that this only applied to fixed disks, since Microsoft is suggesting that ReFS is not supported on "removable media."

FrancWest
Expert
Posts: 408
Liked: 76 times
Joined: Sep 17, 2017 3:20 am
Full Name: Franc
Contact:

Re: BEWARE: possible RAW ReFS volumes after installing January updates

Post by FrancWest »

I opened a case with Microsoft Support. They confirmed the RAW ReFS issue is also an issue with fixed drives, but the patch from yesterday is only for external drives. They are still working on a patch for fixed drives and he will inform me once the patch is available. He couldn't explain though why Microsoft doesn't mention the issue for fixed drives anywhere, but only for external drives. He confirmed he already had multiple cases where other customers also experience the issue with fixed drives.

Gostev
SVP, Product Management
Posts: 29947
Liked: 5873 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: BEWARE: possible RAW ReFS volumes after installing January updates

Post by Gostev »

Thanks for the update Franc!

Chris, sounds like you have some wrong expectation about this pop-up. Its goal was to drive all customers to use ReFS for their backup repository, so it merely checks file system type for the selected volume and recommends using ReFS.

It was never a goal for this functionality to be a validator whether storage hardware meets the ReFS system requirements. The latter is up to customers to ensure, as it is simply impossible to validate programmatically. For example, the most important system requirement for ReFS is that the storage hardware behind is on Microsoft HCL (there is actual technical reason for this, having to deal with the reliable FLUSH command processing). Another consideration is the most current Microsoft recommendations, which are not set in stone and are constantly evolving. In fact, this pop-up was added back when Microsoft did not officially support ReFS on ANYTHING other than Storage Spaces :) it was much later when they relaxed their position based on the feedback from the field and started to officially support ReFS first on basic disks, then later on RAID volumes etc.

FrancWest
Expert
Posts: 408
Liked: 76 times
Joined: Sep 17, 2017 3:20 am
Full Name: Franc
Contact:

Re: BEWARE: possible RAW ReFS volumes after installing January updates

Post by FrancWest »

Hi Anton,

were you able to contact the ReFS team as you mentioned in the digest from last Sunday about this issue?

Gostev
SVP, Product Management
Posts: 29947
Liked: 5873 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: BEWARE: possible RAW ReFS volumes after installing January updates

Post by Gostev »

I've been in touch with them since the day you started the topic :D I like to think KB5010796 is a result of this communication!

Post Reply

Who is online

Users browsing this forum: Bing [Bot] and 24 guests