BEWARE: possible RAW ReFS volumes after installing January updates

[FrancWest]

Ok great, thanks! Too bad KB010796 isn't the full solution (yet)

[corbitech]

Quick update 1/18/2022:

Microsoft just released the OOB patch for Windows 10 Enterprise LTSC and Windows Server 2019 (version 1809).

January 18, 2022—KB5010791 (OS Build 17763.2458) Out-of-band

[candersoncch]

Thanks for the update Franc!

Chris, sounds like you have some wrong expectation about this pop-up. Its goal was to drive all customers to use ReFS for their backup repository, so it merely checks file system type for the selected volume and recommends using ReFS.

It was never a goal for this functionality to be a validator whether storage hardware meets the ReFS system requirements. The latter is up to customers to ensure, as it is simply impossible to validate programmatically. For example, the most important system requirement for ReFS is that the storage hardware behind is on Microsoft HCL (there is actual technical reason for this, having to deal with the reliable FLUSH command processing). Another consideration is the most current Microsoft recommendations, which are not set in stone and are constantly evolving. In fact, this pop-up was added back when Microsoft did not officially support ReFS on ANYTHING other than Storage Spaces it was much later when they relaxed their position based on the feedback from the field and started to officially support ReFS first on basic disks, then later on RAID volumes etc.

Thanks for the info. It would never have occurred to me to check supported hardware for ReFS anymore than it does for NTFS. The more you know, I guess.

[corbitech]

Quick update 1/18/2022:

Microsoft just released the OOB patch for Windows 10 Enterprise LTSC and Windows Server 2019 (version 1809).

January 18, 2022—KB5010791 (OS Build 17763.2458) Out-of-band

Unfortunately, I have another update on this OOB patch. I applied KB5010791 on 1/18/2022. Initially, everything appeared to be fixed. I was able to format my removable hard drive with REFS, and read/write was working properly. However, on 1/20/2022 the removable hard drive reverted to "RAW" status. So this patch failed to resolve the REFS issue for me.

[dlevasseur]

Sorry, this may be long but want cover my backstory and everything...

We just started using Veeam last month. All worked perfectly. I have a backup to the server and then a USB copy to an external disk that I can rotate. Disks are ReFS 64k as recommended. My first test, I took the backup offsite, loaded up Veeam on a machine attached to a test environment, attached the disk restored several VMs. All is good. Lets move on to production. Second test a few weeks later (early January), same situation, all good. Awesome.

Then comes this ReFS issue from Microsoft... So this weekend after all the hubub about out of band patches and everything I decided to do another test to make sure all is good. Take the disk to my test lab. Same machine I used previously, Win11 shows 'RAW'. Ugh. Plug it into my Win10 laptop 'RAW' Ugh. Take it back to the Veeam server which now also sees it as 'RAW' even though I didn't do anything to it.

Fine, rebuild the disk. Decided to clean install Win11 on a completely new machine. Fully updated. Attached the ReFS external Disk, Blue-Screen ReFS.sys error.

So I grabbed 2 more external disks and I'm going to format them NTFS and hope that this all can get squared away.

In short:
Windows Server 2016 (Veeam Backup): Works perfectly
Win11 Fully Updated (Clean install): Blue Screen
Win10 Fully Updated: Disk 'RAW'

[FrancWest]

Once you connect a ReFS disk to another machine with a higher OS version, for example Windows 10 or Windows 11, Server 2022, the ReFS volume is automatically upgraded to a newer version without any notice. If you attach that volume back to a Windows 2016 machine, the volume shows up as raw since that OS can’t handle the new (upgraded) ReFS version.

[Mildur]

Take the disk to my test lab. Same machine I used previously, Win11 shows 'RAW'. Ugh. Plug it into my Win10 laptop 'RAW' Ugh. Take it back to the Veeam server which now also sees it as 'RAW' even though I didn't do anything to it.

Besides usb disks and refs is not supported, you should be careful with using different windows versions for the same refs disks.

You used the disk on a server 2016. Then plugged in to a w11 machine. Then to a w10 machine. Each of this windows systems are using another version of the refs driver. It could happen, that the w11 refs driver will make the partition unreadable for older refs driver like the one on your windows 2016 backup server. The partition will then be shown as RAW.

Going with NTFS is a good decision. If you rotate the usb disk on a weekly basis, then FastClone will not be used. You won‘t benefit from reFS.

[agrob]

how is the status about the patch issue descriebed here? is it now safe to install the latest patches on 2019 server backup repo with ReFS? thanks

[dlevasseur]

Once you connect a ReFS disk to another machine with a higher OS version, for example Windows 10 or Windows 11, Server 2022, the ReFS volume is automatically upgraded to a newer version without any notice. If you attach that volume back to a Windows 2016 machine, the volume shows up as raw since that OS can’t handle the new (upgraded) ReFS version.

Thanks both of you, did not know that. I did a full NTFS backup and restore using my standard method.

I guess that's part of the reason they are no longer supporting it on external drives?

[FrancWest]

how is the status about the patch issue descriebed here? is it now safe to install the latest patches on 2019 server backup repo with ReFS? thanks

Until now I had 2 machines that had the issue. The only way to solve this was to uninstall the update (even the latest OOB update which should solve the ReFS issue), copy the data to an NTFS volume and re-install the update. For me, I couldn't wait any longer since the update contains some critical security fixes also... MS Support isn't of much help either other that they acknowledge the issue but have no fix. On my question if they have any idea what the circumstances are for the issue occurring on fixed drives they state they don't know yet. So it's more or less trial and error -> if after installing the update on a machine containing ReFS volumes the volume is still accessible, you're good. If not, uninstall the update and copy the data off of it, remove the ReFS volume and reinstall the update. Or choose not to install the update if you prefer although with the latest Windows versions when you are not using WSUS that's a bit harder to do since you have to pause updates, but that means no other update will be installed also.

[FrancWest]

I guess that's part of the reason they are no longer supporting it on external drives?

From what I understand they never supported ReFS on external drives from the beginning, indeed probably because of this.

[FrancWest]

Latest update from MS Support: case is being archived with no resolution at the moment. Advice currently is, if the issue occurs after installing the cumulative patch, you have two options:

- uninstall the patch and copy the data to an NTFS volume and re-install the patch
- refrain from installing the patch entirely

I did let them know my frustration about this issue, since they broke it and still don't have a solution or even a cause after several weeks.

[DGhost77]

Running Veeam B&R version 10 on Windows 2012 r2 fully patched.

2 ReFS partition on it, one of them is the biggest one 60 TB connected using ISCSI.
The other one is 15 TB mounted through the datastore of VMWare vsphere cluster of 3 ESXi hosts.

Lost the 2 partitions more than a week ago because of the 2 patchs was installed (was showing as RAW file system of course).

Uninstalled the 2 KB last week then rebooted the server, both partition came back online with the data on it ok. The backups ran on these 2 partitions without any problem last week.

Then this week-end, I lost the 15 TB ReFS partition, now showing again as RAW for the filesystem, the other of 60 TB in ISCSI is still good tough. And the 2 KB has not been reinstalled.

Do I understand that I'm screw and I've lost all the backup hosted on the 15 TB partition?

[Andreas Neufert]

I would maybe contact our support to check with them.

Did maybe Windows Update install the patch again automatically?

[DGhost77]

yeah just found out that patch KB5009624 was reinstalled this weekend, I didn't saw it yesterday. Just uninstalled it again and rebooted but the 15TB volume still show as RAW. The KB5009555 is still not present. Will open a ticket as per your suggestion.

[Gostev]

Latest update from MS Support: case is being archived with no resolution at the moment. Advice currently is, if the issue occurs after installing the cumulative patch, you have two options:

- uninstall the patch and copy the data to an NTFS volume and re-install the patch
- refrain from installing the patch entirely

I did let them know my frustration about this issue, since they broke it and still don't have a solution or even a cause after several weeks.

I'm actually not aware of too many other impacted customers who has fixed drives. Virtually everyone who was impacted by the original issue used USB media, which is perfectly inline with the code changes that caused the issue in the first place. BTW, what is your Microsoft support case ID? I want to share it with my contact in the ReFS team.

Here's the full story as I understood it:

The initial Windows update added some blocks for USB media specifically. The intent of these changes was to block ReFS mount on USB thumb drives, and similar devices as this was a vector of attack. However, this also impacted external USB hard disks, because to the OS they appear similar to those devices. I don't believe it was Microsoft's intention to block these legitimate use cases though.

The following OOB update removed mount blocks from *all* refsv2 volumes (regardless of media or device type). So any refsv2 volume that mounted previously, should absolutely continue to do so after applying the OOB update. If it does not, then it's likely a totally unrelated issue.

[FrancWest]

Hi,

I already converted the production machine back to NTFS. The lab machine having the issue is still at ReFS to test the issue with. However, other machines that do have ReFS (Exchange and Veeam server) still have accessible ReFS volumes after installing the update.

@gostev: the case ID is 2201180040003512

[FrancWest]

Thanks to Veeam Legend @stephc_msft I have finally a solution in our case.

We are running the VM on vSphere 7. After adding the following advanced configuration parameter 'devices.hotplug' with a value of 'false' to the VM under 'edit settings' -> 'VM Options' -> 'Advanced' -> edit configuration parameters the ReFS drives is accessible again. For the parameter see VMware KB https://kb.vmware.com/s/article/1012225?lang=en_us

This setting indeed solves the issue for us on the Windows Server 2019 machine. However, other VMs which still have accessible ReFS volumes after installing the patches don't have this configuration setting and they also have the eject option on the taskbar for the virtual disk. So, although the devices.hotplug setting fixes the issue, it's not the whole story (yet).

[DGhost77]

Thank you FrancWest and stephc_msft! I did the KB you just gave and applied it to our VM in vSphere 6.5 and it solved the problem right away! Sending you virtual cookie or beer.

[FrancWest]

We are getting closer. According to @stephc_msft, the OOB update only fixes the issue for ReFS volume vesions 3.x on removable drives in ws2016 and higher. Our fixed drive which has the issue on the Windows 2019 server still has ReFS version 1.2. Question now is, why isn't this volume upgraded automatically to the latest supported version on Windows 2019.

[FrancWest]

You can query what ReFS version you have on the volume by:

fsutil fsinfo refsinfo <drive letter>

If you have version 1.2 then you have the issue with fixed drives and the only way to correct this is by adding the configuration parameter in vSphere as mentioned earlier.

[agrob]

so if i got things right, it is safe to install the latest patches on a windows server 2019 with ReFS formated with Server 2019 and on local disk. It affects Backup Repos with USB Disks? Thanks

[FrancWest]

I'd first check what the ReFS version is on the internal drive is, if it's higher than 1.2 then probably you're good to go. In our case it was, for some reason, still 1.2 and that turns into RAW after installing the patch. Until you make the configuration change on the VM when running on vSphere.

[agrob]

thanks. just checked it and it is
REFS Version: 3.4
its a physical HPE Server with local Raid 60 ReFS Volumes for Backup Repo.

[FrancWest]

then you shouldn't be affected, be sure to install the OOB also. I think when checking for updates now, it only installs the OOB since it look like it supersedes the original one.

[kyleh]

The devices.hotplug workaround also appears to be working in our environment (VMware 6.7). I was able to install the OOB patch and after setting this parameter the ReFS volume on the VM mounted successfully.

So far we have only seen the issue with fixed disks on Server 2012 VMs.

Our Veeam repos are on 2016 and using ReFS 3.1. I haven't patched them yet but so far we have not seen the issue on 2016 VMs and have several others in our environment with fixed ReFS disks.

I was able to install the original cumulative and the OOB on Server 2019 VMs without issue and did not need the devices.hotplug workaround.

Edited to clarify that I checked ReFS version on 2016/2019 VMs prior to patching them.



Wanted to take a minute and thank the community here for sharing information, experiences, and progress. The information in this thread has been very helpful in making a decision on how to proceed.

[FrancWest]

We were having the issue on server 2022 and 2019 due to the fact that for some reason, which is yet unknown, the ReFS version was still at 1.2... This happened on 2 machines so far. So we either migrated the data to NTFS (the servers weren't any Veeam related servers, so block cloning isn't used or else you would lose those savings) or we added the hotplug parameter to vSphere.

[stfconsulting]

I read about the REFS related issues in the latest round of Windows updates in Gostev's newsletter. Is it safe to roll these patches at this point?

[Gostev]

Please don't create duplicate discussions. Here's the existing thread about this.

[stfconsulting]

Thank you Gostev. Appreciate you pointing me to the right place. I read through the 2 pages here. Our repositories are running 2019. It seems like I would be ok to move forward with the patches at this point?