Should we really Consider ReFS?

[govi]

Hi All,

Recently we came across some stunning reality about using ReFS Drive which are mapped to Windows Server 2019 Machine( storage is Synology NAS).

All these years we are using ReFS formatted drive and presenting to Windows Server 2019. Everything is/was going fine, performance etc etc.Due to compliance issue, we have to test the current backup strategy so that we can draft recovery procedure. So we simulated the recovery procedure, be aware that our backup server is VM( is not ideal scenario but thats the current reality). so we installed VBR software on one of the laptop and imported configuration file and attached ReFS Drive to Windows 11 Pro machine. When we attached the ReFS Drive,Windows 11 pro laptop immediately went in blue screen. When i checked dump file it said something related to below.
MODULE _NAME:ReFS
Image_Name: ReFS.SYS
FAILURE_BUCKET_ID : AV_R(NULL)_ReFS!CreateUniCodeStringDirectorKey.

This happened multiple time, so disconnected iSCSI from Windows 11 laptop and tried from Windows 10 Pro computer and was able to access ReFS drive without any issue from Windows 10 Pro machine. Happy until here that we know what exactly needs to be done when disaster hits and this has been tested in our environment and it works. not fully tested but we know how to reach to backup file.

Due to some Maintenance work, we had to shutdown the backup VM and started again, So as usual we were expecting ReFS drive to be visible in Windows Server 2019 server but we got one surprise that ReFS drives are no longer accessible and shows as RAW. when we ran chkdks /f x: (x is Drive letter) it still shows has file system as ReFS but in GUI of server under management it shows as RAW.
Now question is that why all of sudden we see the RAW format, is it due to LUN corruption but synology console says that LUN is healthy.

So we tried again from Windows 10 computer and we were able to access one of the ReFS Drive. Atleast we were sure that we have the data and hope is still their for us to get the data back.

After some time, we lost the access to this drive as well since we disconnected it from Windows 10 machine, restarted the synology drive hoping that restart will do some trick but that did not happen.

So i lost final hope to recover the data whatever was possible from Windows 10 Machine. Raised support ticket with synology and it was more about ReFS then synology so did not get proper answer from synology and it was expected.

so decided to attach ReFS on Windows 11 Pro again just to see if anything changes, and to my surprise, it did not gave me blue screen like before and was able to access one of the ReFS drive. So immediately started to copy data on other storage whatever was available.

After doing some research and to my understanding that ReFS version is different for Windows Server 2019,2022 then Windows 10 or Windows 11 Pro( recently started to support ReFS?), so when we attached ReFS to windows 10, it automatically upgraded the Drive or version to Windows 10 thus making not compatible with Windows Server 2019 hence we were not able to access the ReFS drive on server? I might be wrong here but 1 or 2 article it says about this that gave me some idea as what is exactly going wrong.

If this is the case, then we must be aware that, once we attach ReFS formatted drive to Windows 10 or Windows 11, probably you wont be attach to windows server thus making backup in accessible.

Probably option will be connect back to Windows 10 or Windows 11 Machine, move all backups to another media and again format on server as ReFS and move backup to newly formatted drive?

If this is the the case, then its really hassle as we need to spend lot of time to copying data back and forth.

Now i have real question that we should be relying on ReFS formatted Drive? or we need to change approach completely ? or I am missing something here which is quick fix to my problem.

I am yet to replicate this issue on spare NAS which confirms my theory.

If any one had similar experience, kindly share. I am expecting some input from Gostev as he is the expert of ReFS


Thanks Govi

[HannesK]

Hello,
I hope the opinion of other people is also okay

Yes, the bluescreen thing between Server 2019 formatted REFS on Windows 11 seems to be normal. That's why I stopped using REFS for Veeam Agent for Windows backups of my laptop. That worked fine with Windows 10. I'm surprised you got it working with Windows 11. Yes, the different version of REFS are expected to lead to the things you see.

Likely REFS should have never been used, because historically the consumer NAS systems did not meet the system requirements for REFS

All ReFS supported configurations must use Windows Server Catalog certified hardware. For other requirements, limitations and known issues, see this Veeam KB article.

I see that there are some Synology systems on it (maybe yours not?)

Best regards,
Hannes

[tyler.jurgens]

The points you've made @Govi along with what @HannesK said has led me to the same conclusion. I no longer use ReFS and I generally recommend a Linux XFS based repository.

However, if you want to use ReFS, you need to follow a few rules and never break them:

1) Always use certified hardware as @HannesK points out. First and foremost. Never roll the dice unless you really don't care about the data.
2) Never mount an ReFS drive to any other OS version than the source. In your case, if you had that ReFS drive on Server 2019, then you could mount it to another Server 2019 OS. But once you mount it to another version of Windows it may change the ReFS file system compatibility and you could get stuck either not getting your data back or being forced into using that other OS going forward.

If you follow those rules you likely won't have as bad of a time as you did.

[edh]

We got repos of 200TB in ReFS without problems working hard everyday. Total 1PB. Just we dont use a Synology in the middle to avoid transport issues. For our perspective it just works.

[govi]

Probably Veeam Certified storage like Dell HPE. but does not fit on price for small office with 5 to 10 user.

However, if you want to use ReFS, you need to follow a few rules and never break them:

1) Always use certified hardware as @HannesK points out. First and foremost. Never roll the dice unless you really don't care about the data.
2) Never mount an ReFS drive to any other OS version than the source. In your case, if you had that ReFS drive on Server 2019, then you could mount it to another Server 2019 OS. But once you mount it to another version of Windows it may change the ReFS file system compatibility and you could get stuck either not getting your data back or being forced into using that other OS going forward.

Agreed so i need to write rule in bold red in my recovery procedure that you must not connect ReFS to any other OS other then source OS.

If you follow those rules you likely won't have as bad of a time as you did.

yes, i do agree from this from current experience

[govi]

We got repos of 200TB in ReFS without problems working hard everyday. Total 1PB. Just we dont use a Synology in the middle to avoid transport issues. For our perspective it just works.

for me its working and we been using for past 5 years in multiple locations. This issue came in light, when we tried to reproduce recovery procedure and connected ReFS Drive to Windows 11 laptop.

None of the article warns about this so i was not aware about this scenario. Not to blame anyone, but learned lesson before its too late.

[mkretzer]

You ALWAYS have to be very careful mounting ReFS on other windows versions. The metadata update cannot be stopped and from that point on you often just cannot access from an older version.

And if your drives are large enough the metadata update can take a long time: post484187.html#p484187
Also, there is no supported option for ReFS on anything but disks it can directly access without much in between. I asked ReFS DEVs and it is really not very flexible if you want to use it in a "supported" way.

For that reason at least for us XFS+LVM is the only useful combination.

[govi]

Yes, i learned lesson now and we know exactly what needs to be done if we plan to upgrade server to higher version.

now working on XFS for bigger and critical locations so that we don't end up with this kind of issue in future. its better to be safe than sorry

[AhmedS]

There are different flavours of ReFS:
Windows 10 (depending on build 1703 - v3.2, 1709 - v3.3, 1803 or later : 3.4 etc
Server 2019 - v3.4
Windows 11 - v3.4 upto v3.14
Server 2022 - v3.7
see https://gist.github.com/XenoPanther/15d ... 2974084ef8 for a compatibility matrix.
I cannot recall which version of server OS, but there was a major rewrite of how ReFS works a couple years back, so there are definitely alot of changes.

The answer to your question is no.
As recommended by others, you should move to Linux XFS immutable storage Repo.
I see Veeam has a great guide on how to set it up and is recommended from a security perspective. And its robust enough to deploy once and not worry about it after.
Thanks

PS: I dont know anything about Synology - just speaking to my ReFS experience.

[Hauke]

We had a few negative results with ReFS in the past to. Also tried using ReFS as a Repo for Veeam. It got slower and slower over the time, and the space usage grew fast.
In fact, we are not using it anymore. Windows Servers are using NTFS only, Veeam is now using Linux devices with XFS (which is functioning MUCH better than ReFS).

[dasfliege]

I highly doubt that this issue is related to synology in particular or any other "unsupported" scenario. We're using ReFS as repo since day one in pretty much every possible constellation and as long as you follow two pretty basic rules, that works just fine.
- Never use a client OS as ReFS repo (Win10/11)
- Re-attach ReFS formated disks only to server OS which is the exact same version or newer. If it's newer, you probably will never be able to attach it back to the original system, unless you update the original system.

However, using XFS may be the way to go today anyways.

[JeroenL]

I don't understand.
Microsoft has some really clear statements about ReFS on Technet.
Veeam has too has really clear statements about ReFS on their support pages.
People choose to neglect these statements and start to complain if things go fubar and they loose data.

Next I see pricing is the reason people choose to neglect the recommendations. What is cheaper? Spend $8000 over 5 years and be fully capable to restore everything you need and have decent hardware vendor support and warranty. Or spend $4000 over the same 5 years with all the uncertainties that come with none supported configurations and the big possibility one cannot use the backup data when it's most needed?

Backup is your last resort. In these days with all the ransomware attacks it's not a matter of IF one has to rely on their backup, but WHEN one has to rely on their backup.

With low budget I would still choose for decent hardware from HPE, Dell or whatever. Use a dedicated hardware server in a separate network segement behind a firewall and create a backup copy to a Wasabi S3 immutable repository or other offsite immutable system. Follow the Veeam security best practices and make it impossible to access the management interface from any "production" system.

The risk is simply too high these days. Backups are way more important than they were 10 or more years ago.
Trust me, I have seen it all. Don't cut expenses on backup!

[KOOLER]

ReFS has earned the moniker 'Record-Eating File System' for a reason. In contrast, Linux paired with XFS offers several advantages: a) It's entirely free, b) Demonstrably outperforms the Windows+ReFS combination (as evidenced by Veeam's published performance benchmarks), and c) Ensures immutability, which by itself is a crucial safeguard against ransomware attacks. In light of these benefits, one might well question the rationale behind persisting with Window+ReFS in this day and age.

[MoritzG-Seidemann]

I am really interested how the Veeam managed hardened repository (https://community.veeam.com/blogs-and-p ... veeam-8779) turns out. Will make the deployment way easier.

[m.novelli]

@govi you should use NAS as secondary backup, primary backup should be a physical Server with good RAID controller and SAS disks. I'm using ReFS since Windows Server 2019 without issues for both backups and inside VM

If you are a small business in Europe you can buy a refurbished HPe or Dell Server here https://www.renewtech.com/ to make a good physical backup Server

Marco

[Gostev]

In light of these benefits, one might well question the rationale behind persisting with Window+ReFS in this day and age.

It's still a good option for all-in-one Windows-based backup server. It's also ultra-reliable on a proper enterprise-grade hardware, as it was already mentioned by a number of customers and service providers.

[tw_anton]

ReFS is ok, but ideally it needs to live on a Windows Server machines. Backend and/or frontend. I experienced similar issue, minus the BSOD, when I attached a large ReFS volume via iSCSI from a Dell SAN. I learned the hard way how ReFS upgrades between Windows versions. But if managed properly, it provides a lot of advantages over NTFS, especially in a Hyper-V environment. That being said, for a Veeam repo, I use hardened XFS, which is frankly easier, cheaper and virtually maintenance free.

[ZipTX]

Just a quick note that ReFS is supported with Windows 11 *WORKSTATION* via command line. ReFS is not supported with Windows 11 Pro. [https://www.microsoft.com/en-us/windows ... 1#Security] You do not want the wrong operating version trying to disable or hamper ReFS with each update. Use Windows 11 Workstation if attaching to ReFS.

[rciscon]

I know that Veeam has been promoting the use of ReFS for several versions now, and I made some tentative moves in that direction until a couple of years ago when a simple Windows Server Update broke access to my ReFS volumes.

The fact that Microsoft would do a general release of a Windows Server Patch/Update without testing it with ReFS was an indication to me that internally Microsoft doesn't give a damn about customers using ReFS to store their data.

I was able to recover the ReFS data after uninstalling the update. Because of this, I no longer user, or even consider running ReFS to store my data. Every time I'm building a Repository server on Windows based on NTFS, I get the Veeam pop up recommending the use of ReFS--I chuckle, and disregard this recommendation... for the sake of my data and my sanity.

[Gostev]

ReFS is not supported with Windows 11 Pro.

Actually, this is not correct any longer... see my post about this.

[BrianBuchanan]

... be aware that our backup server is VM( is not ideal scenario but thats the current reality).

Thanks for sharing your experiences. I too am shying away from ReFS, but haven't run into the automatic meta data upgrade issue you've experienced.

The backup server can be a VM, the repository should not be. The repository should be a bare-metal server, or the Cloud.

That said, I have run a repository as a VM on a dedicated Synology NAS just to take advantage of XFS Fast Block Clone. If your Synology can run VMs it might be an option for you to consider. Your testing becomes easier as the repository is available for both the production system and your test system at the same time.

I also use a handful of high capacity (18 to 24-TB) USB drives (formatted NTFS and rotated weekly) as periodic backup copy job destination and could use those for air-gapped testing.

Note: If Veeam ever publishes a data mover (plug-in, extension, app, etc.) for Synology and could use Fast Block Clone on Synology's Btrfs, then I'd jump on that.

Good luck!

[govi]

I don't understand.
Microsoft has some really clear statements about ReFS on Technet.
Veeam has too has really clear statements about ReFS on their support pages.
People choose to neglect these statements and start to complain if things go fubar and they loose data.

Next I see pricing is the reason people choose to neglect the recommendations. What is cheaper? Spend $8000 over 5 years and be fully capable to restore everything you need and have decent hardware vendor support and warranty. Or spend $4000 over the same 5 years with all the uncertainties that come with none supported configurations and the big possibility one cannot use the backup data when it's most needed?

Backup is your last resort. In these days with all the ransomware attacks it's not a matter of IF one has to rely on their backup, but WHEN one has to rely on their backup.

With low budget I would still choose for decent hardware from HPE, Dell or whatever. Use a dedicated hardware server in a separate network segement behind a firewall and create a backup copy to a Wasabi S3 immutable repository or other offsite immutable system. Follow the Veeam security best practices and make it impossible to access the management interface from any "production" system.

The risk is simply too high these days. Backups are way more important than they were 10 or more years ago.
Trust me, I have seen it all. Don't cut expenses on backup!

I fully agree with you. "risk it for the biscuit" and already looking for ExaGrid appliance after this findings.

[Gostev]

It's not like ExaGrid does not have a history of its own issues including data losses. Or above-mentioned Wasabi for that matter. There are risks in everything, the are just different.

And I think the point of the post you're quoting is different: the focus should be remaining in the supported envelope and don't try to cut corners no matter what storage technology is used. Again, ReFS has been rock solid at scale for those using proper enterprise-grade hardware.

[JeroenL]

Thanks for sharing your experiences. I too am shying away from ReFS, but haven't run into the automatic meta data upgrade issue you've experienced.

The backup server can be a VM, the repository should not be. The repository should be a bare-metal server, or the Cloud.

That said, I have run a repository as a VM on a dedicated Synology NAS just to take advantage of XFS Fast Block Clone. If your Synology can run VMs it might be an option for you to consider. Your testing becomes easier as the repository is available for both the production system and your test system at the same time.

I also use a handful of high capacity (18 to 24-TB) USB drives (formatted NTFS and rotated weekly) as periodic backup copy job destination and could use those for air-gapped testing.

Note: If Veeam ever publishes a data mover (plug-in, extension, app, etc.) for Synology and could use Fast Block Clone on Synology's Btrfs, then I'd jump on that.

Good luck!

Synology is NON-RELIABLE storage.
It features software raid with (no battery backed) RAM based write-cache. If these machines are pushed too hard of a powerfailure occurs during backup you and up with damaged backup data. Almost 99,9% chance of failure!
If Veeam is a smart company, and they are, they will NEVER EVER publish tools to help move to unreliable data targets.
Stay away from Synology or other cheap NAS devices unless you can affort loosing all you backup data.
Hold on to the 3,2,1,0,0 rule. This makes you slightly less dependent on the backup hardware.

There is nothing wrong with ReFS as long as you use it as it is supposed to.
If you run a Windows backup server. Simply use a Veeam Agent for Windows to create backups of the OS and Application drive. Configure you server with at least 3 volumes. C: volume is used for the OS, D: volume is used for Applications such as PostgreSQL and Veeam, E: is used for backup data. C: and D: can be places on 400 to 800GB Raid 1 SSD drives. E: can be SSD or HDD storage, but can be excluded from the Veeam Agent backup.
Store these backups in an encrypted state next to the encrypted Veeam Backup configuration backups and next to the encrypted Veeam Backup Copy data offsite and (if possible) immutable. Keep a copy of the recovery iso there as well.
In case of a huge issue, simple get new hardware, configure the storage and perform a bare metal recovery.
Your backup server should be running again within a few hours. You have your backup copy that you can copy back (if needed) to the original location using Veeam native features. Remap the backups and you are ready to go. No problems with OS versions or other weird things. Create a backup of your backup server as if it was a normal production server, just be a little more carefull in the placement of this backup data.