To ReFS or not to ReFS?

[bonzovt]

I have a customer who has done a Veeam PoC and is moving forward with the design process to fully implement Veeam as their availability solution. They purchased 2 new C-series servers with lots of storage, and I'm curious about the possibility of starting fresh with Windows 2k16 and ReFS as a repository from the get-go. I've used ReFS a little bit in testing to this point, and have seen great performance/space utilization stats. I've also seen some forum posts in the last 6 months that make me a tiny bit concerned about the stability of using ReFS as a backup repository for larger environments. This will be medium sized, with about 400 or so VMs being protected. I know the gotcha about using 64k block size, is there anything else out there I should keep in mind? Does anyone have any reasons to currently rule out ReFS, even given the benefits it can provide? Or have things settled down to this point with newer updates to Win 2k16 and Veeam?

[DaveWatkins]

For most people at this point it's fine, assuming you can give it enough memory. If you have large repo's is when you're most likely to run into issues (ours are ~60TB each). We see very slow deletions and much slower merges than we saw when the volume was new. There have been multiple patches that fixed most of the issues and supposedly there is a patch coming late Feb that will fix everything else. This patch will also be included in the March cumulative patch.

So, at the moment, there are still issues for some people. Those will hopefully be completely fixed come March or earlier if you install the patch manually when it's released

[csydas]

Yeah, I'd hold off on ReFS and wait till Microsoft's patch this February/March. The ReFS topic on the main forum here has been tracking it: veeam-backup-replication-f2/refs-4k-hor ... 9-795.html

I'd wait until some of the folk there who have been battling this for almost 2 years now give you the thumbs up. Not to make fun of their frustration and misery, but might as well learn from their problems (which they are very kindly sharing with the community). 400 VMs is a fair amount of data and once you get in the 15+TB region for ReFS it seems to be what triggers the issues. If you think you can stall out for a few weeks before commissioning servers, probably best to wait and see.

[BartP]

Seeing the update on the ReFS driver is expected in a couple weeks, for sake of the discussion itself....

I tend to factor in the backup storage to decide if i go NTFS or ReFS for a Primary backup repository. For secondary storage and GFS backups ReFS is better than NTFS (an have not let me down so far): faster merges, GFS backup which uses block clones, etc. In the long run, data will be fragmented on the disks, it's inherent of ReFS, so restores can take longer.
Also ReFS comes into play if storage space is limited (often due bad sizing) or if the backup window (backup+merge+backup copy) grows out of control with NTFS merges.

NTFS is great for a fast primary backup storage which has 10k/15k SAS drives and becomes better, the better you configure your storage.
Data is less fragmented and can be defragmented periodically. Sequential read/write = fast. Who doesn't like a 'fast' and responsive backup?
For NTFS i do prefer using full backups once a week (again to decrease data fragmentation) instead of FFI backups.

[jja]

ReFS is pretty much only supported for Storage Space/Storage Spaces Direct.

If you have a san (iscsi/fc), or local drives connected to raid controller (not in pass-through mode), ReFS is NOT supported, and you can experience total dataloss, with no possibility of recovery.

-Jannis

[BartP]

It's an old post, but relevant: veeam-backup-replication-f2/latest-veea ... ml#p256798
"They [MS] want direct access to the actual hard drives and nothing else in the I/O path, not to have to deal with potential bugs of those additional "storage layers that abstract the physical disks". Otherwise, they'd have to test and certify every existing version of every possible layer, which is not doable."

There is a difference between not supported and not working properly, but yeah you are technically correct; the best kind of correct

[DaveWatkins]

ReFS is pretty much only supported for Storage Space/Storage Spaces Direct.

If you have a san (iscsi/fc), or local drives connected to raid controller (not in pass-through mode), ReFS is NOT supported, and you can experience total dataloss, with no possibility of recovery.

-Jannis

This is not correct, it came up in another thread here and Gostev asked the ReFS team directly and they stated it was old documentation and just plain wrong. FC/iSCSI and RAID are all fine for ReFS

[BartP]

in all honesty jja's reply got me off guard.
Trying to find information, this old post was all could find. Basically backing his statement up.
I know it works fine, but have no source at hand to back it up.
Could you help me out on that?

[DaveWatkins]

This is the thread
veeam-backup-replication-f2/latest-veea ... 46019.html

[Gostev]

After more discussions and clarifications with Microsoft, the real issue is indeed the presence of RAID controllers - they don't want to see those in the data path with ReFS.

Why - because they may cache flush operations for I/O optimization purposes, and critical data won't land on the actual disk, getting lost in case of power loss.

In this case:
1. With NTFS volume, this results in a small, contained data corruption.
2. With ReFS volume, there's a good chance of the entire volume loss (if lost data is the file system metadata) - and it sounds like they had enough support cases to justify this restriction.

Now, data will never get lost by enterprise-grade RAID controllers with battery backed write cache (BBWC) which automatically replays once storage is powered on after power loss. So in theory, Microsoft could certify the specific RAID controllers for ReFS. However, there are thousands of RAID controller makes and models, and certifying them all is not doable. Which is why they chose to go with the blank "unsupported" statement instead - what other choice they have?

The biggest (unanswered) question remains: what's up with other existing ReFS use cases then, such as Microsoft Exchange 2016 - which is specifically designed for leveraging ReFS for mailbox datastores – something Exchange team explicitly recommended for new deployments without any reservations as it comes to SAN storage. So of course, ALL of these existing Exchange deployments have been running ReFS on SAN for years now, because that's what customers had in production at the time + Storage Spaces were too new and unproven anyway.

[bonzovt]

just for reference, the Veeam UCS C240 deployment guide explicitly shows creating a RAID 6 virtual drive using the server's internal RAID and formatting that as 64k block size ReFS for use as the repository on that server.

[tsightler]

So in theory, Microsoft could certify the specific RAID controllers for ReFS. However, there are thousands of RAID controller makes and models, and certifying them all is not doable. Which is why they chose to go with the blank "unsupported" statement instead - what other choice they have?

Microsoft has one of the largest hardware certification programs in the world! The choice they have is that they require vendor testing against ReFS to receive the Windows 2016 certified logo. They could easily make vendors implement whatever requirements they want for RAID controllers to get a Windows 2016 certified. The HCL for Windows 2016 certification states the following:

The "Certified for Windows Server 2016" logo identifies hardware components, devices, drivers, systems and solutions that meet Microsoft standards for compatibility and recommended practices with the Windows Server 2016 operating system. Products and Solutions that have earned the Certified for Windows Server logo are also fully supported in Hyper-V environment.

Since the obvious best practice here is that controllers should honor flush requests, then that sounds like that should be a "standard for compatibility and recommend practice". Microsoft currently has hundreds of RAID controllers with Windows 2016 logo. ReFS is a key technology component of Windows 2016, so the controller should be compatible with ReFS to be Windows Server 2016 certified.

Then they just change the statement to "Microsoft does not support ReFS on RAID controllers that are not on the Windows Server 2016 HCL".

[tsightler]

just for reference, the Veeam UCS C240 deployment guide explicitly shows creating a RAID 6 virtual drive using the server's internal RAID and formatting that as 64k block size ReFS for use as the repository on that server.

The guide also indicates that the reference hardware should have a BBWC. Veeam has pretty much always recommended BBWC for RAID controllers on repositories.

[jja]

in all honesty jja's reply got me off guard.
Trying to find information, this old post was all could find. Basically backing his statement up.
I know it works fine, but have no source at hand to back it up.
Could you help me out on that?

This is exactly my issue as well.
I cannot find any official information from Microsoft that ReFS is supported for basic disk on iscsi.
The only information is that it's not supported.
We have about 240TB of Veeam repositories on iscsi formatted with ReFS (consultant did this because ReFS works very nice for veeam).
And if there is even the slightest possibility that this is unsupported, and may result in dataloss, we have to remedy this.
This is why I am trying to get correct official information about this from Microsoft

-Jannis

[jja]

We have gotten official information from Microsoft via a Microsoft partner that did an advisory case.
ReFS on iSCSI is NOT supported.

-Jannis

[grimson]

So basically this invalidates all ReFS installations in summary?
I cannot imagine any Veeam repository based solely on 'Storage Spaces' yet.
Mostly we use a HPE Proliant with the build in Smart Array Controller to create a nice array of disks (also for Hyper-V VM's)

Is it correct to assume now:
Veeam repository only supported on ReFS based on 'Storage Spaces'? (confused )

[StoneFritz]

Did you see the last "Veeam Community Forums Digest" newsletter?

Huge news for all ReFS users! Together with many of you, we've spent countless hours discussing that strange ReFS support policy update from last year, which essentially limited ReFS to Storage Spaces and standalone disks only. So no RAID controllers, no FC or iSCSI LUNs, no nothing – just plain vanilla disks, period. As you know, I've been keeping in touch with Microsoft ReFS team on this issue all the time, translating the official WHYs they were giving me and being devil's advocate, so to speak (true MVP eh). Secretly though, I was not giving up and kept the firm push on them – just because this limitation did not make any sense to me. Still, I can never take all the credit because I know I'd still be banging my head against the wall today if one awesome guy - Andrew Hansen - did not join that Microsoft team as the new PM. He took the issue very seriously and worked diligently to get to the bottom of this, eventually clearing up what in the end appeared to be one big internal confusion that started from a single bad documentation edit.

Bottom line: ReFS is in fact fully supported on ANY storage hardware that is listed on Microsoft HCL. This includes general purpose servers with certified RAID controllers, such as Cisco S3260 (see statement under Basic Disks), as well as FC and iSCSI LUNs on SAN such as HPE Nimble (under Backup Target). What about those flush concerns we've talked about so much? These concerns are in fact 100% valid, but guess what – apparently, Microsoft storage certification process has always included the dedicated flush test tool designed to ensure this command is respected by the RAID controller, with data being protected in all scenarios – including from power loss during write – using technologies like battery-backed write cache (for example, S3260 uses supercapacitor for this purpose). Anyway - I'm super excited to see this resolved, as this was obviously a huge roadblock to ReFS proliferation.

[doum]

The documentation has changed no ? : https://docs.microsoft.com/en-us/window ... s-overview

Backup Target

Deploying ReFS as a backup target is best suited for applications and hardware that implement their own resiliency and availability solutions.

Applications that introduce their own resiliency and availability software solutions can leverage integrity-streams, block-cloning, and the ability to scale and support large data sets.

Note

Backup targets include the above supported configurations. Please contact application and storage array vendors for support details on Fiber Channel and iSCSI SANs. For SANs, if features such as thin provisioning, TRIM/UNMAP, or Offloaded Data Transfer (ODX) are required, NTFS must be used.