To ReFS or not to ReFS?

Availability for the Always-On Enterprise

To ReFS or not to ReFS?

Veeam Logoby afishernwn » Mon Jan 22, 2018 10:04 pm

I have a customer who has done a Veeam PoC and is moving forward with the design process to fully implement Veeam as their availability solution. They purchased 2 new C-series servers with lots of storage, and I'm curious about the possibility of starting fresh with Windows 2k16 and ReFS as a repository from the get-go. I've used ReFS a little bit in testing to this point, and have seen great performance/space utilization stats. I've also seen some forum posts in the last 6 months that make me a tiny bit concerned about the stability of using ReFS as a backup repository for larger environments. This will be medium sized, with about 400 or so VMs being protected. I know the gotcha about using 64k block size, is there anything else out there I should keep in mind? Does anyone have any reasons to currently rule out ReFS, even given the benefits it can provide? Or have things settled down to this point with newer updates to Win 2k16 and Veeam?
afishernwn
Service Provider
 
Posts: 35
Liked: 1 time
Joined: Fri Sep 09, 2016 6:15 pm
Full Name: Adam Fisher

Re: To ReFS or not to ReFS?

Veeam Logoby DaveWatkins » Sun Jan 28, 2018 6:47 am

For most people at this point it's fine, assuming you can give it enough memory. If you have large repo's is when you're most likely to run into issues (ours are ~60TB each). We see very slow deletions and much slower merges than we saw when the volume was new. There have been multiple patches that fixed most of the issues and supposedly there is a patch coming late Feb that will fix everything else. This patch will also be included in the March cumulative patch.

So, at the moment, there are still issues for some people. Those will hopefully be completely fixed come March or earlier if you install the patch manually when it's released
DaveWatkins
Expert
 
Posts: 303
Liked: 78 times
Joined: Sun Dec 13, 2015 11:33 pm

Re: To ReFS or not to ReFS?

Veeam Logoby csydas » Tue Jan 30, 2018 6:38 pm 1 person likes this post

Yeah, I'd hold off on ReFS and wait till Microsoft's patch this February/March. The ReFS topic on the main forum here has been tracking it: veeam-backup-replication-f2/refs-4k-horror-story-t40629-795.html

I'd wait until some of the folk there who have been battling this for almost 2 years now give you the thumbs up. Not to make fun of their frustration and misery, but might as well learn from their problems (which they are very kindly sharing with the community). 400 VMs is a fair amount of data and once you get in the 15+TB region for ReFS it seems to be what triggers the issues. If you think you can stall out for a few weeks before commissioning servers, probably best to wait and see.
csydas
Enthusiast
 
Posts: 67
Liked: 9 times
Joined: Tue Jan 16, 2018 5:14 pm
Full Name: Harvey Carel

Re: To ReFS or not to ReFS?

Veeam Logoby BartP » Fri Feb 02, 2018 8:23 pm 1 person likes this post

Seeing the update on the ReFS driver is expected in a couple weeks, for sake of the discussion itself....

I tend to factor in the backup storage to decide if i go NTFS or ReFS for a Primary backup repository. For secondary storage and GFS backups ReFS is better than NTFS (an have not let me down so far): faster merges, GFS backup which uses block clones, etc. In the long run, data will be fragmented on the disks, it's inherent of ReFS, so restores can take longer.
Also ReFS comes into play if storage space is limited (often due bad sizing) or if the backup window (backup+merge+backup copy) grows out of control with NTFS merges.

NTFS is great for a fast primary backup storage which has 10k/15k SAS drives and becomes better, the better you configure your storage.
Data is less fragmented and can be defragmented periodically. Sequential read/write = fast. Who doesn't like a 'fast' and responsive backup?
For NTFS i do prefer using full backups once a week (again to decrease data fragmentation) instead of FFI backups.
Bart Pellegrino,
Veeam Certified Trainer & Architect
Veeam Vanguard

twitter: @bpellegrino
Check http://backitup.online for VMCE and VMCE-ADO study materials and practice exams
BartP
Certified Trainer
 
Posts: 78
Liked: 29 times
Joined: Mon Aug 31, 2015 8:24 am
Location: Netherlands
Full Name: Bart Pellegrino

Re: To ReFS or not to ReFS?

Veeam Logoby jja » Mon Feb 05, 2018 11:38 am

ReFS is pretty much only supported for Storage Space/Storage Spaces Direct.

If you have a san (iscsi/fc), or local drives connected to raid controller (not in pass-through mode), ReFS is NOT supported, and you can experience total dataloss, with no possibility of recovery.

-Jannis
jja
Enthusiast
 
Posts: 45
Liked: 8 times
Joined: Wed Nov 13, 2013 6:40 am
Full Name: Jannis Jacobsen

Re: To ReFS or not to ReFS?

Veeam Logoby BartP » Mon Feb 05, 2018 9:30 pm

It's an old post, but relevant: veeam-backup-replication-f2/latest-veeam-community-forums-digest-oct-2-oct-8-t46019.html#p256798
"They [MS] want direct access to the actual hard drives and nothing else in the I/O path, not to have to deal with potential bugs of those additional "storage layers that abstract the physical disks". Otherwise, they'd have to test and certify every existing version of every possible layer, which is not doable."

There is a difference between not supported and not working properly, but yeah you are technically correct; the best kind of correct :)
Bart Pellegrino,
Veeam Certified Trainer & Architect
Veeam Vanguard

twitter: @bpellegrino
Check http://backitup.online for VMCE and VMCE-ADO study materials and practice exams
BartP
Certified Trainer
 
Posts: 78
Liked: 29 times
Joined: Mon Aug 31, 2015 8:24 am
Location: Netherlands
Full Name: Bart Pellegrino

Re: To ReFS or not to ReFS?

Veeam Logoby DaveWatkins » Mon Feb 05, 2018 9:53 pm 1 person likes this post

jja wrote:ReFS is pretty much only supported for Storage Space/Storage Spaces Direct.

If you have a san (iscsi/fc), or local drives connected to raid controller (not in pass-through mode), ReFS is NOT supported, and you can experience total dataloss, with no possibility of recovery.

-Jannis

This is not correct, it came up in another thread here and Gostev asked the ReFS team directly and they stated it was old documentation and just plain wrong. FC/iSCSI and RAID are all fine for ReFS
DaveWatkins
Expert
 
Posts: 303
Liked: 78 times
Joined: Sun Dec 13, 2015 11:33 pm

Re: To ReFS or not to ReFS?

Veeam Logoby BartP » Mon Feb 05, 2018 10:55 pm

in all honesty jja's reply got me off guard.
Trying to find information, this old post was all could find. Basically backing his statement up.
I know it works fine, but have no source at hand to back it up.
Could you help me out on that?
Bart Pellegrino,
Veeam Certified Trainer & Architect
Veeam Vanguard

twitter: @bpellegrino
Check http://backitup.online for VMCE and VMCE-ADO study materials and practice exams
BartP
Certified Trainer
 
Posts: 78
Liked: 29 times
Joined: Mon Aug 31, 2015 8:24 am
Location: Netherlands
Full Name: Bart Pellegrino

Re: To ReFS or not to ReFS?

Veeam Logoby DaveWatkins » Tue Feb 06, 2018 7:18 am

DaveWatkins
Expert
 
Posts: 303
Liked: 78 times
Joined: Sun Dec 13, 2015 11:33 pm

Re: To ReFS or not to ReFS?

Veeam Logoby Gostev » Wed Feb 07, 2018 12:05 pm

After more discussions and clarifications with Microsoft, the real issue is indeed the presence of RAID controllers - they don't want to see those in the data path with ReFS.

Why - because they may cache flush operations for I/O optimization purposes, and critical data won't land on the actual disk, getting lost in case of power loss.

In this case:
1. With NTFS volume, this results in a small, contained data corruption.
2. With ReFS volume, there's a good chance of the entire volume loss (if lost data is the file system metadata) - and it sounds like they had enough support cases to justify this restriction.

Now, data will never get lost by enterprise-grade RAID controllers with battery backed write cache (BBWC) which automatically replays once storage is powered on after power loss. So in theory, Microsoft could certify the specific RAID controllers for ReFS. However, there are thousands of RAID controller makes and models, and certifying them all is not doable. Which is why they chose to go with the blank "unsupported" statement instead - what other choice they have?

The biggest (unanswered) question remains: what's up with other existing ReFS use cases then, such as Microsoft Exchange 2016 - which is specifically designed for leveraging ReFS for mailbox datastores – something Exchange team explicitly recommended for new deployments without any reservations as it comes to SAN storage. So of course, ALL of these existing Exchange deployments have been running ReFS on SAN for years now, because that's what customers had in production at the time + Storage Spaces were too new and unproven anyway.
Gostev
Veeam Software
 
Posts: 21992
Liked: 2536 times
Joined: Sun Jan 01, 2006 1:01 am
Location: Baar, Switzerland

Re: To ReFS or not to ReFS?

Veeam Logoby afishernwn » Wed Feb 07, 2018 3:59 pm

just for reference, the Veeam UCS C240 deployment guide explicitly shows creating a RAID 6 virtual drive using the server's internal RAID and formatting that as 64k block size ReFS for use as the repository on that server.
afishernwn
Service Provider
 
Posts: 35
Liked: 1 time
Joined: Fri Sep 09, 2016 6:15 pm
Full Name: Adam Fisher

Re: To ReFS or not to ReFS?

Veeam Logoby tsightler » Wed Feb 07, 2018 4:46 pm 1 person likes this post

Gostev wrote:So in theory, Microsoft could certify the specific RAID controllers for ReFS. However, there are thousands of RAID controller makes and models, and certifying them all is not doable. Which is why they chose to go with the blank "unsupported" statement instead - what other choice they have?

Microsoft has one of the largest hardware certification programs in the world! The choice they have is that they require vendor testing against ReFS to receive the Windows 2016 certified logo. They could easily make vendors implement whatever requirements they want for RAID controllers to get a Windows 2016 certified. The HCL for Windows 2016 certification states the following:
The "Certified for Windows Server 2016" logo identifies hardware components, devices, drivers, systems and solutions that meet Microsoft standards for compatibility and recommended practices with the Windows Server 2016 operating system. Products and Solutions that have earned the Certified for Windows Server logo are also fully supported in Hyper-V environment.

Since the obvious best practice here is that controllers should honor flush requests, then that sounds like that should be a "standard for compatibility and recommend practice". Microsoft currently has hundreds of RAID controllers with Windows 2016 logo. ReFS is a key technology component of Windows 2016, so the controller should be compatible with ReFS to be Windows Server 2016 certified.

Then they just change the statement to "Microsoft does not support ReFS on RAID controllers that are not on the Windows Server 2016 HCL".
tsightler
Veeam Software
 
Posts: 5046
Liked: 1962 times
Joined: Fri Jun 05, 2009 12:57 pm
Full Name: Tom Sightler

Re: To ReFS or not to ReFS?

Veeam Logoby tsightler » Wed Feb 07, 2018 4:47 pm 1 person likes this post

afishernwn wrote:just for reference, the Veeam UCS C240 deployment guide explicitly shows creating a RAID 6 virtual drive using the server's internal RAID and formatting that as 64k block size ReFS for use as the repository on that server.

The guide also indicates that the reference hardware should have a BBWC. Veeam has pretty much always recommended BBWC for RAID controllers on repositories.
tsightler
Veeam Software
 
Posts: 5046
Liked: 1962 times
Joined: Fri Jun 05, 2009 12:57 pm
Full Name: Tom Sightler

Re: To ReFS or not to ReFS?

Veeam Logoby jja » Thu Feb 08, 2018 9:16 am 1 person likes this post

BartP wrote:in all honesty jja's reply got me off guard.
Trying to find information, this old post was all could find. Basically backing his statement up.
I know it works fine, but have no source at hand to back it up.
Could you help me out on that?

This is exactly my issue as well.
I cannot find any official information from Microsoft that ReFS is supported for basic disk on iscsi.
The only information is that it's not supported.
We have about 240TB of Veeam repositories on iscsi formatted with ReFS (consultant did this because ReFS works very nice for veeam).
And if there is even the slightest possibility that this is unsupported, and may result in dataloss, we have to remedy this.
This is why I am trying to get correct official information about this from Microsoft

-Jannis
jja
Enthusiast
 
Posts: 45
Liked: 8 times
Joined: Wed Nov 13, 2013 6:40 am
Full Name: Jannis Jacobsen

Re: To ReFS or not to ReFS?

Veeam Logoby jja » Mon Feb 26, 2018 6:09 am

We have gotten official information from Microsoft via a Microsoft partner that did an advisory case.
ReFS on iSCSI is NOT supported.

-Jannis
jja
Enthusiast
 
Posts: 45
Liked: 8 times
Joined: Wed Nov 13, 2013 6:40 am
Full Name: Jannis Jacobsen

Next

Return to Veeam Backup & Replication



Who is online

Users browsing this forum: Exabot [Bot], Google [Bot], Google Feedfetcher, kelaros, Majestic-12 [Bot] and 50 guests