ReFS is not supported on SAN-attached storage

[JLundgren]

Hello,

in the Veeam Community Forums Digest[Oct 2 - Oct 8, 2017], Anton writes about Micrososoft statement:

"ReFS is not supported on SAN-attached storage"

This due to possible lack of flush functionality on low end SAN storage RAID controllers.

Why is this possible issue just a case for SAN storage RAID controllers and seem not to apply for RAID controllers housed inside a server together with local disks ?
Or have I misunderstood the statement ?

Regards,

JohnnyL

[HannesK]

ReFS is not supported on SAN-attached storage.

According to a "Microsoft Advisory Case" a customer of mine did some weeks ago, this is wrong. The intention of this sentence was "ReFS on shared volumes is not supported". I have no idea when this will be changed but I have a forwarded Microsoft mail that officially states that ReFS on SAN is fully supported.

[JLundgren]

Hello Hannes,

but according to the latest Digest, Anton has spoken to the source of this comment at Microsoft ?
Can you please clear this out ?

Regards,

Johnny Lundgren

[Gostev]

Correct, I have ran into the very person who has added this comment to the article, and he gave me his reasoning behind one. This was during Q&A after S2D/ReFS breakout session he presented, so there were a few other attendees participating in the discussion as well.

Hannes - Microsoft is a big company and I think we're in wrong position to clear out the confusion between their multiple departments for me, the answer I've got is good enough - even if I personally would put this note in a different manner now that I know what issue it is talking about.

Anyway, the main point of my post was that the technical issue that this statement is supposed to address is NOT specific to ReFS. And what I really wanted everyone to take away from my post was that any backup storage which controller that does not process the flush command correctly is not supported by Veeam regardless of the file system used.This has always been the case, so not something new. And yes Johnny, it most certainly applies to RAID controllers in regular servers as well, because nothing prevents them from potentially implementing ATA specification incorrectly. Limiting the problem to SAN-attached storage and ReFS as the note does is the classic case of survivor's bias.

The good news is that as I noted, in our experience the issue seems to be somewhat isolated to low-end consumer-grade storage hardware, and in general is not that common. To explain the actual issue in a bit more details: a part of ATA specification is the flush command that must cause the storage controller to write all data from its write cache to disk media disk before returning success. But some storage controllers will return success immediately upon receiving the flush command, without actually flushing the buffers to disk (or without waiting for the flushing to complete). This can happen either due to a firmware bug, or simply to make the storage perform better in performance tests. This behavior may cause data corruption in situations such as sudden power loss, because the application or file system thinks that the data has landed on media when it actually did not. Flush is absolutely critical for transactional workloads because this is how storage transactions actually get finalized - and most applications may experience data corruption or data loss issues when backed by such storage. Even Notepad and Paint can be impacted - as after clicking Save the data will still remain in cache, and the following unexpected computer shutdown will cause that data to be lost.

By the way, if you ever wondered why Microsoft does not support Storage Spaces in a VM for example, which also works perfectly fine, then the reason is the same. They want direct access to the actual hard drives and nothing else in the I/O path, not to have to deal with potential bugs of those additional "storage layers that abstract the physical disks". Otherwise, they'd have to test and certify every existing version of every possible layer, which is not doable.

[Matts N]

Hello Anton,
I feel there is word missing here:

"any backup storage which controller that does process the flush command correctly is not supported by Veeam regardless of the file system used"

Shouldn't it read

"any backup storage which controller that does NOT process the flush command correctly is not supported by Veeam regardless of the file system used" ?

Maybe nit-picking, but still, could get a bit confusing. As it did for me.

// Matts

[nmdange]

I assume they actually mean you can't use SAN storage for Storage Spaces Direct, not ReFS. Or maybe that you can't use ReFS on shared storage spaces with SAS JBODs which used NTFS in 2012 R2. Certain features of ReFS like real-time tiering and self-healing can only be used in combination with Storage Spaces Direct.

[Gostev]

@Matts, thanks for the catch - corrected.

@nmdange this was my first thought as well, that they really meant ReFS in conjunction with S2D... that would make sense.

In any case, a few days ago one of our customers has opened an advisory case with Microsoft regarding this note, and got the confirmation that ReFS file system is supported on a SAN, as long as the disk is not "shared" - plus the promise that this note will be updated soon. This customer already runs ReFS on SAN, so they also got the explicit confirmation that their environment is fully supported by Microsoft.

[jja]

Do you know if there are any updates regarding this?
It seems the note has not been updated, so a bit worried they have found some new issues that makes it not supported again.

-Jannis

[jja]

We have gotten official information from Microsoft via a partner that did an advisory case.
ReFS on iSCSI is NOT supported.

-Jannis

[mweissen13]

Well, Gostev from Veeam says it's ok, Microsoft seems to say it's a no-go. Since we use ReFS on iSCSI we would like to see a definitive answer to this question. Is it really a bad idea. If so, what are the alternatives? The answer surely can't be local disk...

[Gostev]

That's an old topic, here's the more recent thread. Thanks!

[mweissen13]

Hi Gostev, thanks for the pointer. But jja has made the very same statement on the "more recent thread". So what gives? At least there still seems to be confusion, specifically on Microsoft's side. Of course nobody sane in the mind would put backups on a RAID controller or SAN that won't properly honor flush requests or without a BBWC and a proper UPS attached. This could blow up not only ReFS but a number of other applications as well.

[Gostev]

Yep, that is exactly my point too... Veeam would not support such storage regardless of the file system, as our VBK format is transactional and those transactions leverage flushes too.

I suspect what likely happened is they got a few corrupts on some low end storage in support, could not recover the volume - and decided to answer this possible issue once and for all with the blanket statement that is extremely convenient to quickly close Severity 1 support cases with.

And there's also a conspiracy theory that this is their way to push S2D into the masses

[nmdange]

So Gostev since you have contact with the ReFS dev team, maybe you can have a conversation with them and clear this up Sad to say personally I wouldn't trust what MS Support says, only what the product group says. My gut feeling is that it should be supported on any SAN or RAID controller that properly honors the flush command. I mean it's the same type of issue with picking cheap consumer disk drives for S2D. https://blogs.technet.microsoft.com/fil ... sumer-ssd/

[JayL]

Guys,

What's new about the support of ReFS on SAN volumes ?

[Gostev]

Hi Jay, ReFS on SAN volumes is fully supported, and the official Microsoft documentation was updated a few months ago to reflect this.
Basically, it was always supported the whole story was just one big misunderstanding that started from a small documentation edit.

[Jeff M]

Not only am I using 64k REFS on SAN Storage but have built SOBR repos. I have some strange issues with some Linux backups to these SOBR and the storage is seeming to fill up very quickly. The largest backup is about 30TB. I am doing backup copies to non SOBR REFS SAN Luns. Using Per-vm backups to repos. I am doing Synthetic fulls Monthly not weekly. Is this bad? Lastly these are physical server backups using Backup server managed agents Linux and Windows. Using almost all the latest greatest technologies. Using Dell MD Storage. Only issues I see going on 2 months now is Storage filling rapidly.

Do I need any changes? Does anyone know of updated best practices or documentation I can learn more on REFS. What patches do I need for Stable REFS?

Jeff

[Mike Resseler]

Jeff,

Just to clarify. You are backing up VM's (Linux VM's) to a SOBR repository (with locality?) and perform a synthetic full every month? (I'm looking at the backup job now, not the backup copy job...)

[Gostev]

Guys, since the main question of this thread has been answered, I am closing this thread to prevent further offtopic.

Just to recap:

ReFS on SAN volumes is fully supported, and the official Microsoft documentation was updated a few months ago to reflect this.

Thanks!