Veeam Backup Repository file system NTFS or ReFS?

[Gostev]

Yep, I was answering your question. We didn't even announce V13 yet so there's long way to go. A number of significant updates are still planned for V12 before that comes.

[RWeiss]

We have a bit of another opinion. I absolutely vote agains ReFS and 100% for NTFS. ReFS was never meant to be used on stand-alone volumes without parity. You'll loose the self healing ability of the filesystem. But whenever you run into an issue on your volume, you're in big trouble. There is practically no technical documentation on recovering your ReFS volumes. There is practically no tooling available either. You have refsutil.exe which is about as useful as a solar powered torch. We've had the bad luck of having two power strokes in the rack our backup machine (with ReFS) was in. The result was corruption on the volume both times. ReFS detects that, but as it was a stand-alone volume (ie. no parity and S2D or anything) it was unable to heal itself, and therefore both times it just marked the curreupted files as inaccessible. So there we were, we had something like 12TB of files inaccessible, but no way (not even through MS support) to get those files actually removed and reclaim the space. As a stand alone filesystem it's absolutely not mature enough to us, not even in server 2022. If MS can't even free up the space used by inaccessible files (simply not visible to the user anymore) that what else can I say than that.
One thing to note by the way, long shot though, that just as with deduplication, if you happen to have a block which is used in multiple backup files, and for some reason that block gets corrupted, you'll obviously loose all backups containing that block (or at least they won't be 100% correct anymore). Now your hardware should cover for that with media scans / scrubs or what have you. But still I don't like that idea.

Another thing is performance over time, when running from spindles. ReFS has some great features like block cloning, which one could explain as live deduplication, but then a bit smarter. This is great, but in the end your data, even of full backups, will be scattered all around your disks. We have one 12x10TB SATA 7200rpm with a P840i controller with... 2 or 4GB of cache (not sure anymore) for one of our repositories. We initially used ReFS on that, and it's great. Until, after in our case about 9-12 months, we started seeing severe performance degradation, and after about 2 years it was more or less completely useless, with crawling restores at 10MB/sec max. Try to restore a 5TB file server with that, or copy a job to tape. Now I recon 15krpm SAS disks would have performed better, but spindles just have their limits and throwing 80% random IO at them will most likely kill all spindles setup.

Now all depends on your requirements and hardware specs. When using SSD's, the performance degradation because of the fragmentation of files because of block-cloning will be much less (but still there). We still use ReFS on a remote repository which does have SSDs and it performs quite well. But for our main repository which is running from large spindles (as 100+TB of enterprise SSD storage is simply rather expensive for 'just' backups) ReFS gave us nothing but trouble. ReFS is fine until you run into an issue, and then you're on your own. In my eyes your backup should be something to rely on, and not specifically needs fancy or modern stuff (like block cloning, even given how nice it sounds). It should be reliable and proven, because when the shit hit the fan, you need it. With NTFS, we improved the stability and sustained backup performance a zillion times over ReFS (we now do about 300MB/sec opposed to 10 with ReFS). We moved back NTFS and we'll never look back for the data we really care about.

Probably not a popular opinion, but this is my honest view on the situation

And RAID5 is even out of the question if your array gets over 12TB. The URE rate of at least SATA drives is still specced at 10^14. That statistically would mean you can expect an unrecoverable read error every 12TB. If you lost a disk in your RAID5 and are rebuilding, the array has no way to detect or fix that, as there is no parity left. And that means, in theory, a rebuild would potentially corrupt your data. Now this URE at 10^14 is a very conservative value, it has been the same for at least 15 years now, while I'm convinced harddisks are in fact getting more reliable in terms of errors per TB or something. But as an example, a while ago WD announced their 22TB line of harddisks, and the URE rate is still specced at 10^14. And if you like your data, I think you should respect the specs, and that means RAID 5 is out of the question. One could even debate if old-skool RAID is the proper solution for really large arrays at all, but sometimes that's all you have especially when your machines are simply equiped with a RAID controller.

I have a similar opinion - for us, reliability / predictability is more important than higher performance in specific situations. If the performance deteriorates over the years, that is not really acceptable. Of course, file merging are expensive operations and in that case we would benefit a lot from it, but we work with active full backups in a lot of cases which is why this is a bit less relevant.
It's funny that you have a bad story with a dual power stroke - we just had a very similar thing. A power supply in one of our JBODs with 400TB (4x RAID6 with 10 disks each) shorted and took out the whole JBOD. Even though the JBOD had massive issues with the other power supply initially, and we lost multiple disks in RAIDs multiple times suddenly (disks missing, failing ,...) after a week the whole JBOD is back online without a single corruption issue. We only had to rebuild 3 drives in total (the drives themselves were fine). This all ran on a single LSI/Avago/Broadcom controller (i.e. hardware RAID) and NTFS volumes. I still find this quite incredible because I would not have faulted either the RAID controller nor the file system if we had lost all data, but it managed to keep everything together. We validated nearly 80TB of backups by now with the Veeam validation tool and have not found a single error...
We nevertheless ordered a new backup server and are again planning to go with hardware RAID and actually 22TB spinning disks - it's the reason I wanted to inform myself again regarding ReFS vs. NTFS but I'm still heavily leaning towards NTFS for our use case.

[dimaslan]

Isn't this something that could be addressed in ReFS using weekly (or shorter) synthetic fulls? I understand the risk of a corruption to a block shared by multiple full backups can cause a lot of trouble, however so can a corruption of an incremental restore point. For NTFS volumes you tend to do forever incremental so losing a whole 60 or 90 day chain is a risk too.
If one would like to keep the synthetic full advantages there is also XFS on a Linux physical server as a backup repository which can also provide immutability.

[Gostev]

You're correct in that NTFS does not provide any better protection against data loss. You need to test your backups regardless of backup storage you're using, and if you worry about storage-based corruption specifically then this is exactly what our automatic backup health check feature (storage-level corruption guard) is designed to deliver.

[Harti]

I am really interested, does really nobody has problems using ReFS?

I have tried to use ReFS since Microsoft said it is safe to use more than 10 years ago. I used it on single disks, external Raid 5 Storage, Internal software Raid 0 volumes.
I always used it for data I already have somewhere else, so only backup data
I cannot tell how oftern I lost the data - everytime I used it somewhere I stopped after turning on my computer and having an empty disk. Data recovery tools like Uneraser and reclaime can recover the data, but the Filesystem structure is mostly gone and they do not work on deduplicated volumes.
The last times I used it for fun between 2 RAID 5, one 14TB and one 7TB, storage systems to most data duplicated with robocopy
After 3(!) times dataloss on the ReFS Volume last year I stopped using ReFS and use only NTFS.
What is so special with Veeam, that ReFS is working for it?

[Gostev]

Oh, ReFS certainly works fine with Veeam. Vast majority if our Cloud Connect service providers and enterprise customers have been hosting their backup repositories on ReFS so we're talking exabytes of data on ReFS volumes with zero data loss issues (literally) over many, many years now. You just need a proper RAID controller with battery-backed write cache from Windows HCL in your backup repository server.

If your RAID controller doesn't have BBWC or if you use software RAID, then you will better off with XFS. This file system was designed and developed over 30 years ago so it doesn't expect much reliability for the underlying storage

[mkretzer]

Even with non HCL RAID Controllers and windows dynamic discs our > 1,4 PB of ReFS backup repos work flawlessly - they are just slower than XFS and less flexible.
Even with all the crashes in the beginning we never had data loss!!