Veeam Backup Repository file system NTFS or ReFS?

[adam900331]

Hy!

I am implementing the Veeam B&R with direct attached local storage in physical backup server. I would like to format the volume which will be used in backup repository. What is the suggestion related to the file system? Which can I use to best performance? NTFS or ReFS? Which allocation unit size can be set?

Thanks.

[rennerstefan]

Hi Adam,

I would recommend to use ReFS as you will get the block cloning function with it.
Allocation unit size 64k.

Details can be found here:
https://bp.veeam.com/vbr/3_Build_struct ... block.html

Thanks

[jorgedlcruz]

Hello,
Stefan already shared a great link. I would go ReFS as well.

Just bear in mind requirements as it seems it is all in one system, so you size it properly in terms of RAM for the SQL Express, plus the VBR components and management, and the most important the concurrent Backup Repository tasks. I am guessing the Proxies will be virtual? If that so, maybe even choose Linux if you like.

Also, as a quick question if you do not mind, any strategy for Backup copies? Perhaps Object Storage with immutability? Or a Veeam Service Provider? Maybe Tape?

Thanks a lot!

[adam900331]

Hy!

Thanks the suggestion, I will use RFeS file system on backup repository.
The Backup Proxy will be physical server. The physical server has 16 cpu core and 64Gb RAM and will has direct SAN access to SAN network for direct SAN transport mode.

The backup plan is the following: the backup server which has local storage will be the primary backup repository(RFeS) and HPE StoreOnce will secondary backup repository. There will be backup copy job to copy the primary backup repository content to StoreOnce. And also will be tape library, the plan is: copy to tape from primary backup repository to tape.

So, the backup infrastructure only on-premise.

Thanks.

[jorgedlcruz]

Thanks for coming back to us, Adam. It sounds really good, even if that Proxy is physical, you can leverage direct-SAN on Linux, just saying to save you one license perhaps, not needed if you are all Windows.

Great to see such a solid plan with the Backup Copies to HPE StoreOnce, and to Tape. Great design. Make sure to enable immutability in HPE StoreOnce when Veeam v12 comes out, you might need latest HPE StoreOnce firmware.

Other than that, awesome!

[adam900331]

Hello!

Thank you for Your feedback.

If I have more technical question, I will ask in forum.

Adam

[vmtech123]

64k REFS all the way .

[adam900331]

Hy!

I have another question. I checked the mentioned link and I will use in backup job "Local target" compression setting and the backup repository will be ReFS with 64kb block size. But what kind of RAID level should I use for the best performance? RAID5 with 128kb strip size is suitable? The server contain 7 pieces 2Tb SAS disks.

Thanks.

[barlid]

Hi Adam,

For backup performance purpose, In my opinion you can use RAID 5 since offers better performance than RAID 6, for block size on link shared, the best practice for ReFS use 256 block size configuration.
But since raid 5 only tolerate 1 disk failure, you need to consider put hot spare disk on configuration if possible.

Thanks

[MarkBoothmaa]

For stripe size I use 256kb
I use RAID 6 but as it gives me the extra parity and that is built into our sizing.

[agrob]

i would also recommend using Raid6 instead of Raid5. specially if you have an array with traditional HDD Drives with big sizes. It can take hours or days to rebuild if one such big drive fails. It also depends on the load you have on the Array appart of rebuild. For exampe if you have a lot of backup I/O going on.

[rennerstefan]

I mostly agree with what is mentioned above. Nevertheless the decision if RAID5 or RAID6 should depend on a couple of facts:
1. How many disks do I have? In your case RAID5 will leave you with 5 data/performance disks (5+1; +1 Hot Spare), while RAID6 will only give you 4 data/performance disks. With only a little number of disks the performance may not be as good an of course you looks almost 50% of the volume in your case

2. How does my backup strategy look like? if you for example use Veeam Snapshot Orchestration on your primary array plus backup copy jobs so that you have multiple copies of your backup it is a very different story than having only one single repository where i loose all my backups in case it fails

Hope that also helps to decide.

[RGijsen]

We have a bit of another opinion. I absolutely vote agains ReFS and 100% for NTFS. ReFS was never meant to be used on stand-alone volumes without parity. You'll loose the self healing ability of the filesystem. But whenever you run into an issue on your volume, you're in big trouble. There is practically no technical documentation on recovering your ReFS volumes. There is practically no tooling available either. You have refsutil.exe which is about as useful as a solar powered torch. We've had the bad luck of having two power strokes in the rack our backup machine (with ReFS) was in. The result was corruption on the volume both times. ReFS detects that, but as it was a stand-alone volume (ie. no parity and S2D or anything) it was unable to heal itself, and therefore both times it just marked the curreupted files as inaccessible. So there we were, we had something like 12TB of files inaccessible, but no way (not even through MS support) to get those files actually removed and reclaim the space. As a stand alone filesystem it's absolutely not mature enough to us, not even in server 2022. If MS can't even free up the space used by inaccessible files (simply not visible to the user anymore) that what else can I say than that.
One thing to note by the way, long shot though, that just as with deduplication, if you happen to have a block which is used in multiple backup files, and for some reason that block gets corrupted, you'll obviously loose all backups containing that block (or at least they won't be 100% correct anymore). Now your hardware should cover for that with media scans / scrubs or what have you. But still I don't like that idea.

Another thing is performance over time, when running from spindles. ReFS has some great features like block cloning, which one could explain as live deduplication, but then a bit smarter. This is great, but in the end your data, even of full backups, will be scattered all around your disks. We have one 12x10TB SATA 7200rpm with a P840i controller with... 2 or 4GB of cache (not sure anymore) for one of our repositories. We initially used ReFS on that, and it's great. Until, after in our case about 9-12 months, we started seeing severe performance degradation, and after about 2 years it was more or less completely useless, with crawling restores at 10MB/sec max. Try to restore a 5TB file server with that, or copy a job to tape. Now I recon 15krpm SAS disks would have performed better, but spindles just have their limits and throwing 80% random IO at them will most likely kill all spindles setup.

Now all depends on your requirements and hardware specs. When using SSD's, the performance degradation because of the fragmentation of files because of block-cloning will be much less (but still there). We still use ReFS on a remote repository which does have SSDs and it performs quite well. But for our main repository which is running from large spindles (as 100+TB of enterprise SSD storage is simply rather expensive for 'just' backups) ReFS gave us nothing but trouble. ReFS is fine until you run into an issue, and then you're on your own. In my eyes your backup should be something to rely on, and not specifically needs fancy or modern stuff (like block cloning, even given how nice it sounds). It should be reliable and proven, because when the shit hit the fan, you need it. With NTFS, we improved the stability and sustained backup performance a zillion times over ReFS (we now do about 300MB/sec opposed to 10 with ReFS). We moved back NTFS and we'll never look back for the data we really care about.

Probably not a popular opinion, but this is my honest view on the situation

And RAID5 is even out of the question if your array gets over 12TB. The URE rate of at least SATA drives is still specced at 10^14. That statistically would mean you can expect an unrecoverable read error every 12TB. If you lost a disk in your RAID5 and are rebuilding, the array has no way to detect or fix that, as there is no parity left. And that means, in theory, a rebuild would potentially corrupt your data. Now this URE at 10^14 is a very conservative value, it has been the same for at least 15 years now, while I'm convinced harddisks are in fact getting more reliable in terms of errors per TB or something. But as an example, a while ago WD announced their 22TB line of harddisks, and the URE rate is still specced at 10^14. And if you like your data, I think you should respect the specs, and that means RAID 5 is out of the question. One could even debate if old-skool RAID is the proper solution for really large arrays at all, but sometimes that's all you have especially when your machines are simply equiped with a RAID controller.

[ITP-Stan]

If you have plenty of space, choose NTFS as your primary repo.
For the secondary repo, with GFS perhaps, I would choose ReFS 64K.

[dloseke]

I recommend a third option, XFS.

REFS is great, but it certainly has it's caveats. First off, Microsoft had a string of updates this year that caused REFS volumes to show up as RAW disks, and until the updates were removed, the backups would continue to fail. After the updates were removed and the server rebooted, the volumes showed up again as REFS and backups could continue. There was a fix found to disable the drive from showing as a removable disk and so far this has prevented this from happening in the future.

That said, I was using REFS for pretty much everything, but given some issues that can arise when using prosumer/SMB-grade NAS devices such as QNAP and Synology and coupling those possible issues with the shortcomings of REFS, I no longer use REFS when a NAS is the storage backing device. For those devices, we are using NTFS right now but plan on moving forward with using Linux repository servers and the XFS filesystem when possible. If we're using a physical Windows server with a battery-backed cache, then REFS works great.

[ITP-Stan]

If you use a Synology NAS, and plan to use XFS, do you present an iSCSI volume on a virtual Linux repository server?
And then format the volume there with XFS to use fastcloning?

[jorgedlcruz]

Hello Stan,
I would recommend that way, yes. In terms of performance it is never the best performance, as you will have proxies working, sending data to another ESXi, and that ESXi sending data finally to the Synology.

That, and of course that somebody can gain access to the ESXi, and therefore open the console, do something like reboot in safe mode to change root password, etc. The usual caveats.

But if all is hardened and secure enough, on this specific case. Yes, iSCSI to a Linux VM, XFS, and enable immutability.

Thanks!

[Ciso_2021]

Just go with a real hardware for security And Linux with MFA access.
VMware as back repository its kinda s crazy lately

[ITP-Stan]

If the money is there, of-course.

For small clients maybe a little PC with Linux on it and iSCSI volume from the Synology NAS.

[albertwt]

Yes, for Linux, use XFS with the below parameters:

Code: Select all

parted --script /dev/sdb "mklabel gpt" 
parted --script /dev/sdb "mkpart primary 0% 100%" 
mkfs.xfs -b size=4096 -m reflink=1,crc=1 /dev/sdb -f

and for Windows, use the ReFS, with the below parameters:

Code: Select all

$paramFormatVolume = @{
	DriveLetter	       = 'B'
	FileSystem		   = 'REFS'
	AllocationUnitSize = 65536
	Force			   = $true
	Confirm		       = $false
	NewFileSystemLabel = "BackupRepository"
	UseLargeFRS	       = $true
}

Format-Volume @paramFormatVolume

[mkretzer]

One more tip: Use LVM under the XFS - this way you can move the data to a new storage device (for example to an external FC or iSCSI system) without interruption to backup via pvmove. We also use smaller volumes/partitions/raids under the lvm so we have smaller units to move.
Even with V12 you need to stop backups to move them!

[Ciso_2021]

Is refs only using a fast cloning or it’s also immutable as Ubuntu harden repo ?
I just figured out we cannot use the hardened repo with our offices.
Each office has own veeam server and we use copy jobs at night to the hardend repository.

[mkretzer]

ReFS has no immutability build in.

[Gostev]

Now all depends on your requirements and hardware specs. When using SSD's, the performance degradation because of the fragmentation of files because of block-cloning will be much less (but still there). We still use ReFS on a remote repository which does have SSDs and it performs quite well. But for our main repository which is running from large spindles (as 100+TB of enterprise SSD storage is simply rather expensive for 'just' backups) ReFS gave us nothing but trouble [...] With NTFS, we improved the stability and sustained backup performance a zillion times over ReFS (we now do about 300MB/sec opposed to 10 with ReFS).

We have found an issue with this on our end. Apparently our engine still contains and makes an NTFS-specific file system call that does not apply to ReFS. While it does nothing bad, it significantly reduces the processing performance on heavily fragmented ReFS volumes. We will try to remove it in the next update.

[DataDefender]

Just switched our primary backup storage from NTFS to ReFS with a 64k cluster size, configured in RAID 6 with a hot spare. At the moment, it's performing quite well. Using synchronous full backups with fast cloning instead of active full backups for our Agent Backups saves us a ton of time. VMs configured for forward incremental backups don't seem to benefit as much, though.

I hope we won't experience the mentioned performance degradation. The RAID 6 setup is on spindle drives, and currently, it manages roughly 1 GB/s for both reads and writes.

Cheers,
DD

[Gostev]

Eventually you will but it will take a while to surface.

[karsten123]

we have a heavily used ReFS repository on Server 2016 for 3+years. flawless so far. hardware is HPE Apollo 4200, 24x SAS RAID60.

[mkretzer]

@Gostev will there be an update for V12.1 or will it be in V13?

[Gostev]

V12 will still be getting many updates

[mkretzer]

@Gostev I mean a thread-relevant update about this issue "Apparently our engine still contains and makes an NTFS-specific file system call that does not apply to ReFS".
Sorry if thats what you meant