Host-based backup of VMware vSphere VMs.
Post Reply
adam900331
Veteran
Posts: 300
Liked: 22 times
Joined: Dec 01, 2019 7:27 pm
Contact:

Veeam Backup Repository file system NTFS or ReFS?

Post by adam900331 »

Hy!

I am implementing the Veeam B&R with direct attached local storage in physical backup server. I would like to format the volume which will be used in backup repository. What is the suggestion related to the file system? Which can I use to best performance? NTFS or ReFS? Which allocation unit size can be set?

Thanks.
rennerstefan
Veeam Software
Posts: 627
Liked: 146 times
Joined: Jan 22, 2015 2:39 pm
Full Name: Stefan Renner
Location: Germany
Contact:

Re: Veeam Backup Repository file system NTFS or ReFS?

Post by rennerstefan » 3 people like this post

Hi Adam,

I would recommend to use ReFS as you will get the block cloning function with it.
Allocation unit size 64k.

Details can be found here:
https://bp.veeam.com/vbr/3_Build_struct ... block.html

Thanks
Stefan Renner

Veeam PMA
jorgedlcruz
Veeam Software
Posts: 1355
Liked: 613 times
Joined: Jul 17, 2015 6:54 pm
Full Name: Jorge de la Cruz
Contact:

Re: Veeam Backup Repository file system NTFS or ReFS?

Post by jorgedlcruz » 1 person likes this post

Hello,
Stefan already shared a great link. I would go ReFS as well.

Just bear in mind requirements as it seems it is all in one system, so you size it properly in terms of RAM for the SQL Express, plus the VBR components and management, and the most important the concurrent Backup Repository tasks. I am guessing the Proxies will be virtual? If that so, maybe even choose Linux if you like.

Also, as a quick question if you do not mind, any strategy for Backup copies? Perhaps Object Storage with immutability? Or a Veeam Service Provider? Maybe Tape?

Thanks a lot!
Jorge de la Cruz
Senior Product Manager | Veeam ONE @ Veeam Software

@jorgedlcruz
https://www.jorgedelacruz.es / https://jorgedelacruz.uk
vExpert 2014-2024 / InfluxAce / Grafana Champion
adam900331
Veteran
Posts: 300
Liked: 22 times
Joined: Dec 01, 2019 7:27 pm
Contact:

Re: Veeam Backup Repository file system NTFS or ReFS?

Post by adam900331 » 1 person likes this post

Hy!

Thanks the suggestion, I will use RFeS file system on backup repository.
The Backup Proxy will be physical server. The physical server has 16 cpu core and 64Gb RAM and will has direct SAN access to SAN network for direct SAN transport mode.

The backup plan is the following: the backup server which has local storage will be the primary backup repository(RFeS) and HPE StoreOnce will secondary backup repository. There will be backup copy job to copy the primary backup repository content to StoreOnce. And also will be tape library, the plan is: copy to tape from primary backup repository to tape.

So, the backup infrastructure only on-premise.

Thanks.
jorgedlcruz
Veeam Software
Posts: 1355
Liked: 613 times
Joined: Jul 17, 2015 6:54 pm
Full Name: Jorge de la Cruz
Contact:

Re: Veeam Backup Repository file system NTFS or ReFS?

Post by jorgedlcruz » 1 person likes this post

Thanks for coming back to us, Adam. It sounds really good, even if that Proxy is physical, you can leverage direct-SAN on Linux, just saying to save you one license perhaps, not needed if you are all Windows.

Great to see such a solid plan with the Backup Copies to HPE StoreOnce, and to Tape. Great design. Make sure to enable immutability in HPE StoreOnce when Veeam v12 comes out, you might need latest HPE StoreOnce firmware.

Other than that, awesome!
Jorge de la Cruz
Senior Product Manager | Veeam ONE @ Veeam Software

@jorgedlcruz
https://www.jorgedelacruz.es / https://jorgedelacruz.uk
vExpert 2014-2024 / InfluxAce / Grafana Champion
adam900331
Veteran
Posts: 300
Liked: 22 times
Joined: Dec 01, 2019 7:27 pm
Contact:

Re: Veeam Backup Repository file system NTFS or ReFS?

Post by adam900331 »

Hello!

Thank you for Your feedback. :)

If I have more technical question, I will ask in forum.

Adam
vmtech123
Veeam Legend
Posts: 235
Liked: 133 times
Joined: Mar 28, 2019 2:01 pm
Full Name: SP
Contact:

Re: Veeam Backup Repository file system NTFS or ReFS?

Post by vmtech123 » 2 people like this post

64k REFS all the way .
adam900331
Veteran
Posts: 300
Liked: 22 times
Joined: Dec 01, 2019 7:27 pm
Contact:

Re: Veeam Backup Repository file system NTFS or ReFS?

Post by adam900331 »

Hy!

I have another question. I checked the mentioned link and I will use in backup job "Local target" compression setting and the backup repository will be ReFS with 64kb block size. But what kind of RAID level should I use for the best performance? RAID5 with 128kb strip size is suitable? The server contain 7 pieces 2Tb SAS disks.

Thanks.
barlid
Enthusiast
Posts: 29
Liked: 17 times
Joined: Nov 01, 2019 3:37 am
Full Name: Barli Dharmajaya
Location: Jkt, ID
Contact:

Re: Veeam Backup Repository file system NTFS or ReFS?

Post by barlid » 1 person likes this post

Hi Adam,

For backup performance purpose, In my opinion you can use RAID 5 since offers better performance than RAID 6, for block size on link shared, the best practice for ReFS use 256 block size configuration.
But since raid 5 only tolerate 1 disk failure, you need to consider put hot spare disk on configuration if possible.

Thanks
MarkBoothmaa
Veeam Legend
Posts: 179
Liked: 47 times
Joined: Mar 22, 2017 11:10 am
Full Name: Mark Boothman
Location: Darlington, United Kingdom
Contact:

Re: Veeam Backup Repository file system NTFS or ReFS?

Post by MarkBoothmaa » 1 person likes this post

For stripe size I use 256kb
I use RAID 6 but as it gives me the extra parity and that is built into our sizing.
agrob
Veteran
Posts: 380
Liked: 48 times
Joined: Sep 05, 2011 1:31 pm
Full Name: Andre
Contact:

Re: Veeam Backup Repository file system NTFS or ReFS?

Post by agrob » 1 person likes this post

i would also recommend using Raid6 instead of Raid5. specially if you have an array with traditional HDD Drives with big sizes. It can take hours or days to rebuild if one such big drive fails. It also depends on the load you have on the Array appart of rebuild. For exampe if you have a lot of backup I/O going on.
rennerstefan
Veeam Software
Posts: 627
Liked: 146 times
Joined: Jan 22, 2015 2:39 pm
Full Name: Stefan Renner
Location: Germany
Contact:

Re: Veeam Backup Repository file system NTFS or ReFS?

Post by rennerstefan » 1 person likes this post

I mostly agree with what is mentioned above. Nevertheless the decision if RAID5 or RAID6 should depend on a couple of facts:
1. How many disks do I have? In your case RAID5 will leave you with 5 data/performance disks (5+1; +1 Hot Spare), while RAID6 will only give you 4 data/performance disks. With only a little number of disks the performance may not be as good an of course you looks almost 50% of the volume in your case

2. How does my backup strategy look like? if you for example use Veeam Snapshot Orchestration on your primary array plus backup copy jobs so that you have multiple copies of your backup it is a very different story than having only one single repository where i loose all my backups in case it fails

Hope that also helps to decide.
Stefan Renner

Veeam PMA
RGijsen
Expert
Posts: 124
Liked: 25 times
Joined: Oct 10, 2014 2:06 pm
Contact:

Re: Veeam Backup Repository file system NTFS or ReFS?

Post by RGijsen » 2 people like this post

We have a bit of another opinion. I absolutely vote agains ReFS and 100% for NTFS. ReFS was never meant to be used on stand-alone volumes without parity. You'll loose the self healing ability of the filesystem. But whenever you run into an issue on your volume, you're in big trouble. There is practically no technical documentation on recovering your ReFS volumes. There is practically no tooling available either. You have refsutil.exe which is about as useful as a solar powered torch. We've had the bad luck of having two power strokes in the rack our backup machine (with ReFS) was in. The result was corruption on the volume both times. ReFS detects that, but as it was a stand-alone volume (ie. no parity and S2D or anything) it was unable to heal itself, and therefore both times it just marked the curreupted files as inaccessible. So there we were, we had something like 12TB of files inaccessible, but no way (not even through MS support) to get those files actually removed and reclaim the space. As a stand alone filesystem it's absolutely not mature enough to us, not even in server 2022. If MS can't even free up the space used by inaccessible files (simply not visible to the user anymore) that what else can I say than that.
One thing to note by the way, long shot though, that just as with deduplication, if you happen to have a block which is used in multiple backup files, and for some reason that block gets corrupted, you'll obviously loose all backups containing that block (or at least they won't be 100% correct anymore). Now your hardware should cover for that with media scans / scrubs or what have you. But still I don't like that idea.

Another thing is performance over time, when running from spindles. ReFS has some great features like block cloning, which one could explain as live deduplication, but then a bit smarter. This is great, but in the end your data, even of full backups, will be scattered all around your disks. We have one 12x10TB SATA 7200rpm with a P840i controller with... 2 or 4GB of cache (not sure anymore) for one of our repositories. We initially used ReFS on that, and it's great. Until, after in our case about 9-12 months, we started seeing severe performance degradation, and after about 2 years it was more or less completely useless, with crawling restores at 10MB/sec max. Try to restore a 5TB file server with that, or copy a job to tape. Now I recon 15krpm SAS disks would have performed better, but spindles just have their limits and throwing 80% random IO at them will most likely kill all spindles setup.

Now all depends on your requirements and hardware specs. When using SSD's, the performance degradation because of the fragmentation of files because of block-cloning will be much less (but still there). We still use ReFS on a remote repository which does have SSDs and it performs quite well. But for our main repository which is running from large spindles (as 100+TB of enterprise SSD storage is simply rather expensive for 'just' backups) ReFS gave us nothing but trouble. ReFS is fine until you run into an issue, and then you're on your own. In my eyes your backup should be something to rely on, and not specifically needs fancy or modern stuff (like block cloning, even given how nice it sounds). It should be reliable and proven, because when the shit hit the fan, you need it. With NTFS, we improved the stability and sustained backup performance a zillion times over ReFS (we now do about 300MB/sec opposed to 10 with ReFS). We moved back NTFS and we'll never look back for the data we really care about.

Probably not a popular opinion, but this is my honest view on the situation :)

And RAID5 is even out of the question if your array gets over 12TB. The URE rate of at least SATA drives is still specced at 10^14. That statistically would mean you can expect an unrecoverable read error every 12TB. If you lost a disk in your RAID5 and are rebuilding, the array has no way to detect or fix that, as there is no parity left. And that means, in theory, a rebuild would potentially corrupt your data. Now this URE at 10^14 is a very conservative value, it has been the same for at least 15 years now, while I'm convinced harddisks are in fact getting more reliable in terms of errors per TB or something. But as an example, a while ago WD announced their 22TB line of harddisks, and the URE rate is still specced at 10^14. And if you like your data, I think you should respect the specs, and that means RAID 5 is out of the question. One could even debate if old-skool RAID is the proper solution for really large arrays at all, but sometimes that's all you have especially when your machines are simply equiped with a RAID controller.
ITP-Stan
Service Provider
Posts: 201
Liked: 55 times
Joined: Feb 18, 2013 10:45 am
Full Name: Stan (IF-IT4U)
Contact:

Re: Veeam Backup Repository file system NTFS or ReFS?

Post by ITP-Stan » 1 person likes this post

If you have plenty of space, choose NTFS as your primary repo.
For the secondary repo, with GFS perhaps, I would choose ReFS 64K.
dloseke
Service Provider
Posts: 60
Liked: 28 times
Joined: Jul 13, 2018 3:33 pm
Full Name: Derek M. Loseke
Contact:

Re: Veeam Backup Repository file system NTFS or ReFS?

Post by dloseke » 1 person likes this post

I recommend a third option, XFS.

REFS is great, but it certainly has it's caveats. First off, Microsoft had a string of updates this year that caused REFS volumes to show up as RAW disks, and until the updates were removed, the backups would continue to fail. After the updates were removed and the server rebooted, the volumes showed up again as REFS and backups could continue. There was a fix found to disable the drive from showing as a removable disk and so far this has prevented this from happening in the future.

That said, I was using REFS for pretty much everything, but given some issues that can arise when using prosumer/SMB-grade NAS devices such as QNAP and Synology and coupling those possible issues with the shortcomings of REFS, I no longer use REFS when a NAS is the storage backing device. For those devices, we are using NTFS right now but plan on moving forward with using Linux repository servers and the XFS filesystem when possible. If we're using a physical Windows server with a battery-backed cache, then REFS works great.
Derek M. Loseke, Senior Systems Engineer | Veeam Legend 2022-2023 | VMSP/VMTSP | VCP6-DCV | VSP/VTSP | CCNA | https://technotesanddadjokes.com | @dloseke
ITP-Stan
Service Provider
Posts: 201
Liked: 55 times
Joined: Feb 18, 2013 10:45 am
Full Name: Stan (IF-IT4U)
Contact:

Re: Veeam Backup Repository file system NTFS or ReFS?

Post by ITP-Stan »

If you use a Synology NAS, and plan to use XFS, do you present an iSCSI volume on a virtual Linux repository server?
And then format the volume there with XFS to use fastcloning?
jorgedlcruz
Veeam Software
Posts: 1355
Liked: 613 times
Joined: Jul 17, 2015 6:54 pm
Full Name: Jorge de la Cruz
Contact:

Re: Veeam Backup Repository file system NTFS or ReFS?

Post by jorgedlcruz »

Hello Stan,
I would recommend that way, yes. In terms of performance it is never the best performance, as you will have proxies working, sending data to another ESXi, and that ESXi sending data finally to the Synology.

That, and of course that somebody can gain access to the ESXi, and therefore open the console, do something like reboot in safe mode to change root password, etc. The usual caveats.

But if all is hardened and secure enough, on this specific case. Yes, iSCSI to a Linux VM, XFS, and enable immutability.

Thanks!
Jorge de la Cruz
Senior Product Manager | Veeam ONE @ Veeam Software

@jorgedlcruz
https://www.jorgedelacruz.es / https://jorgedelacruz.uk
vExpert 2014-2024 / InfluxAce / Grafana Champion
Ciso_2021
Enthusiast
Posts: 52
Liked: 7 times
Joined: Sep 13, 2021 7:19 pm
Full Name: Julien Ange
Contact:

Re: Veeam Backup Repository file system NTFS or ReFS?

Post by Ciso_2021 »

Just go with a real hardware for security And Linux with MFA access.
VMware as back repository its kinda s crazy lately
ITP-Stan
Service Provider
Posts: 201
Liked: 55 times
Joined: Feb 18, 2013 10:45 am
Full Name: Stan (IF-IT4U)
Contact:

Re: Veeam Backup Repository file system NTFS or ReFS?

Post by ITP-Stan »

If the money is there, of-course.

For small clients maybe a little PC with Linux on it and iSCSI volume from the Synology NAS.
albertwt
Veeam Legend
Posts: 879
Liked: 46 times
Joined: Nov 05, 2009 12:24 pm
Location: Sydney, NSW
Contact:

Re: Veeam Backup Repository file system NTFS or ReFS?

Post by albertwt » 1 person likes this post

Yes, for Linux, use XFS with the below parameters:

Code: Select all

parted --script /dev/sdb "mklabel gpt" 
parted --script /dev/sdb "mkpart primary 0% 100%" 
mkfs.xfs -b size=4096 -m reflink=1,crc=1 /dev/sdb -f
and for Windows, use the ReFS, with the below parameters:

Code: Select all

$paramFormatVolume = @{
	DriveLetter	       = 'B'
	FileSystem		   = 'REFS'
	AllocationUnitSize = 65536
	Force			   = $true
	Confirm		       = $false
	NewFileSystemLabel = "BackupRepository"
	UseLargeFRS	       = $true
}

Format-Volume @paramFormatVolume
--
/* Veeam software enthusiast user & supporter ! */
mkretzer
Veeam Legend
Posts: 1140
Liked: 387 times
Joined: Dec 17, 2015 7:17 am
Contact:

Re: Veeam Backup Repository file system NTFS or ReFS?

Post by mkretzer » 1 person likes this post

One more tip: Use LVM under the XFS - this way you can move the data to a new storage device (for example to an external FC or iSCSI system) without interruption to backup via pvmove. We also use smaller volumes/partitions/raids under the lvm so we have smaller units to move.
Even with V12 you need to stop backups to move them!
Ciso_2021
Enthusiast
Posts: 52
Liked: 7 times
Joined: Sep 13, 2021 7:19 pm
Full Name: Julien Ange
Contact:

Re: Veeam Backup Repository file system NTFS or ReFS?

Post by Ciso_2021 »

Is refs only using a fast cloning or it’s also immutable as Ubuntu harden repo ?
I just figured out we cannot use the hardened repo with our offices.
Each office has own veeam server and we use copy jobs at night to the hardend repository.
mkretzer
Veeam Legend
Posts: 1140
Liked: 387 times
Joined: Dec 17, 2015 7:17 am
Contact:

Re: Veeam Backup Repository file system NTFS or ReFS?

Post by mkretzer » 1 person likes this post

ReFS has no immutability build in.
Gostev
Chief Product Officer
Posts: 31457
Liked: 6648 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Veeam Backup Repository file system NTFS or ReFS?

Post by Gostev »

RGijsen wrote: Sep 26, 2022 7:56 amNow all depends on your requirements and hardware specs. When using SSD's, the performance degradation because of the fragmentation of files because of block-cloning will be much less (but still there). We still use ReFS on a remote repository which does have SSDs and it performs quite well. But for our main repository which is running from large spindles (as 100+TB of enterprise SSD storage is simply rather expensive for 'just' backups) ReFS gave us nothing but trouble [...] With NTFS, we improved the stability and sustained backup performance a zillion times over ReFS (we now do about 300MB/sec opposed to 10 with ReFS).
We have found an issue with this on our end. Apparently our engine still contains and makes an NTFS-specific file system call that does not apply to ReFS. While it does nothing, it significantly reduces the processing performance particularly on fragmented ReFS volumes. We will try to remove it in the next update, which should increase performance on fragmented ReFS volumes significantly.
Post Reply

Who is online

Users browsing this forum: No registered users and 77 guests