2016 ReFS and file allocation size

Availability for the Always-On Enterprise

2016 ReFS and file allocation size

Veeam Logoby chrmol » Tue Nov 08, 2016 7:35 am

Quick question - when using ReFS in Windows 2016 for Veeam Repo. - Should it be formatted with 4k or 64k ?
(Lab upgrade in progress :-) )
chrmol
Enthusiast
 
Posts: 36
Liked: 2 times
Joined: Mon May 17, 2010 7:41 pm
Location: Denmark
Full Name: Christian Moeller

Re: 2016 ReFS and file allocation size

Veeam Logoby dellock6 » Tue Nov 08, 2016 8:39 am

Hi,
we had internally the same discussion lat week, and Tom Sightler executed some tests to verify some information. It seems there is a bit of overhead using 64k cluster size, expected as there are chances where a smaller block uses the entire 64k cluster but leaving some space empty. Even with this limit, we feel like 64k could be the suggested size, as also Microsoft seems to suggest this size for all the scenarios where large files are in use (Exchange and SQL databases, Hyper-V virtual machine disks).
Luca Dell'Oca
EMEA Cloud Architect @ Veeam Software

@dellock6
http://www.virtualtothecore.com
vExpert 2011-2012-2013-2014-2015-2016
Veeam VMCE #1
dellock6
Veeam Software
 
Posts: 4877
Liked: 1280 times
Joined: Sun Jul 26, 2009 3:39 pm
Location: Varese, Italy
Full Name: Luca Dell'Oca

Re: 2016 ReFS and file allocation size

Veeam Logoby chrmol » Tue Nov 08, 2016 9:31 am

Ok, Ill go with 64K :-)
chrmol
Enthusiast
 
Posts: 36
Liked: 2 times
Joined: Mon May 17, 2010 7:41 pm
Location: Denmark
Full Name: Christian Moeller

Re: 2016 ReFS and file allocation size

Veeam Logoby nmdange » Tue Nov 08, 2016 2:57 pm

I thought ReFS only supported 64k cluster size?
nmdange
Expert
 
Posts: 150
Liked: 48 times
Joined: Thu Aug 20, 2015 9:30 pm

Re: 2016 ReFS and file allocation size

Veeam Logoby tsightler » Tue Nov 08, 2016 4:21 pm 3 people like this post

In versions of WIndows prior to Windows 2016 the 64K cluster size was the only available option for ReFS, however, on Windows 2016 with ReFS 3, you can select either 4K or 64K clusters. I was able to format even a 250TB volume with 4K clusters so I'm not exactly sure when the cutoff is where you have to use 64K in Windows 2016. While I don't know this for sure, my guess is that Microsoft introduced 4K clusters for ReFS to make the filesystem more attractive for generic file server workloads, where the overhead of storing millions of relatively small files in 64K blocks was a huge barrier to acceptance.

In relatively small scale testing (100's of GBs, not TBs) to this point the space difference between 4K and 64K clusters is <10%, although the exact amount will vary based on the selected Veeam block size and compression ratio achieved with the specific data set (the above results were using Veeam defaults and a small mix of Windows and Linux VMs). However, some of this overhead is offset by the space saved from having to track 16x more clusters when using 4K. It's actually fairly easy to estimate the overhead if you know the Veeam block size (which by default is 1MB) and compression ratio.

For now my recommendation for a Veeam repository, as well as our internal consensus, is to use 64K clusters, as it seems to be the safe bet from a scalability and stability perspective and we know it will work well for every size repository. It's hard to imagine that, as the size of the repository grows, the benefit of having to track 16x less clusters won't be an advantage for the larger block sizes. That's not to say that a 4K cluster size repository won't work, but we simply don't have enough data from the field to suggest when it will break, and, in the end, that's the most important question that must be answered before something can be recommended. As we perform additional testing and continue to discuss with Microsoft, as well as collect more results from the field, this recommendation may certainly change.
tsightler
Veeam Software
 
Posts: 4687
Liked: 1699 times
Joined: Fri Jun 05, 2009 12:57 pm
Full Name: Tom Sightler

Re: 2016 ReFS and file allocation size

Veeam Logoby nmdange » Tue Nov 08, 2016 5:09 pm

If the block size is 1MB wouldn't the backup files always be aligned to 1MB block size (and hence would never have a block of data less than 1mb let alone 64k in size), which would mean no overhead for the Veeam backup files?
nmdange
Expert
 
Posts: 150
Liked: 48 times
Joined: Thu Aug 20, 2015 9:30 pm

Re: 2016 ReFS and file allocation size

Veeam Logoby dellock6 » Tue Nov 08, 2016 5:19 pm

This is true if compression is turned off. If it's on, the original block is 1MB, but it then becomes anything between 1MB (uncompressable block) and any lower value. Say I have after compression a file that is 644k, this consumes 10 * 64k blocks plus a 4k space. So I have 60k unused and unusable in the 11th block.
Luca Dell'Oca
EMEA Cloud Architect @ Veeam Software

@dellock6
http://www.virtualtothecore.com
vExpert 2011-2012-2013-2014-2015-2016
Veeam VMCE #1
dellock6
Veeam Software
 
Posts: 4877
Liked: 1280 times
Joined: Sun Jul 26, 2009 3:39 pm
Location: Varese, Italy
Full Name: Luca Dell'Oca

Re: 2016 ReFS and file allocation size

Veeam Logoby nmdange » Tue Nov 08, 2016 5:45 pm

Ahhhh that makes sense. Unless you have a huge number of backup files, I wouldn't think the overhead would be that big, and really on a multi-TB volume what's a few 60k here and there :)
nmdange
Expert
 
Posts: 150
Liked: 48 times
Joined: Thu Aug 20, 2015 9:30 pm

Re: 2016 ReFS and file allocation size

Veeam Logoby tsightler » Tue Nov 08, 2016 6:56 pm 6 people like this post

nmdange wrote:Ahhhh that makes sense. Unless you have a huge number of backup files, I wouldn't think the overhead would be that big, and really on a multi-TB volume what's a few 60k here and there :)

True, but remember, the alignment is per-block within the backup file, not per-file, because blocks within the backup file are the elements we want to be able to clone, and clone operations have to be on cluster boundaries.

After compression, each block backed up by Veeam will have somewhere between 0-4K of overhead when stored on ReFS with 4K clusters, and somewhere between 0-64K of overhead when stored on ReFS volume with 64K clusters. Perhaps a simpler way to look at it, 4K clusters will have an average 2K overhead, and 64K cluster will have an average 32K overhead. That's 30K more overhead per source block, because the number of blocks in a given VM is fixed, based on the configured block size.

Simple example: Single VM with 10GB of data
10GB/1MB blocks = 10,000 blocks

10,000 blocks * 2K = 20MB estimated overhead with 4K clusters
10,000 blocks * 32K = 320MB estimated overhead with 64K clusters

Now let's calculate the overhead as a % of total space. Let's assume a somewhat typical 2.5x compression ratio so that means:
10GB/2.5x compression = 4GB VBK

So the backup file with no overhead would be 4GB so the overhead of 4K clusters (20MB) is 0.5% while the overhead with 64K clusters (320MB) is more like 8%. This means the resulting backup files are 0.5% and 8% larger than if it had no alignment, or roughly 7.5% difference between 4K and 64K.

Of course the math changes if the compression ratio changes, say we only get 1.5x compression, so suddenly the backup file is 6.65GB but the overhead per block doesn't change. Since the backup itself is larger, the fixed space of the overhead represents a smaller percentage, only ~5% with 64K clusters, and 0.3% with 4K.

Also, the math changes if you change the block size. If you went from the default of Local target (1MB blocks), to LAN target (512K blocks), suddenly that's 20,000 blocks and, since the alignment overhead is per-block, that doubles my overhead with 64K clusters to 640MB. The backup size is likely to be the same, so that 8% overhead just became 16%!

Once again, some of that overhead is made up for my having smaller allocation tables and I believe the proper balance is default settings for Veeam block size and 64K clusters, but it's interesting to understand the impact of it all. :D
tsightler
Veeam Software
 
Posts: 4687
Liked: 1699 times
Joined: Fri Jun 05, 2009 12:57 pm
Full Name: Tom Sightler

Re: 2016 ReFS and file allocation size

Veeam Logoby nmdange » Tue Nov 08, 2016 8:15 pm

And that is why I love these forums :D
nmdange
Expert
 
Posts: 150
Liked: 48 times
Joined: Thu Aug 20, 2015 9:30 pm

Re: 2016 ReFS and file allocation size

Veeam Logoby chrmol » Wed Nov 16, 2016 5:21 pm

In my lab I use a windows 10 machine as Veeam repository (for copy jobs).
... But does ReFS in Windows 10 provide the same functionality as Windows 2016 ?
chrmol
Enthusiast
 
Posts: 36
Liked: 2 times
Joined: Mon May 17, 2010 7:41 pm
Location: Denmark
Full Name: Christian Moeller

Re: 2016 ReFS and file allocation size

Veeam Logoby v.Eremin » Wed Nov 16, 2016 6:30 pm

... But does ReFS in Windows 10 provide the same functionality as Windows 2016 ?

No, it does not.
v.Eremin
Veeam Software
 
Posts: 12703
Liked: 920 times
Joined: Fri Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin

Re: 2016 ReFS and file allocation size

Veeam Logoby vClintWyckoff » Wed Nov 16, 2016 8:54 pm

So you have manually enabled to permit the formatting of ReFS on Win 10?

Just run...

Code: Select all
C:\WINDOWS\system32>fsutil fsinfo volumeinfo <DRIVE>


...against both ReFS volumes and you'll see which functions each volume supports.
vClintWyckoff
Veeam Software
 
Posts: 348
Liked: 79 times
Joined: Sat Oct 27, 2012 1:22 am
Location: Technical Evangelist
Full Name: Clint Wyckoff

Re: 2016 ReFS and file allocation size

Veeam Logoby chrmol » Thu Nov 17, 2016 7:06 am

Actually what I meant was – does Veeam make use of the new features of ReFS if I enable it on an Windows 10 machine (running as Veeam Proxy / Repository) ?
chrmol
Enthusiast
 
Posts: 36
Liked: 2 times
Joined: Mon May 17, 2010 7:41 pm
Location: Denmark
Full Name: Christian Moeller

Re: 2016 ReFS and file allocation size

Veeam Logoby Mike Resseler » Thu Nov 17, 2016 7:15 am

Hi Christian,

No, we can't since Microsoft hasn't enabled that technology on ReFS for Windows 10. What happens when you format a volume in Windows 10 with ReFS is that it acts more like a regular file-system. Our friends over at Starwind Software have written a great blog about it: https://www.starwindsoftware.com/blog/l ... ion-part-1

So for now, Windows 10 ReFS does not support the great functionality that the server version has. But lately we are seeing MSFT being able (through their insider program) to quickly release new functionality (with some child issues they are working on :-)), so who knows in the future... )

Hope it helps
cheers
Mike
Mike Resseler
Veeam Software
 
Posts: 2805
Liked: 343 times
Joined: Fri Feb 08, 2013 3:08 pm
Location: Belgium, the land of the fries, the beer, the chocolate and the diamonds...
Full Name: Mike Resseler

Next

Return to Veeam Backup & Replication



Who is online

Users browsing this forum: Bing [Bot], Google [Bot], Inferno131, jim3cantos, NightBird and 39 guests