2016 ReFS and file allocation size

[chrmol]

Quick question - when using ReFS in Windows 2016 for Veeam Repo. - Should it be formatted with 4k or 64k ?
(Lab upgrade in progress )

[dellock6]

Hi,
we had internally the same discussion lat week, and Tom Sightler executed some tests to verify some information. It seems there is a bit of overhead using 64k cluster size, expected as there are chances where a smaller block uses the entire 64k cluster but leaving some space empty. Even with this limit, we feel like 64k could be the suggested size, as also Microsoft seems to suggest this size for all the scenarios where large files are in use (Exchange and SQL databases, Hyper-V virtual machine disks).

[chrmol]

Ok, Ill go with 64K

[nmdange]

I thought ReFS only supported 64k cluster size?

[tsightler]

In versions of WIndows prior to Windows 2016 the 64K cluster size was the only available option for ReFS, however, on Windows 2016 with ReFS 3, you can select either 4K or 64K clusters. I was able to format even a 250TB volume with 4K clusters so I'm not exactly sure when the cutoff is where you have to use 64K in Windows 2016. While I don't know this for sure, my guess is that Microsoft introduced 4K clusters for ReFS to make the filesystem more attractive for generic file server workloads, where the overhead of storing millions of relatively small files in 64K blocks was a huge barrier to acceptance.

In relatively small scale testing (100's of GBs, not TBs) to this point the space difference between 4K and 64K clusters is <10%, although the exact amount will vary based on the selected Veeam block size and compression ratio achieved with the specific data set (the above results were using Veeam defaults and a small mix of Windows and Linux VMs). However, some of this overhead is offset by the space saved from having to track 16x more clusters when using 4K. It's actually fairly easy to estimate the overhead if you know the Veeam block size (which by default is 1MB) and compression ratio.

For now my recommendation for a Veeam repository, as well as our internal consensus, is to use 64K clusters, as it seems to be the safe bet from a scalability and stability perspective and we know it will work well for every size repository. It's hard to imagine that, as the size of the repository grows, the benefit of having to track 16x less clusters won't be an advantage for the larger block sizes. That's not to say that a 4K cluster size repository won't work, but we simply don't have enough data from the field to suggest when it will break, and, in the end, that's the most important question that must be answered before something can be recommended. As we perform additional testing and continue to discuss with Microsoft, as well as collect more results from the field, this recommendation may certainly change.

[nmdange]

If the block size is 1MB wouldn't the backup files always be aligned to 1MB block size (and hence would never have a block of data less than 1mb let alone 64k in size), which would mean no overhead for the Veeam backup files?

[dellock6]

This is true if compression is turned off. If it's on, the original block is 1MB, but it then becomes anything between 1MB (uncompressable block) and any lower value. Say I have after compression a file that is 644k, this consumes 10 * 64k blocks plus a 4k space. So I have 60k unused and unusable in the 11th block.

[nmdange]

Ahhhh that makes sense. Unless you have a huge number of backup files, I wouldn't think the overhead would be that big, and really on a multi-TB volume what's a few 60k here and there

[tsightler]

Ahhhh that makes sense. Unless you have a huge number of backup files, I wouldn't think the overhead would be that big, and really on a multi-TB volume what's a few 60k here and there

True, but remember, the alignment is per-block within the backup file, not per-file, because blocks within the backup file are the elements we want to be able to clone, and clone operations have to be on cluster boundaries.

After compression, each block backed up by Veeam will have somewhere between 0-4K of overhead when stored on ReFS with 4K clusters, and somewhere between 0-64K of overhead when stored on ReFS volume with 64K clusters. Perhaps a simpler way to look at it, 4K clusters will have an average 2K overhead, and 64K cluster will have an average 32K overhead. That's 30K more overhead per source block, because the number of blocks in a given VM is fixed, based on the configured block size.

Simple example: Single VM with 10GB of data
10GB/1MB blocks = 10,000 blocks

10,000 blocks * 2K = 20MB estimated overhead with 4K clusters
10,000 blocks * 32K = 320MB estimated overhead with 64K clusters

Now let's calculate the overhead as a % of total space. Let's assume a somewhat typical 2.5x compression ratio so that means:
10GB/2.5x compression = 4GB VBK

So the backup file with no overhead would be 4GB so the overhead of 4K clusters (20MB) is 0.5% while the overhead with 64K clusters (320MB) is more like 8%. This means the resulting backup files are 0.5% and 8% larger than if it had no alignment, or roughly 7.5% difference between 4K and 64K.

Of course the math changes if the compression ratio changes, say we only get 1.5x compression, so suddenly the backup file is 6.65GB but the overhead per block doesn't change. Since the backup itself is larger, the fixed space of the overhead represents a smaller percentage, only ~5% with 64K clusters, and 0.3% with 4K.

Also, the math changes if you change the block size. If you went from the default of Local target (1MB blocks), to LAN target (512K blocks), suddenly that's 20,000 blocks and, since the alignment overhead is per-block, that doubles my overhead with 64K clusters to 640MB. The backup size is likely to be the same, so that 8% overhead just became 16%!

Once again, some of that overhead is made up for my having smaller allocation tables and I believe the proper balance is default settings for Veeam block size and 64K clusters, but it's interesting to understand the impact of it all.

[nmdange]

And that is why I love these forums

[chrmol]

In my lab I use a windows 10 machine as Veeam repository (for copy jobs).
... But does ReFS in Windows 10 provide the same functionality as Windows 2016 ?

[veremin]

... But does ReFS in Windows 10 provide the same functionality as Windows 2016 ?

No, it does not.

[vClintWyckoff]

So you have manually enabled to permit the formatting of ReFS on Win 10?

Just run...

Code: Select all

C:\WINDOWS\system32>fsutil fsinfo volumeinfo <DRIVE>

...against both ReFS volumes and you'll see which functions each volume supports.

[chrmol]

Actually what I meant was – does Veeam make use of the new features of ReFS if I enable it on an Windows 10 machine (running as Veeam Proxy / Repository) ?

[Mike Resseler]

Hi Christian,

No, we can't since Microsoft hasn't enabled that technology on ReFS for Windows 10. What happens when you format a volume in Windows 10 with ReFS is that it acts more like a regular file-system. Our friends over at Starwind Software have written a great blog about it: https://www.starwindsoftware.com/blog/l ... ion-part-1

So for now, Windows 10 ReFS does not support the great functionality that the server version has. But lately we are seeing MSFT being able (through their insider program) to quickly release new functionality (with some child issues they are working on ), so who knows in the future... )

Hope it helps
cheers
Mike

[chrmol]

Thanks Mike.

[ChrisGundry]

Hi All,

We have a new storage appliance and we have setup 3 new ReFS volumes for our backup repos. We have started backups to these new volumes and we are seeing that the space usage on disk does not seems to be as expected. For example, on one of the volumes:
If I select all files on the disk and select properties I get 871GB.
If I right click the volume and check used space I get 975GB.
Shadowcopies is disabled and I have selected all hidden files and system folders when checking properties. There is nothing in 'system volume info' folder either.

The ReFS is formatted with 4k cluster size as that is what was recommended to me previously, but I see current recommendation now seems to be 64k which is slightly annoying as the volumes are now setup.

We are using Veeam 9.5 and the Veeam server and Repo are the same server which is running Server 2016.

We have the same issue with all 3 ReFS volumes. I need to move our backups over to this new storage ASAP but I am very concerned that this space issue is getting worse the more data I put on the volume.

The repo setting to align blocks is selected for some reason, could this have anything to do with it?

Can anyone advise why you think this might be happening?

Thanks!

[PTide]

Hi,

It's not quite clear where you get that info from, could you provide a couple of screenshots for clarity sake please?

Thanks

[ChrisGundry]

I have just re-created one of the volumes with 64k size and re-created the repo without aligning blocks and started a job, which isn't finished yet.

Files are showing size of 45.7GB
Disk is showing 286GB used!!

[ChrisGundry]

Screenshot here:
http://imgur.com/dX2PXkd

[ChrisGundry]

My thinking being that the size of the files should match the size of the used space. The ReFS and spaceless full backups should actually mean that the space used at the disk level shows less space used than the size of the files shows.

But as we don't have any spaceless full backups yet, the space should just match and be the same, within a small margin to account for the 64k block size and a file that is less than 64k taking up a full 64k. But there is very little data on this volume at present.

Currently it now shows 79GB used in file properties and the disk properties shows 310GB used!
Treesize also only shows 79GB being used.
Treesize also shows 42.6TB free of 42.9TB so again 300GB usage, but not in the Veeam files...??

[ChrisGundry]

I have now stopped the Veeam job and deleted the files but it still shows as 234GB in use. Down from 330GB when I stopped the job...

The difference between 330GB and 234GB is the size of the backup file.

I have now re-formatted the ReFS volume fresh and it imediatly shows as 234GB in use...

Is this some ReFS data taking up a % of the overall size. This volume is a fairly large 43TB, so its about 0.55% of the total being used up.

I re-formatted as NTFS and it shows the full 42.9TB available with no phantom space being used up...

E: drive volume is 8.69TB in size. It is showing 152GB used on the disk but only 73.1GB used in files. 152GB minus 73.1GB gives us 78.9GB. If we work out 78.9GB as a % of the total capacity we get 0.88% which is close to the 0.55% above.

F: drive volume is 15.9TB in size. It is showing 1034.24GB used on the disk but only 936GB used in files. 1034GB minus 936GB gives us 98GB. If we work out 98GB as a % of the total capacity we get 0.64% which is pretty close to the 0.55% above.

Does that make any sense to anyone??

[PTide]

Please check this post for a detailed explanation in regards to ReFS overhead.

Thanks

[tsightler]

My thinking being that the size of the files should match the size of the used space. The ReFS and spaceless full backups should actually mean that the space used at the disk level shows less space used than the size of the files shows.

But as we don't have any spaceless full backups yet, the space should just match and be the same, within a small margin to account for the 64k block size and a file that is less than 64k taking up a full 64k. But there is very little data on this volume at present.

Currently it now shows 79GB used in file properties and the disk properties shows 310GB used!
Treesize also only shows 79GB being used.
Treesize also shows 42.6TB free of 42.9TB so again 300GB usage, but not in the Veeam files...??

Did you check the volume immediately after you formatted it? ReFS sets aside a significant amount of space during format, by my observation around one-half of 1% of the size of the volume as reserved space, but it doesn't seem to be completely linear so that's just an estimate. With a 47TB disk that would be almost 250GB shown as used immediately after formatting. 4K clusters set aside more space than 64K clusters, the gap widens as the volume size goes up.

[ChrisGundry]

I don't know why PTide merged my post with this thread as it doesn't seem to reference anything about the issue I am seeing.

Hi tsightler,

Thanks for your reply which confirms what I have been seeing. Yes, I checked straight after formatting. It's unusual to see it as 'used' space though, rather than usually I would expect to see the overall capacity being lower after formatting, rather than it showing up as full capacity but with some 'used' space.

It seems like I am seeing ~0.5% overhead on 64k and between 0.5 and 1% on 4k block sizes.

Do you know of any documentation that explains what this is and why it happens? I can't seem to see anything!

Dmitriy ahs also just emailed me to say he just replicated the issue in a test environment and confirmed the same ~0.53% lost space... Odd.

Thanks

[tsightler]

It's unusual for a Windows filesystem, it's not that unusual for filesystems on other operating systems.