ReFS - reclaim space on array?

[HendersonD]

We are using a ReFS volume under Windows Server 2016 as the target for our backup jobs. This is working fine. I know by default Microsoft has chosen to leave trim disabled on ReFS v2 volumes. This means as space is freed up on the ReFS volume, our Nimble storage array will not see these savings. Two questions:

1. In the long run, could this cause issues? Let's say after a year of doing backups to a ReFS volume as VMs to be backed up come and go, could my storage array start complaining that it is short on space when in reality if space could be reclaimed there is no shortage?
2. I did come across this command that I believe enables trim (space reclamation) on ReFS volumes. Has anyone tried using this command? Does Microsoft leave trim disabled by default for some reason like performance? Could I enable trim under Windows Server 2016 during a time when there is little activity, reclaim the space on my array, and then disable trim once again?
   
   fsutil behavior set DisableDeleteNotify ReFS 0

[nmdange]

I would contact Nimble to see if they support using ReFS with TRIM on their arrays. You really want to make sure that your SAN vendor has tested it before turning it on.

[DonZoomik]

Whether Nimble "supports" ReFS is pretty much irrelevant as Nimble provides block storage and it doesn't really care about file system used, it only sees SCSI commands.

According to this it should work: https://docs.microsoft.com/en-us/window ... l-behavior
While there is no specific reason specified, I would enable it, retrim volume (to free any already freed space) and leave it enabled (for runtime reclamation) - if you hit any problems you can disable it without restart. While I've never used Nimble, I haven't heard of any performance problems with it's SCSI UNMAP implementation. You may try it outside working hours, to be on the safe side.

[HendersonD]

Does anyone know why MIcrosoft chooses to leave this off by default under Windows Server 2016?

Is there any way to check on the progress of space reclamation? Just trying to get an idea of how long this will take to reclaim space once I turn it on

[DonZoomik]

When you just turn it on, it only starts reclaiming new deleted space.
To clean up all unused disk space, set DisableDeleteNotify to zero and run either in Command Prompt "defrag C: /L /V" or in PowerShell "Optimize-Volume C -ReTrim -Verbose" (replace volume label with your own).
It usually runs usually within a few minutes even on big filesystems. That reclaims any free space up to the point. If you keep DisableDeleteNotify to zero, Windows will continue reclaiming space at runtime. It's still wise to keep degragmentation maintenance job (disk optimization would be a better name) as occasionally space will not be fully reclaimed at runtime (usually due to block boundary alignment issues - i'm not sure about Nimble, but some other SANs have limitations in this regard) - regular job will clean it up.

[HendersonD]

When I see the command you mention that has defrag in it, I wonder if this is applicable to solid state drives. When I think of defrag I think of moving blocks around to get blocks that belong together contiguous. This is not needed for solid state drives.

[DonZoomik]

If you run "defrag /?", you'll see that it has many options, including "/L" parameter that does ATA TRIM/SCSI UNMAP/NVMe DISCARD based on storage bus.
I'd say that "no defrag" for SSDs is not entirely true. For example
* Heavy file system fragmentation can cause an increase in IOPS as many more commands are required to access data. While SSDs provide much more IOPS, they're still not free resources and may increase application latency.
* NTFS has MFT metadata limitations with heavy fragmentation, see here https://support.microsoft.com/kb/967351 - note that this applies also to Veeam that uses big files (especially with NTFS deduplication)
* Block alignment problems reduce reclamation efficiency. Let's say that your SAN has some internal block size (or SCSI UNMAP granularity) of 1MB (very common, I can't find anything specific about Nimble on Google). If there's a 64k block of data in that 1MB, that space cannot be reclaimed. Even worse, if that 64k data spans two 1MB blocks, neither can be reclaimed. Defragmenting volume can help prevent that - just yesterday I defragmented a 4TB thin VMware VM to reclaim ~1TB of disk space (due to heavy fragmentation, it was consuming nearly all of 4TB even though it had a bit over 1TB of free space).

[HendersonD]

On the Windows server I ran this at the command prompt to enable trim under ReFS
fsutil behavior set DisableDeleteNotify ReFS 0

I then ran this powershell command to actually free up space
Optimize-Volume E -ReTrim -Verbose

Here is the results. The total space trimmed is shown as 3.63 TB. I am not seeing this space savings on the volume when I am logged into this Windows server and I am not seeing it on the volume shown in the Nimble array GUI.

Post Defragmentation Report:
VERBOSE:
Volume Information:
VERBOSE: Volume size = 13.99 TB
VERBOSE: Cluster size = 64 KB
VERBOSE: Used space = 9.47 TB
VERBOSE: Free space = 4.52 TB
VERBOSE:
Retrim:
VERBOSE: Backed allocations = 14335
VERBOSE: Allocations trimmed = 440489
VERBOSE: Total space trimmed = 3.63 TB

I then used this command at a command prompt and got the same results
defrag E: /L /V

Not sure what is happening. Perhaps I am reading this output wrong and there is little space savings on my ReFS volum

[DonZoomik]

I have no experience with Nimble but maybe you have to specially enable reclamation per LUN or per Target? Google tells me that thin provisioned LUNs seem to be the only option so it shouldn't be a matter of fully provisioned LUNs.

[Gostev]

Microsoft explicitly does not support TRIM on ReFS > https://docs.microsoft.com/en-us/window ... s-overview

For SANs, if features such as thin provisioning, TRIM/UNMAP, or Offloaded Data Transfer (ODX) are required, NTFS must be used.

[HendersonD]

In the Nimble OS documentation is this:

When deleting files on a Windows Server 2016 HPE Nimble Storage volume that was formatted with ReFS, the user may find the space is not freed from the array. The reason for this is that Microsoft has chosen to leave trim disabled by default on ReFS v2 volumes. This is detailed in the following Microsoft technet article:
https://technet.microsoft.com/en-us/win ... l-behavior

In the "Parameters" list, review the "DisableDeleteNotify" section for details. For customers using ReFS v2, on Windows Server 2016, deleted files will not be unmapped by default. You need to enable trim by executing:
fsutil behavior set DisableDeleteNotify ReFS 0

-----------------------------------------------------------------------------
Even in Microsoft's documentation of Fsutil, it implies that trim is disbaled by default for Windows Server 2016 but can be enabled
https://docs.microsoft.com/en-us/window ... l-behavior
-----------------------------------------------------------------------------

It appears that in the end there is no space reclamation for ReFS under Windows Server 2016

[BuffaloBucks]

Have you tried using SDelete from sysinternals?

I have run into the same thing with our Western Digital/Tegile array. I have run this and it does clean free space and sends "UNMAP" like commands back to the array. While I was running sdelete, I watch the LUN on the array and kept refreshing and watched the used space on the array drop so it does work.

Do I consider doing this manually great, no, but since it is backups I can live with it. IMO, it is worth it to be able to use ReFS and take advantage of the fast clone with synthetic fulls.

[b.vanhaastrecht]

Yup, Sdelete works with ReFS volume on NFS datastores hosted on Netapp FAS.

Code: Select all

sdelete64 -z <driveletter>

Good find!

[PAF]

I can add that sdelete64 -z also worked for me, unmapping terabytes of free space at several 64 TB ReFS volumes.
The OS is WS2016 ver. 1607, and the storage is a SAS connected Dell Compellent CT-SCv2080 ver. 7.5.2.8.
According to Resource Monitor, the write speed of sdelete64 is 735 MB/s or 2,5 TB/h at an array of 84 x 6TB 7k disks.

I have also added:
fsutil behavior set DisableDeleteNotify ReFS 0

I made a test adding a 100 GB file and deleting it again, but I have no clear indication of that space are actually being unmapped automatically.
Also Optimize-Volmume -ReTrim or defrag /L seems not to actually unmap anything.
However, with NTFS volumes it works as expected.

So a workaround for ReFS volumes seems to be to run sdelete from time to time on a schedule.

[Marvellous]

I'll second that: SDELETE was the only thing to work reliably and demonstrably for us.

Optimise Volume commands spat errors for us, and defrag apparently did nothing, but SDELETE works.

FSUTIL setting didn't seem to have much impact, and we couldn't verify that it was actually doing anything. We eventually ended up upgrading the proxy to Win2022, although in theory, Win2019 should have worked. Still unclear on the behaviour of the auto-UNMAP under Win2022, but SDELETE certainly does work: we dropped from 95% back down to 60%, and now run SDELETE regularly.

[Matt.Sharpe]

Hi Guys,

Just checking on the SDELETE impact. I've tried running on a test drive and it started using the free space on the required drive. It also didn't remove a GB of space on the storage array pool. What was your experience of the space impact on windows side when running it on your REFS filesystem?

[mkretzer]

sdelete creates a dummy file and fills up the storage shortly 100 %, this is normal. The backend array must be able to detect these zero writes, so behavior is dependent on the array. On our systems there is a "zero reclaim feature" we have to actively start.