Comprehensive data protection for all workloads
Post Reply
HendersonD
Expert
Posts: 158
Liked: 8 times
Joined: Jul 23, 2011 12:35 am

ReFS - reclaim space on array?

Post by HendersonD »

We are using a ReFS volume under Windows Server 2016 as the target for our backup jobs. This is working fine. I know by default Microsoft has chosen to leave trim disabled on ReFS v2 volumes. This means as space is freed up on the ReFS volume, our Nimble storage array will not see these savings. Two questions:
  1. In the long run, could this cause issues? Let's say after a year of doing backups to a ReFS volume as VMs to be backed up come and go, could my storage array start complaining that it is short on space when in reality if space could be reclaimed there is no shortage?
  2. I did come across this command that I believe enables trim (space reclamation) on ReFS volumes. Has anyone tried using this command? Does Microsoft leave trim disabled by default for some reason like performance? Could I enable trim under Windows Server 2016 during a time when there is little activity, reclaim the space on my array, and then disable trim once again?

    fsutil behavior set DisableDeleteNotify ReFS 0

nmdange
Expert
Posts: 503
Liked: 130 times
Joined: Aug 20, 2015 9:30 pm
Contact:

Re: ReFS - reclaim space on array?

Post by nmdange »

I would contact Nimble to see if they support using ReFS with TRIM on their arrays. You really want to make sure that your SAN vendor has tested it before turning it on.

DonZoomik
Veeam Legend
Posts: 270
Liked: 72 times
Joined: Nov 25, 2016 1:56 pm
Contact:

Re: ReFS - reclaim space on array?

Post by DonZoomik »

Whether Nimble "supports" ReFS is pretty much irrelevant as Nimble provides block storage and it doesn't really care about file system used, it only sees SCSI commands.

According to this it should work: https://docs.microsoft.com/en-us/window ... l-behavior
While there is no specific reason specified, I would enable it, retrim volume (to free any already freed space) and leave it enabled (for runtime reclamation) - if you hit any problems you can disable it without restart. While I've never used Nimble, I haven't heard of any performance problems with it's SCSI UNMAP implementation. You may try it outside working hours, to be on the safe side.

HendersonD
Expert
Posts: 158
Liked: 8 times
Joined: Jul 23, 2011 12:35 am

Re: ReFS - reclaim space on array?

Post by HendersonD »

Does anyone know why MIcrosoft chooses to leave this off by default under Windows Server 2016?

Is there any way to check on the progress of space reclamation? Just trying to get an idea of how long this will take to reclaim space once I turn it on

DonZoomik
Veeam Legend
Posts: 270
Liked: 72 times
Joined: Nov 25, 2016 1:56 pm
Contact:

Re: ReFS - reclaim space on array?

Post by DonZoomik »

When you just turn it on, it only starts reclaiming new deleted space.
To clean up all unused disk space, set DisableDeleteNotify to zero and run either in Command Prompt "defrag C: /L /V" or in PowerShell "Optimize-Volume C -ReTrim -Verbose" (replace volume label with your own).
It usually runs usually within a few minutes even on big filesystems. That reclaims any free space up to the point. If you keep DisableDeleteNotify to zero, Windows will continue reclaiming space at runtime. It's still wise to keep degragmentation maintenance job (disk optimization would be a better name) as occasionally space will not be fully reclaimed at runtime (usually due to block boundary alignment issues - i'm not sure about Nimble, but some other SANs have limitations in this regard) - regular job will clean it up.

HendersonD
Expert
Posts: 158
Liked: 8 times
Joined: Jul 23, 2011 12:35 am

Re: ReFS - reclaim space on array?

Post by HendersonD »

When I see the command you mention that has defrag in it, I wonder if this is applicable to solid state drives. When I think of defrag I think of moving blocks around to get blocks that belong together contiguous. This is not needed for solid state drives.

DonZoomik
Veeam Legend
Posts: 270
Liked: 72 times
Joined: Nov 25, 2016 1:56 pm
Contact:

Re: ReFS - reclaim space on array?

Post by DonZoomik »

If you run "defrag /?", you'll see that it has many options, including "/L" parameter that does ATA TRIM/SCSI UNMAP/NVMe DISCARD based on storage bus.
I'd say that "no defrag" for SSDs is not entirely true. For example
* Heavy file system fragmentation can cause an increase in IOPS as many more commands are required to access data. While SSDs provide much more IOPS, they're still not free resources and may increase application latency.
* NTFS has MFT metadata limitations with heavy fragmentation, see here https://support.microsoft.com/kb/967351 - note that this applies also to Veeam that uses big files (especially with NTFS deduplication)
* Block alignment problems reduce reclamation efficiency. Let's say that your SAN has some internal block size (or SCSI UNMAP granularity) of 1MB (very common, I can't find anything specific about Nimble on Google). If there's a 64k block of data in that 1MB, that space cannot be reclaimed. Even worse, if that 64k data spans two 1MB blocks, neither can be reclaimed. Defragmenting volume can help prevent that - just yesterday I defragmented a 4TB thin VMware VM to reclaim ~1TB of disk space (due to heavy fragmentation, it was consuming nearly all of 4TB even though it had a bit over 1TB of free space).

HendersonD
Expert
Posts: 158
Liked: 8 times
Joined: Jul 23, 2011 12:35 am

Re: ReFS - reclaim space on array?

Post by HendersonD »

On the Windows server I ran this at the command prompt to enable trim under ReFS
fsutil behavior set DisableDeleteNotify ReFS 0

I then ran this powershell command to actually free up space
Optimize-Volume E -ReTrim -Verbose

Here is the results. The total space trimmed is shown as 3.63 TB. I am not seeing this space savings on the volume when I am logged into this Windows server and I am not seeing it on the volume shown in the Nimble array GUI.

Post Defragmentation Report:
VERBOSE:
Volume Information:
VERBOSE: Volume size = 13.99 TB
VERBOSE: Cluster size = 64 KB
VERBOSE: Used space = 9.47 TB
VERBOSE: Free space = 4.52 TB
VERBOSE:
Retrim:
VERBOSE: Backed allocations = 14335
VERBOSE: Allocations trimmed = 440489
VERBOSE: Total space trimmed = 3.63 TB

I then used this command at a command prompt and got the same results
defrag E: /L /V

Not sure what is happening. Perhaps I am reading this output wrong and there is little space savings on my ReFS volum

DonZoomik
Veeam Legend
Posts: 270
Liked: 72 times
Joined: Nov 25, 2016 1:56 pm
Contact:

Re: ReFS - reclaim space on array?

Post by DonZoomik »

I have no experience with Nimble but maybe you have to specially enable reclamation per LUN or per Target? Google tells me that thin provisioned LUNs seem to be the only option so it shouldn't be a matter of fully provisioned LUNs.

Gostev
SVP, Product Management
Posts: 28524
Liked: 5135 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: ReFS - reclaim space on array?

Post by Gostev »

Microsoft explicitly does not support TRIM on ReFS > https://docs.microsoft.com/en-us/window ... s-overview
For SANs, if features such as thin provisioning, TRIM/UNMAP, or Offloaded Data Transfer (ODX) are required, NTFS must be used.

HendersonD
Expert
Posts: 158
Liked: 8 times
Joined: Jul 23, 2011 12:35 am

Re: ReFS - reclaim space on array?

Post by HendersonD »

In the Nimble OS documentation is this:

When deleting files on a Windows Server 2016 HPE Nimble Storage volume that was formatted with ReFS, the user may find the space is not freed from the array. The reason for this is that Microsoft has chosen to leave trim disabled by default on ReFS v2 volumes. This is detailed in the following Microsoft technet article:
https://technet.microsoft.com/en-us/win ... l-behavior

In the "Parameters" list, review the "DisableDeleteNotify" section for details. For customers using ReFS v2, on Windows Server 2016, deleted files will not be unmapped by default. You need to enable trim by executing:
fsutil behavior set DisableDeleteNotify ReFS 0

-----------------------------------------------------------------------------
Even in Microsoft's documentation of Fsutil, it implies that trim is disbaled by default for Windows Server 2016 but can be enabled
https://docs.microsoft.com/en-us/window ... l-behavior
-----------------------------------------------------------------------------

It appears that in the end there is no space reclamation for ReFS under Windows Server 2016

afoegen@winona.edu
Influencer
Posts: 17
Liked: 1 time
Joined: Oct 02, 2019 5:52 pm
Full Name: Al
Location: Minnesota
Contact:

Re: ReFS - reclaim space on array?

Post by afoegen@winona.edu »

Have you tried using SDelete from sysinternals?

I have run into the same thing with our Western Digital/Tegile array. I have run this and it does clean free space and sends "UNMAP" like commands back to the array. While I was running sdelete, I watch the LUN on the array and kept refreshing and watched the used space on the array drop so it does work.

Do I consider doing this manually great, no, but since it is backups I can live with it. IMO, it is worth it to be able to use ReFS and take advantage of the fast clone with synthetic fulls.

b.vanhaastrecht
Veeam Legend
Posts: 688
Liked: 122 times
Joined: Aug 26, 2013 7:46 am
Full Name: Bastiaan van Haastrecht
Location: The Netherlands
Contact:

Re: ReFS - reclaim space on array?

Post by b.vanhaastrecht »

Yup, Sdelete works with ReFS volume on NFS datastores hosted on Netapp FAS.

Code: Select all

sdelete64 -z <driveletter>
Good find!
========================================
Veeam ProPartner and Cloud Connect Provider

Post Reply

Who is online

Users browsing this forum: No registered users and 58 guests