1PB Lun thinprovisioned repository PureFlashArray REFS Problem

[alexandre.nakagawa]

HI,
i'm strugling with a situation here.

a few weeks ago we finished the setup of a phisycal windows machines (server 2022) (dell) with HBA connected to a SAN and PureStorage C60 FlashArray. (500TB Raw. 1.5PB promised / provisioned usage by pure vendor)

this proxy/repository server is dedicated to backup a SQL Database of 200TB of data (this is why we have a 1PB lun presented to it).

the repository is windows based with REFS formated volume, in 64KB block (no windows dedup enabled).
repository is configured to "decompress before write" as per purestorage recomendation, because pure storage delivery a thin provisioned lun and have dedup capabilities.
repository is SOBR with offload to azure (we keep only last chain for quick recovery from local storage, everthing else is offloades to azure after 7 days)

In the initial setup and days was good. but the problem started after some backup rotations, when the data got offloaded and files are deleted, the windows refs volume slowly increase free space until windows say the 1PB volume has ony 50TB used (with 50TB files from veeam) as expected.

but from purestorage perspective, the LUN is using almost 250TB of unique /raw, non deduplicated data. and every new backup chain the lun increase the size.

doing some research I Found that REFS volumes does not support trim / unmap commands (only to storage spaces), and maybe this is afecting all dedup capabilites of purestorage and space reclamation.

Anyone has a similar scenario?

Do I need to enable "dedup friendly" or other default dedup from veeam instead of decompress before write?
Is better to replace REFS with NTFS and live without synthetic "block clone" but have unmap / trim / and odx offload to storage?

Any other alternatives?

My pure C60 is nearly 0 dedup ratio (the production ones on esxi has about 9:1 dedup ratio in VMFS datastores), I think refs is fuc****g the array dedup and space reclamation.

[mkretzer]

Hello,

for such big LUNs i am always a fan of Ubuntu+XFS including block cloning and then enabling trim. We also have a nearly 1 PB repo and never had and issue with this.

But in theory, unmap should work with ReFS. In the past our rotating disk arrays were slightly overloaded because trim was working on our ReFS repo. What does "fsutil behavior query disabledeletenotify" show?
For many arrays trim support must be announced to the OS but i guess this is default for Pure.

What happens if you manually use sdelete or optimize-volume (https://support.purestorage.com/Solutio ... eclamation)?

Markus

[alexandre.nakagawa]

C:\Windows\system32>fsutil behavior query disabledeletenotify"
NTFS DisableDeleteNotify = 0 (Allows TRIM operations to be sent to the storage device)
ReFS DisableDeleteNotify = 0 (Allows TRIM operations to be sent to the storage device)

when I try do use optimize-volume it says that trim can't be used with slabs below 8mb
Optimize-Volume : Neither slab consolidation nor slab analysis can run if slabs are less than 8 MB.
Activity ID: {1f31e157-561e-4f41-902b-6a1de5944248}

[mkretzer]

Interesting, so unmap is enabled in windows. I found one forum suggesting the message you get means windows does not detect the volume as thin provisioned.

What about "Get-WMIObject -Class MSFT_Disk -Namespace 'ROOT\Microsoft\Windows\Storage' | Select-Object ProvisioningType,UniqueId,Number"? Type 1 should be thin, type 2 is fixed.

[alexandre.nakagawa]

all except S.O is thin provisioned.

unmap is enabled, but per MS documentation it only works on storagespaces.

I deleted the 1PB volume, detroyed on purestorage, recreate and now doing some backups, let's see if after the backups are finished and deleted the purestorage will make the reclain.




Provisioningtype number uniqueid
---------------- ------ --------
1 5 624A9370A9D0DF5E15C74FDF000DA01B
1 1 624A9370AC5C1753D75C4885000113F7
1 3 624A9370AC5C1753D75C4885000113F9
1 4 624A9370AC5C1753D75C48850001293E
1 2 624A9370AC5C1753D75C488500012940
1 6 624A9370AC5C1753D75C488500012944
1 7 624A9370AC5C1753D75C488500012946
1 8 624A9370AC5C1753D75C48850001294A
2 0 614187702731BB00294274039FB21B49