Windows 2019, large REFS and deletes

[mkretzer]

I wonder if this is the same issue we are facing with 2004 SAC... Sadly, Microsoft found no real issue in their first analysis. Did you already talk to MS support?

[Gostev]

We have 2 Veeam v11 repositories, both Windows Server 2019 v1809 and the behavior is totally different.
One with refs.sys 10.0.17763.1971 : work great
One with refs.sys 10.0.17763.2090 : catastrophic behavior. Copy jobs worked great, but CPU at 100% when merge (fast clone). Even with 1 single job...
Server is totally unresponsive after a couple of hours

Are these exact same systems in terms of hardware and software, especially in terms of antivirus and its settings (Veeam backup files excluded from real-time monitoring on both)? I would take a closer look there first, as from what I am hearing from Microsoft there were no changes in the block cloning code path between 1971 and 2090. Only one small change elsewhere around new hardware support, which I cannot see how it can possibly affect block cloning performance.

[mkretzer]

Also, did you check that DisableDeleteNotify is still enabled? In our system we found that it got disabled after an update which also lead to strange performance issues after deletes...
-> "fsutil behavior query DisableDeleteNotify"

[adrenaline_x]

What is the correct setting for fsutil behavior query DisableDeleteNotify

Mine is showing 0 and i noticed that after the last update to the server the latest merge of the oldest incremental (forever incremental) is taking close to 20 hours where is finished within hours before.

Nothing else about the system has changed either.

[Gostev]

0 for disabled means enabled so you don't have this set correct. Disable it by executing "fsutil behavior set DisableDeleteNotify ReFS 1" command.

As for a longer merge, did you compare incremental backup sizes? Because seeing a few times bigger than usual increment is not something abnormal after a Windows or software update for example. So if a few hours merge was "normal" in your environment, then 3-5x increase to 20 hours is not necessarily an indication of an issue yet.

[mkretzer]

Gostev,

you mean "fsutil behavior set disabledeletenotify ReFS 1" - otherwise its done for NTFS by default! And you don't want to do this if your local system drive is NTFS on SSD!

Markus

[Gostev]

Yes, of course I only wanted to say is that the command has to set 1 as the value. Corrected the previous post.

[karsten123]

Did anyone test Server 2022 allready?

[mkretzer]

veeam-backup-replication-f2/refs-server ... 76161.html

At least Veeam did do no tests yet...

[AlexandreD]

Hi everyone,

I have a problem with deleting large files on a ReFS Volume.
Backup Copy are not available on Veeam Cloud Connect Server.
So I have to backup all data on a repository with a Veeam Agent for Windows (we don't have S3 storage for doing SOBR capacity tier/offload).

I have one R740XD2 for the cloud repository, behind cloud connect infrastructure. (3 volumes on it, 3x53 TB, for VSS Snapshot Support).
This is a Windows Repository on Windows Server Standard 2019 1809 version 17763.1697
On this R740XD2, I have install a Veeam Agent Windows, which backup data everyday on a other XD2 in an another Datacenter.

Full backup are very long (because there are 100TB used) but incremental backup are OK in a 24h RPO.

Few days ago, for some reason, i did a new job, with a new full. I deleted old job.
Delete job in veeam agent doesn't deleted backup chain, so i deleted manually the chain in the destination repository.
I didn't get back free space. ReFS doesn't reclaim space.

What can I do ? Do you think I can open a veeam case for this ?
I found some tips, doing a defrag, sdelete64, chkdsk, what is the best way for get back free space ?

Thank you for your help.

Alexandre

[Gostev]

Assuming you waited sufficient time for ReFS to release disk space, as I believe it is a background process, you should open a case with Microsoft. Since you manually deleted a bunch of files from ReFS and did not get back free space, this has nothing to deal with Veeam at this point. I will however say, you're the first to report something like this in all these years. So make sure you're not mistaken somewhere in your observations.

[AlexandreD]

Hi Gostev,

I did this suppression two days ago. Today i have no free space yet. I'm one hundred percent sure with my observations, i deleted several big files (one vbk 60TB, and some vib 5TB) in the windows files explorer. Files are gone, nothing in the recycle bin, because files are too big, and no free space on my ReFS volume.

I searched for a solution all day yesterday. I don't find anything about reclaim spaces on a ReFS volume, commandline, a background process or anything.

Unfortunately, we are a service provider with SPLA agreement, and no access to any Microsoft support.

I'll probably have to format the volume, lose my new full backup, and start over.

[Seve CH]

Hi Alexandre
We have a W2019 Veeam server with 300+TB Repo. I've recently removed 60TB of backups and Windows took one hour or so to free it up. I've done that before (W2016 and W2019) without any problem.

If the free space is not coming back, maybe you have a filesystem problem. If I recall correctly, ReFS makes bad files invisible (you cannot access them, you cannot delete them, clever...). You might have had a problem with your disks or something like that, and ReFS has just hidden your data. Check your event log on that time.
"Salvaging data - If a volume becomes corrupted and an alternate copy of the corrupted data doesn't exist, ReFS removes the corrupt data from the namespace"
https://docs.microsoft.com/en-us/window ... s-overview
Regards