File being used by another process on ReFS - defragsvc related

[willrussell]

Hi,

Support case ID - 03202268

We're looking after about a dozen VBR installations (all on completely different networks), and we've noticed a common theme with many of the backups. It seems that intermittently (about once every one/two weeks), each backup fails due to the error ""Error: The process cannot access the file because it is being used by another process.". Examining the logs confirms that it's the metadata file that is the one that is locked out. Usually the backup runs the next time without issue. The backup failing is random, so across all the backups we look after, we see it on most days.

The common thing with all of the backups that fail in this way is that they use ReFS repositories. All use 64k blocksize. Some are via iSCSI, some are DAS. None are using storage spaces.
As per most servers we look after, the VBR servers are all running Server 2016 and back up Hyper-V hosts. We also run Veeam Availability Console agents on all servers.

I configured Process Monitor on one of the servers (a server which sees the issue every week) and found that 'defragsvc' had a handle on the file while Veeam was running a backup. This backup failed with the above mentioned error. This was odd, as the use of defragsvc at that time (approx 1pm on a Friday) didn't tally with the 'Defrag' scheduled task that does drive maintenance/optimisation (and I can confirm this by looking at the 'History' of the scheduled task). To be sure though, on that server I stopped & disabled the defragsvc (AKA 'Optimize drives') service and left it a month. The error message didn't come back within that time. This tells me that defragsvc is regularly interfering with the ReFS repositories and causing a lockout of files during backup.

At the request of Veeam support, I've now re-enabled defragsvc on that server and have disabled it on another VBR server we look after (just to confirm the problem consistently is related to defragsvc).

While I'm waiting for the results of that, I just wondered if anyone else had any ideas? Why is defragsvc doing stuff to ReFS repositories so frequently, and outside of the normal scheduled task hours?

Thanks for any help!

[Mgamerz]

I thought defrag on ReFS with block cloning should be off? I remember reading about defrag in some old veeam threads. Cause you can't really defrag a block that is referenced multiple times since the data can't be in two places at once?

(Forgive my terminology here, I don't know the exact terms)

[foggy]

You're most likely refer to this explanation.

[willrussell]

Thanks for the link. Both of your comments confirm that I don't really want to defrag a ReFS volume. That's fine by me.

Is disabling defragsvc the right way to go about this though? Surely doing that also prevents defragging of all NTFS volumes too (e.g. OS drive).

Is there a way of preventing defrag operations on ReFS only?

[willrussell]

Pretty sure we've found the culprit.

Our RMM software (Connectwise Automate, AKA LabTech) is running a 'disk inventory' every 3 hours. In doing this, it also seems to be running a defrag analyse job. On NTFS volumes of any size, this takes a few seconds. On ReFS 64k volumes it takes minutes or even hours (depending on both disk speed and volume size). I can replicate this behaviour with manual defrag analyse commands in PowerShell too. We'll get the config of the RMM software changed.

Still not sure why a simple defrag analysis takes ages on ReFS, but as has already been pointed out, this isn't necessarily something we want to do anyway.

Thanks all.

[Gostev]

Great information, thanks Will. Obviously this is not something we would be able to understand from our end, so thanks for sharing!