ReFS and slow file copy speed

[b.vanhaastrecht]

Hello,

We use SOBR with 4 extends, all are 10TB in size and are formatted ReFS 64k on one single Windows 2016 box. Those extends are backed by seperate VMWare datastores on NetApp NFS volumes.

We recently needed to manually move/copy some .vib and .vbs files cross volume/extend. The file copy was initiated within the 2016 box from volume to volume. We notice the copy speed was not going faster than about 50MB/s. We started to investigate as we would expect much higher speeds.

We noticed no difference in .vbs or .vib file, but the file age does make a difference. Files older than one day were stuck at 50MB/s and fresh files were going 200MB/s. The backed NetApp does use post compression and deduplication, but with tests on a seperate machine the deduped and compressed files were actually faster in copy then non toughed files, so it ruled out the NetApp. There's no dedupe enabled in Windows, so we can't find a cause why +1 day files copy only at 50MB/s.

Anyone has any ideas about this?

Regards,
Bastiaan

[b.vanhaastrecht]

I would like to add: When copying non-veeam files the speed is always as expected: 200MB/s. So only Veeam .vbs and .vib files are affected.

[Gostev]

Fragmentation due to block cloning?

[b.vanhaastrecht]

Makes sense with .vbk's, but we are seeing the same with .vib's. Those arent formed via block cloning, right?

[Gostev]

You are correct, however future VBKs will reference blocks brought by earlier VIBs, which may in turn affect VIB blocks metadata and thus slow down VIBs processing. ReFS architecture is not a public knowledge, so I am just theorizing - obviously, this question is better to be addressed to Microsoft Support.

[pkuikman]

Hello,

We use SOBR with 4 extends, all are 10TB in size and are formatted ReFS 64k on one single Windows 2016 box. Those extends are backed by seperate VMWare datastores on NetApp NFS volumes.

We recently needed to manually move/copy some .vib and .vbs files cross volume/extend. The file copy was initiated within the 2016 box from volume to volume. We notice the copy speed was not going faster than about 50MB/s. We started to investigate as we would expect much higher speeds.

We noticed no difference in .vbs or .vib file, but the file age does make a difference. Files older than one day were stuck at 50MB/s and fresh files were going 200MB/s. The backed NetApp does use post compression and deduplication, but with tests on a seperate machine the deduped and compressed files were actually faster in copy then non toughed files, so it ruled out the NetApp. There's no dedupe enabled in Windows, so we can't find a cause why +1 day files copy only at 50MB/s.

Anyone has any ideas about this?

Regards,
Bastiaan

It's an old topic, but did you ever figure out how to speed up the copy/paste action?

[b.vanhaastrecht]

Nope, just a workaround. Use robocopy with /MT:16 or higher to utilize multiple threads while copying files.

[squebel]

Running into the same issue although seen this before. Also seeing heavy latency during the write operation on the destination volume. Seems entirely related to the structure of the vbk and the read/write operation of the vbk on ReFS. When running load testing on the same exact disk and simulating write speeds, it's incredibly fast and low latency. Not sure what weirdness is happening but it's strange indeed.