Very Slow Fast clone Windows Server 2019

[wageck]

We are running Windows Server 2019 LSC with Veeam V10 and are seeing extremely slow Fast Clone processes.

Case #04788020 is open but the support engineer can't help and refers us to Microsoft Support.....

The following is configured / deployed

Windows Server 2019, Version 1809 OS Build 17763.1817
Clean Install (only Dell Drivers & Tools) on Dell R740 with 2 x Intel Xeon Gold 6244 @ 3,60GHz (32 Logical Cores), 512GB Memory
refs.sys File version: 10.0.17763.1728 / date modified 3/16/2021 5:06 PM
Local Storage Pool with 458 TB Capacity (Free Space 1,92TB = 1 SSD Spare), Splitted in 4 volumes like
Volume Capacity = 56,9TB (80GB WriteCacheSize, 800GB SSD, 56,1TB HDD)
Allocated = 56,9TB
Used pool Space: 114TB


REFS Info:
fsutil.exe fsInfo refsInfo E:
REFS Volume Serial Number : 0xfc3ea1213ea0d5c4
REFS Version : 3.4
Number Sectors : 0x000000038e79c000
Total Clusters : 0x0000000038e79c00
Free Clusters : 0x000000000086810f
Total Reserved : 0x0000000000108c56
Bytes Per Sector : 4096
Bytes Per Physical Sector : 4096
Bytes Per Cluster : 65536
Checksum Type: CHECKSUM_TYPE_NONE


With Following (Registry) settings:
fsutil behavior set DisableDeleteNotify ReFS 1
Get-Volume |?{$_.FileSystemType -Like 'ReFS' } | Set-FileIntegrity -Enable $true "$_.DriveLetter"+":"

HKLM\SYSTEM\CurrentControlSet\Services\Disk | TimeOutValue = 120
HKLM\SYSTEM\CurrentControlSet\Control\FileSystem |
ReFS DisableDeleteNotify = 1
RefsDisableCachedPins = 1
RefsDisableDeleteNotification = 1
ReFSDisableDeleteNotify = 1
RefsDisableLastAccessUpdate = 1
RefsEnableInlineTrim = 1
RefsEnableLargeWorkingSetTrim = 1
RefsNumberOfChunksToTrim = 128
RefsProcessedDeleteQueueEntryCountThreshold = 2048
HKLM\SOFTWARE\Veeam\Veeam Backup and Replication |
ReFSDedupeBlockClone = 1
DisableRefsIntegrityStreams = 0
SOBRSyntheticFullCompressRate = 3
SOBRIncrementCompressRate = 5
SOBRFullCompressRate = 25
SobrForceExtentSpaceUpdate = 1


I run the Tool from http://www.toughdev.com/content/2015/05 ... -metafile/ to cleanup the refs meta memory but my Memory Usage is still the same!

The first test is with no realy Backup Load on the system and the next is with full backup load !

befor memory cleaning wihout load onthe system:
C:\Windows\system32>"C:\scripts\block-clone-spd.exe" E:\temp\ 20
block-clone-spd utility, v0.3.3. Vsevolod Zubarev 2018-2020.
refs.sys driver version: 10.0.17763.1728
Volume file system is ReFS.
Block cloning is available.
Cluster size is 65536 bytes.
Free space available: 4110.743 GiB.
Will create three files 20 GiB each, for a total of up to 60 GiB.
Writing random file "E:\temp\01.data"...
████████████████████████████████████████████████████████████████████████████████████████████████████████████ 20480/20480Writing random file "E:\temp\02.data"...
████████████████████████████████████████████████████████████████████████████████████████████████████████████ 20480/20480Writing new file "E:\temp\cloned.data" via block cloning...
All block cloning took 178.193s.
Average speed: 114.932 MiB/s

after memory cleaning and with load on the system:
C:\Windows\system32>"C:\scripts\block-clone-spd.exe" E:\temp\ 20
block-clone-spd utility, v0.3.3. Vsevolod Zubarev 2018-2020.
refs.sys driver version: 10.0.17763.1728
Volume file system is ReFS.
Block cloning is available.
Cluster size is 65536 bytes.
Free space available: 1852.736 GiB.
Will create three files 20 GiB each, for a total of up to 60 GiB.
Writing random file "E:\temp\01.data"...
████████████████████████████████████████████████████████████████████████████████████████████████████████████ 20480/20480Writing random file "E:\temp\02.data"...
████████████████████████████████████████████████████████████████████████████████████████████████████████████ 20480/20480Writing new file "E:\temp\cloned.data" via block cloning...
All block cloning took 19.982s.
Average speed: 1024.934 MiB/s

What do we miss?

[wageck]

Microsoft Support Case 2105180050002889 is open

[Gostev]

Make sure you don't have overlapping tape out jobs accessing backup files involved in the fast cloning... that's the only known issue we're aware of with V10.

[wageck]

That is realy a hint, but i dont get why block-clone-spd.exe is then slow