XFS and Fast Clone (reflink) very slow on merging?

[ldunlop]

We have been struggling to understand why our XFS fast clone (reflink) merges are incredibly slow. The merges are for client backups on a Linux Repository that come in via a Cloud Connect gateway. We notice that there is a lot of chatter between the Linux Repo and the Cloud Connect Gateway (which in itself is odd, one would have thought that the CC gateway should just issue a merge command to the Linux Repo and let it do its thing)

On this tenant's 2TB vbk file it takes a bit over 18 HOURS to complete the "fast" clone at a rate of 31.4 MBps.

This is the problem: it's taking way too long to do the fast clone.

Multiple fast clones at the same time for different tenants but to the same underlying filesystem also increase the size of their destination files at a rate of 31.4 MBps EACH. This shows the filesystem itself is perfectly capable of supporting a much faster fast clone rate.

In fact, doing a "cp --reflink=always xxx.vbk test" on the same filesystem completes in just over 8 MINUTES for a 2TB file, a rate of 4.2 GBps (the actual rate varies during the copy e.g. between 3 and 6 GBps).

I.e. the reflink copy is better than 130 TIMES FASTER than the fast clone.

We understand the fast clone is a little more involved because it's interweaving blocks from multiple files into the destination file, but that should be a trivial amount of overhead.

Has anybody had a similar experience or have some suggestions for what we might be doing wrong?

[Andreas Neufert]

Please provide a support case ID for this issue, as requested when you click New Topic.

[Gostev]

Yeah... first and foremost, you should work with our support to create the special performance debug log to understand what specific I/O operations that take a long time. Don't just assume it's block cloning that is slow. We were recently investigating a similar issue with perceived poor ReFS block cloning performance, and these debug stats showed it had nothing to deal with the actual block cloning function performance.

[Andreas Neufert]

Customer shared the case number by PM. It is 04538735

[nn@zitcom.dk]

Anything new in this? I'm experiencing the same with XFS fast clone. Seeing really long merges, 10+ for 100-200 GB.

[Andreas Neufert]

Hi Nicolaj, if you experience this for your Cloud connect service, please open a support ticket and reference to ticket 04538735 to get the needed solution (reg key) after checking that it is the same root cause.