Expected fastclone merge times?

[tibus_alun]

We have a 146TB repo with 50TB of space used. The repo is a direct attached storage server with 2019 1809, REFS and 64K cluster size in RAID6 with write-back active.
32GB RAM, Intel Silver 4208 dual proc, 16core/32 thread.
Veeam B&R 10.0.1.4854
6 jobs running daily in forever forward incremental with 90 restore points and backing up VMs via directSAN over 10G iscsi network.
Each job is chained after another, so no merge operations clash.
The jobs are roughly even, with the full VBK about 3-4TB and incremental about 300-400GB, give or take.

While each daily incremental job completes the backup portion in about 20-30mins, the bulk of time in each is the task "Full backup file merge completed successfully [fast clone]", which takes 1.5 - 2 hours on each. This results in backup window of about 12 hours. As its the fastclone merges taking the most time and based on above detail, are such times expected? and/or any thoughts on changes to help improve this?

Thanks,

Alun.

[PetrM]

Hi Alun,

I'd say that it's not expected, especially the fact that merge is 3-4x slower than the incremental run. I'd suggest to open a support request and to ask our engineers to examine debug logs to understand what exactly slows down merge. Also, take a look at full backup file compact, perhaps merge would take less time to complete after the full is compacted.

Thanks!

[HannesK]

Hello,
that sounds too long for me (well, the number of disks is missing).

I suggest the following (in that order)
- check whether REFS.SYS is at least version 10.0.17763.1369
- add at least 32 GB RAM (I assume that the server is out of RAM)
- upgrade to V11 (we implemented some improvements with REFS)


Best regards,
Hannes

[tibus_alun]

Hi Alun,

I'd say that it's not expected, especially the fact that merge is 3-4x slower than the incremental run. I'd suggest to open a support request and to ask our engineers to examine debug logs to understand what exactly slows down merge. Also, take a look at full backup file compact, perhaps merge would take less time to complete after the full is compacted.

Thanks!

Thanks, I will look to schedule a compact task on one of the jobs, that will at least allow me to see a comparison with those without.

[tibus_alun]

Hello,
that sounds too long for me (well, the number of disks is missing).

I suggest the following (in that order)
- check whether REFS.SYS is at least version 10.0.17763.1369
- add at least 32 GB RAM (I assume that the server is out of RAM)
- upgrade to V11 (we implemented some improvements with REFS)


Best regards,
Hannes

- 10.0.17763.1369 is current REFS.SYS
- RAM usage is always around 50%, so not out of RAM
- Will plan an upgrade to v11

[tibus_alun]

Hi Alun,

I'd say that it's not expected, especially the fact that merge is 3-4x slower than the incremental run. I'd suggest to open a support request and to ask our engineers to examine debug logs to understand what exactly slows down merge. Also, take a look at full backup file compact, perhaps merge would take less time to complete after the full is compacted.

Thanks!

I scheduled a compact task yesterday and reran the job manually. The fastclone compact operation took 5 hours at the end of the job. This morning the job ran again on schedule and this time the fastclone merge took 20mins instead of the 2 hours for the previous merge. I will now look to schedule weekly compact on each of the 6 jobs. This should dramaticly change our backup window.

Thanks for the suggestion!