Backup Copy Performance Very Slow After Recreating Job

[wa15]

An odd issue here:

I recreated a backup copy that was to/from the same Veeam server but different volumes (for long-term GFS archiving to Cloud) and after recreating it, the performance is now less than half of what it used to be. The main difference I can think of is that now there is encryption enabled on this job. I was seeing 150MB/s originally, but now I am seeing 60MB/s at best.

The bottleneck on both jobs is showing source.

I have confirmed that the CPU, memory and disk are not under stress (CPU is at less than 30%, memory is at less than 35% and disk queue length is minimal). The server does not have any Antivirus installed (oddly enough, when the job performance was better, it did have AV installed). I have also tried rebooting the server.

Any ideas what could be going on? I did create a support ticket 03932713

[PetrM]

Hello!

According to bottleneck statistics the slowest operation is reading data from the disk on which source backup files are located.

I think there are 2 main possible causes which can slow down backup copy read process:
1) The disk itself is slow on read

2) Source full backup fragmentation:
For example, this side effect occurs when reversed or forever incremental chain is used for primary backup.
Backup copy transfers actual VM state, it opens source backup files in order to process necessary data blocks.
Read speed will become lower if backup copy needs to process some data blocks from such fragmented full.
Full backup file compact on primary backup chain might be helpful in this case.

Encryption usually increases CPU usage but it shouldn't affect read speed or do you mean that source backup files are encrypted?

Anyway, the best option is to continue working with our support team and to wait for conclusion.

Thanks!

[wa15]

Thanks, still working with support on this but here is an interesting behavior:

When starting this job (which just copies from one repository to another repository on the same server), the performance starts out good (300MB/sec+). However over the course of several hours, without any other jobs starting, it slowly tapers off to 60MB/s. This is a server full of internal drives (26 to be exact) and looking at metrics on this box (disk queue length, IOPS), the disk is not stressed even at 300MB/sec. Is this type of tapering off normal? One explanation could be the RAID cache getting full, but even with no cache I would expect more than 60MB/sec read speed.

Any thoughts?

[PetrM]

Hello!

The idea about cache is interesting and may be some hardware performance tests are required.

Let's wait for results of troubleshooting from our support team.
Please provide us with case number for our reference.

Thanks!