Slow replication speed when using CBT

[vmwarektnx]

I'm experiencing very slow replication in our environment

Have set up the following to reproduce the problem

Veeam 6.0.0.181
Veeam installed in a VM and using virtual appliance mode
A single 30GB VM stored on a FC EVA volume replicated to a Datadomain appliance witch is presented to ESXi as an NFS volume

Without CBT
Initial replication : Hard disk 1 30GB - 28 GB Read at 28MB/Sec 17:09
Next replication : Hard disk 1 30GB - 28 GB Read at 102MB/Sec 04:40

With CBT
Initial replication : Hard disk 1 30GB - 26.4 GB Read at 30MB/Sec 14:09
Next replication : Hard disk 1 30GB - 885 MB Read at 3MB/Sec 05:44 (CBT takes LONGER and have a very low transfer rate)

Does anyone know how Veeam does incremental backup when CBT is not used?
It seems that the NON-CBT replication job uses a lot more CPU one the Veeam VM

[foggy]

Roger, what does the bottleneck stats show during the incremental with CBT enabled?

[Gostev]

Does anyone know how Veeam does incremental backup when CBT is not used?
It seems that the NON-CBT replication job uses a lot more CPU one the Veeam VM

The whole source image is read and each block is compared with its previous state to determine if it has changed. Because the whole image must be processed, of course the CPU load is much higher than with CBT, where we process only changed block.

Your performance issue is likely due to the target storage speed, and nothing to deal with using CBT. What is reported as the bottleneck by the job?

[tsightler]

Without CBT
Initial replication : Hard disk 1 30GB - 28 GB Read at 28MB/Sec 17:09
Next replication : Hard disk 1 30GB - 28 GB Read at 102MB/Sec 04:40

With CBT
Initial replication : Hard disk 1 30GB - 26.4 GB Read at 30MB/Sec 14:09
Next replication : Hard disk 1 30GB - 885 MB Read at 3MB/Sec 05:44 (CBT takes LONGER and have a very low transfer rate)

I think you are likely misinterpreting the data. For one thing, the statistic is not a measure of the disk transfer rate, or the replication transfer rate, it is a very simple calculation of amount of data read/time. So let's look at how Veeam calculates the "MB/Sec" number:

Note that the incremental replication without CBT shows that it "Read" 28GB in 4:40 (280 seconds). So to get MB/sec you do (28GB * 1024)/280 sec = 102MB/sec.

For the incremental replication with CBT, Veeam only had to "Read" 885MB in 5:44 (344 seconds). So to get MB/sec you use the same formula, that's 885MB/344sec = 3MB/s

So in the first case, of course the MB/sec is very high, as Veeam read 28GB of data, but likely had to transfer very little of this data across the wire (the important part). In the second case, Veeam read 885MB, far less data, but probably similar amounts had to cross the wire, perhaps even more data (impossible to know with the information provided since you didn't post the amount of data transferred in each of these jobs).

[vmwarektnx]

Thanks for the feedback

I realize that the data transfer rate is calculation of amount of data read/time.
The thing that bothers me is that the replica without CBT takes a shorter time to replicate compared to with CBT when hardly any changes has been made to VM

With CBT : Source 1% Proxy 13% Network 3% Target 99%
Without CBT : Source 98% Proxy 58% Network 1% Target 45%

The "Target 99%" when CBT is used seems weird to me since the data transferred to the target should be the same with and without CBT and the job without CBT completes faster.

The amount of data transferred is specified :
Without CBT all the jobs transfer about 28 GB
With CBT only the initial replication transfer 28 GB, the next 885 MB

We actually replaced another backup software in this production environment to lower the replication windows by introducing CBT with Veeam. I was very surprised when Veeam actually took 4 hours longer to replicate than the old software witch does not support CBT.

[tsightler]

With CBT : Source 1% Proxy 13% Network 3% Target 99%
Without CBT : Source 98% Proxy 58% Network 1% Target 45%

This is exactly what would be expected, when transferring with CBT the source is not going to be the bottleneck because we only read 885MB from the source, the target will be the bottleneck. On the other hand, when Veeam did not use CBT, it had to read the entire 28GB again, even though it only transferred some small subset of that data, thus the source was the bottleneck as 99% of the time was spent reading the 28GB worth of data, most of which didn't need to be transferred to the target.

The amount of data transferred is specified :
Without CBT all the jobs transfer about 28 GB
With CBT only the initial replication transfer 28 GB, the next 885 MB

This is not the amount of data transferred, this is the amount of data read from the source, which is not the same thing. In the non-CBT example we had to read all 28GB of data, but only a small amount of that would have been transferred. The amount of data actually transferred to the target during the job is displayed on the real-time stats screen in the "Data" section in the upper-middle. You can also right-click the job and run "Report" from the context menu to see the amount of data transferred. The amount of data that has to actually be sent to the target is typically the important part of replication.

For example, if you "Non-CBT" run read 28GB, but only found 100MB of change data to transfer, then it's very likely it might faster than a "CBT" job run that had to transfer 800MB.