Discussions specific to the VMware vSphere hypervisor
Post Reply
vmwarektnx
Novice
Posts: 7
Liked: never
Joined: Jan 26, 2010 8:42 am
Full Name: Roger Knutsen

Slow replication speed when using CBT

Post by vmwarektnx » Mar 06, 2012 3:21 pm

I'm experiencing very slow replication in our environment

Have set up the following to reproduce the problem

Veeam 6.0.0.181
Veeam installed in a VM and using virtual appliance mode
A single 30GB VM stored on a FC EVA volume replicated to a Datadomain appliance witch is presented to ESXi as an NFS volume

Without CBT
Initial replication : Hard disk 1 30GB - 28 GB Read at 28MB/Sec 17:09
Next replication : Hard disk 1 30GB - 28 GB Read at 102MB/Sec 04:40

With CBT
Initial replication : Hard disk 1 30GB - 26.4 GB Read at 30MB/Sec 14:09
Next replication : Hard disk 1 30GB - 885 MB Read at 3MB/Sec 05:44 (CBT takes LONGER and have a very low transfer rate)

Does anyone know how Veeam does incremental backup when CBT is not used?
It seems that the NON-CBT replication job uses a lot more CPU one the Veeam VM

foggy
Veeam Software
Posts: 17591
Liked: 1467 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Slow replication speed when using CBT

Post by foggy » Mar 06, 2012 3:38 pm

Roger, what does the bottleneck stats show during the incremental with CBT enabled?

Gostev
SVP, Product Management
Posts: 23854
Liked: 3207 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Slow replication speed when using CBT

Post by Gostev » Mar 06, 2012 7:18 pm

vmwarektnx wrote:Does anyone know how Veeam does incremental backup when CBT is not used?
It seems that the NON-CBT replication job uses a lot more CPU one the Veeam VM
The whole source image is read and each block is compared with its previous state to determine if it has changed. Because the whole image must be processed, of course the CPU load is much higher than with CBT, where we process only changed block.

Your performance issue is likely due to the target storage speed, and nothing to deal with using CBT. What is reported as the bottleneck by the job?

tsightler
VP, Product Management
Posts: 5281
Liked: 2140 times
Joined: Jun 05, 2009 12:57 pm
Full Name: Tom Sightler
Contact:

Re: Slow replication speed when using CBT

Post by tsightler » Mar 06, 2012 7:51 pm

vmwarektnx wrote:Without CBT
Initial replication : Hard disk 1 30GB - 28 GB Read at 28MB/Sec 17:09
Next replication : Hard disk 1 30GB - 28 GB Read at 102MB/Sec 04:40

With CBT
Initial replication : Hard disk 1 30GB - 26.4 GB Read at 30MB/Sec 14:09
Next replication : Hard disk 1 30GB - 885 MB Read at 3MB/Sec 05:44 (CBT takes LONGER and have a very low transfer rate)
I think you are likely misinterpreting the data. For one thing, the statistic is not a measure of the disk transfer rate, or the replication transfer rate, it is a very simple calculation of amount of data read/time. So let's look at how Veeam calculates the "MB/Sec" number:

Note that the incremental replication without CBT shows that it "Read" 28GB in 4:40 (280 seconds). So to get MB/sec you do (28GB * 1024)/280 sec = 102MB/sec.

For the incremental replication with CBT, Veeam only had to "Read" 885MB in 5:44 (344 seconds). So to get MB/sec you use the same formula, that's 885MB/344sec = 3MB/s

So in the first case, of course the MB/sec is very high, as Veeam read 28GB of data, but likely had to transfer very little of this data across the wire (the important part). In the second case, Veeam read 885MB, far less data, but probably similar amounts had to cross the wire, perhaps even more data (impossible to know with the information provided since you didn't post the amount of data transferred in each of these jobs).

vmwarektnx
Novice
Posts: 7
Liked: never
Joined: Jan 26, 2010 8:42 am
Full Name: Roger Knutsen

Re: Slow replication speed when using CBT

Post by vmwarektnx » Mar 07, 2012 8:28 am

Thanks for the feedback

I realize that the data transfer rate is calculation of amount of data read/time.
The thing that bothers me is that the replica without CBT takes a shorter time to replicate compared to with CBT when hardly any changes has been made to VM

With CBT : Source 1% Proxy 13% Network 3% Target 99%
Without CBT : Source 98% Proxy 58% Network 1% Target 45%

The "Target 99%" when CBT is used seems weird to me since the data transferred to the target should be the same with and without CBT and the job without CBT completes faster.

The amount of data transferred is specified :
Without CBT all the jobs transfer about 28 GB
With CBT only the initial replication transfer 28 GB, the next 885 MB

We actually replaced another backup software in this production environment to lower the replication windows by introducing CBT with Veeam. I was very surprised when Veeam actually took 4 hours longer to replicate than the old software witch does not support CBT.

tsightler
VP, Product Management
Posts: 5281
Liked: 2140 times
Joined: Jun 05, 2009 12:57 pm
Full Name: Tom Sightler
Contact:

Re: Slow replication speed when using CBT

Post by tsightler » Mar 07, 2012 2:24 pm

vmwarektnx wrote: With CBT : Source 1% Proxy 13% Network 3% Target 99%
Without CBT : Source 98% Proxy 58% Network 1% Target 45%
This is exactly what would be expected, when transferring with CBT the source is not going to be the bottleneck because we only read 885MB from the source, the target will be the bottleneck. On the other hand, when Veeam did not use CBT, it had to read the entire 28GB again, even though it only transferred some small subset of that data, thus the source was the bottleneck as 99% of the time was spent reading the 28GB worth of data, most of which didn't need to be transferred to the target.
vmwarektnx wrote: The amount of data transferred is specified :
Without CBT all the jobs transfer about 28 GB
With CBT only the initial replication transfer 28 GB, the next 885 MB
This is not the amount of data transferred, this is the amount of data read from the source, which is not the same thing. In the non-CBT example we had to read all 28GB of data, but only a small amount of that would have been transferred. The amount of data actually transferred to the target during the job is displayed on the real-time stats screen in the "Data" section in the upper-middle. You can also right-click the job and run "Report" from the context menu to see the amount of data transferred. The amount of data that has to actually be sent to the target is typically the important part of replication.

For example, if you "Non-CBT" run read 28GB, but only found 100MB of change data to transfer, then it's very likely it might faster than a "CBT" job run that had to transfer 800MB.

Post Reply

Who is online

Users browsing this forum: BenjaminB, Dmitrii and 37 guests