Slow CBT rate on WAN replications

[kpnz]

We're trying to make replication between our two sites more efficient so we can replicate more VM's over to our DR site.

The problem we have however is that a replication done using CBT is slower than doing a full copy, and slower than a replication without CBT enabled. I've got a call logged with Veeam (00475050) but haven't really had any success or even any indication as to the problem.

Below is the result of a replication using CBT:

18/11/2013 7:03:17 p.m. :: Queued for processing at 18/11/2013 7:03:17 p.m.
18/11/2013 7:03:18 p.m. :: Required backup infrastructure resources have been assigned
18/11/2013 7:03:20 p.m. :: VM processing started at 18/11/2013 7:03:20 p.m.
18/11/2013 7:03:20 p.m. :: VM size: 100.0 GB
18/11/2013 7:03:23 p.m. :: Discovering replica VM
18/11/2013 7:03:32 p.m. :: Creating VM snapshot
18/11/2013 7:04:13 p.m. :: Preparing replica VM
18/11/2013 7:04:43 p.m. :: Processing configuration
18/11/2013 7:05:09 p.m. :: Creating helper snapshot
18/11/2013 7:05:25 p.m. :: Using backup proxy FHCMGTVP01 for disk Hard disk 1 [hotadd]
18/11/2013 7:05:25 p.m. :: Using target proxy FHCAKLDRVP01 for disk Hard disk 1 [hotadd]
>> 18/11/2013 7:05:36 p.m. :: Hard disk 1 (40.0 GB) #### read at 3MB/s [CBT]
18/11/2013 7:23:52 p.m. :: Swap file blocks skipped: 44.0 MB
18/11/2013 7:23:54 p.m. :: Using backup proxy FHCMGTVP01 for disk Hard disk 2 [hotadd]
18/11/2013 7:23:54 p.m. :: Using target proxy FHCAKLDRVP01 for disk Hard disk 2 [hotadd]
>> 18/11/2013 7:24:06 p.m. :: Hard disk 2 (60.0 GB) #### read at 4MB/s [CBT]
18/11/2013 7:28:54 p.m. :: Removing VM snapshot
18/11/2013 7:29:48 p.m. :: Deleting helper snapshot
18/11/2013 7:31:17 p.m. :: 1 restore point removed by retention policy
18/11/2013 7:41:41 p.m. :: Finalizing
18/11/2013 7:41:45 p.m. :: Busy: Source 8% > Proxy 1% > Network 4% > Target 99%
18/11/2013 7:41:45 p.m. :: Primary bottleneck: Target
18/11/2013 7:41:45 p.m. :: Network traffic verification detected no corrupted blocks
18/11/2013 7:41:45 p.m. :: Processing finished at 18/11/2013 7:41:45 p.m.

Below is the same job without CBT:

19/11/2013 10:33:26 a.m. :: Queued for processing at 19/11/2013 10:33:26 a.m.
19/11/2013 10:33:26 a.m. :: Required backup infrastructure resources have been assigned
19/11/2013 10:33:28 a.m. :: VM processing started at 19/11/2013 10:33:28 a.m.
19/11/2013 10:33:28 a.m. :: VM size: 100.0 GB
19/11/2013 10:33:31 a.m. :: Discovering replica VM
19/11/2013 10:33:48 a.m. :: Creating VM snapshot
19/11/2013 10:34:39 a.m. :: Preparing replica VM
19/11/2013 10:34:59 a.m. :: Processing configuration
19/11/2013 10:35:22 a.m. :: Creating helper snapshot
19/11/2013 10:35:39 a.m. :: Using backup proxy 1 for disk Hard disk 1 [hotadd]
19/11/2013 10:35:40 a.m. :: Using target proxy 2 for disk Hard disk 1 [hotadd]
>> 19/11/2013 10:35:50 a.m. :: Hard disk 1 (40.0 GB) #### read at 55MB/s
19/11/2013 10:47:40 a.m. :: Swap file blocks skipped: 6.0 GB
19/11/2013 10:47:42 a.m. :: Using backup proxy 1 for disk Hard disk 2 [hotadd]
19/11/2013 10:47:43 a.m. :: Using target proxy 2 for disk Hard disk 2 [hotadd]
>> 19/11/2013 10:47:53 a.m. :: Hard disk 2 (60.0 GB) #### read at 80MB/s
19/11/2013 11:02:05 a.m. :: Removing VM snapshot
19/11/2013 11:02:58 a.m. :: Deleting helper snapshot
19/11/2013 11:04:02 a.m. :: 1 restore point removed by retention policy
19/11/2013 11:09:28 a.m. :: Finalizing
19/11/2013 11:09:32 a.m. :: Busy: Source 85% > Proxy 11% > Network 15% > Target 36%
19/11/2013 11:09:32 a.m. :: Primary bottleneck: Source
19/11/2013 11:09:32 a.m. :: Network traffic verification detected no corrupted blocks
19/11/2013 11:09:32 a.m. :: Processing finished at 19/11/2013 11:09:32 a.m.

From what I've read on other similar threads is that the Target Storage is the issue (although all the threads I found weren't conclusive)- I'm not convinced this is the case in this instance. I'm not seeing the esxi host or the target proxy under any load whilst replications are running. As an example a particular VM was initially only taking 4 hours to replicate now that same VM is taking over 12 hours. Hardware hasn't changed in this time, only Veeam versions.

[veremin]

I'm wondering whether pair of proxy used during both runs was the same, since it's not that clear from the log provided (FHCMGTVP01/FHCAKLDRVP01, backup proxy 1/target proxy 1). Thanks.

[kpnz]

Sorry they are the same. I just edited one log and remove the names but not the other.

[veremin]

The other suspicious thing is the discrepancy in the bottleneck statistics, more specifically, amount of time target component was busy during different runs (99%, 36%, accordingly). As far as I can see, two runs were conducted at different times - 7:30, 10 30 am. So, may it be that at 7-30 the target host was overloaded with the side activities (apart from replication), and, therefore, could not provide any better performance?

Thanks.

[foggy]

Could you also provide the processed/read/transferred counters for those job runs (with and without CBT), Thanks.