Struggling with WAN incremental replication

[mcwill]

Hopefully someone can point out where I'm going wrong with incremental replication, we are seeing occasions when the incremental job is taking longer than the initial full run!

Our setup is currently Primary Site with 4Mbs leased line (shortly to increase to 50Mbs) replicating to DR site with 10Mbs Down/2Mbs Up and it appears we are being limited by the remote site's send speed of 2Mbs.



Hopefully the image above can explain what we are seeing on a VM with a 4GB disk, just before 13:00 the initial replication was kicked off and finished just after 15:30. This was fine, we were contrained by the speed of the primary site link and if we were to put this into production initial replication would be by portable disk.

However, once the replication had completed we started an incremental replication which took 1.5 hours to transfer 122MB of changes (I assume the .vbr size is what I should be looking at.)

Both initial & incremental replication ran with "SAN/NBD with changed block tracking".

Any suggestions on how to improve this?

Thanks
Iain McWilliams

[Gostev]

Hello Iain, I will be able to give you some suggestions on improving this, but first I need to know if you are replicating to "fat" ESX, or ESXi host?

[mcwill]

Anton,

Thanks for the offer of help, all servers (3 at primary site + 1 at DR site) are ESXi boxes.

Equallogic SAN at primary site & QNAP iscsi box at DR site.

Regards,
Iain

[Gostev]

Iain, in case of ESXi I recommend installing Veeam Backup in the target site (for replication jobs). This way, VMDK rebuilding will be local in the target site. Right now it is done over the network, and your target site send speed is being bottleneck.

Obviously, another option would be to increase your target site send speed, then you will be able to keep your current deployment unchanged.

Hope this helps,
Thanks!

[mcwill]

Anton,

Thanks for the suggestion but unfortunately it didn't work.

After setting up a Veeam server at the DR site it is only able to pull data at a max rate of 2Mbs compared to 4Mbs when pushing the same data.

There is a 33-38ms ping latency between the two sites, is that too high for the ESXi management network to handle?

Thanks,
Iain

[Gostev]

Hmm, I did not expect this issue... but yes, from your testing it sounds like VMware NBD is not optimized well to such conditions. Are you sure that the link was at full speed, not busy with anything else? Do you have some traffic stats for this scenario, similar to screenshot you have posted above? I want developers to take a look.

[mcwill]

Hi Anton,

The link was running at 50% of its capacity, see...



This is the same link that happily reached 500kBps in my first post. The above image is 2 runs, I didn't have the patience to wait for the first run to complete, so stopped it and tried rebooting the Veeam server to see if that would make any difference.

Just to make sure there is no congestion at the receiving end I've just pulled down a ISO via http over the same link and was able to peg the transmit at 4Mbs.

Regards,
Iain