Comprehensive data protection for all workloads
Post Reply
mcwill
Enthusiast
Posts: 64
Liked: 10 times
Joined: Jan 16, 2010 9:47 am
Full Name: Iain McWilliams
Contact:

Struggling with WAN incremental replication

Post by mcwill »

Hopefully someone can point out where I'm going wrong with incremental replication, we are seeing occasions when the incremental job is taking longer than the initial full run!

Our setup is currently Primary Site with 4Mbs leased line (shortly to increase to 50Mbs) replicating to DR site with 10Mbs Down/2Mbs Up and it appears we are being limited by the remote site's send speed of 2Mbs.

Image

Hopefully the image above can explain what we are seeing on a VM with a 4GB disk, just before 13:00 the initial replication was kicked off and finished just after 15:30. This was fine, we were contrained by the speed of the primary site link and if we were to put this into production initial replication would be by portable disk.

However, once the replication had completed we started an incremental replication which took 1.5 hours to transfer 122MB of changes (I assume the .vbr size is what I should be looking at.)

Both initial & incremental replication ran with "SAN/NBD with changed block tracking".

Any suggestions on how to improve this?

Thanks
Iain McWilliams
Gostev
Chief Product Officer
Posts: 31460
Liked: 6648 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Struggling with WAN incremental replication

Post by Gostev »

Hello Iain, I will be able to give you some suggestions on improving this, but first I need to know if you are replicating to "fat" ESX, or ESXi host?
mcwill
Enthusiast
Posts: 64
Liked: 10 times
Joined: Jan 16, 2010 9:47 am
Full Name: Iain McWilliams
Contact:

Re: Struggling with WAN incremental replication

Post by mcwill »

Anton,

Thanks for the offer of help, all servers (3 at primary site + 1 at DR site) are ESXi boxes.

Equallogic SAN at primary site & QNAP iscsi box at DR site.

Regards,
Iain
Gostev
Chief Product Officer
Posts: 31460
Liked: 6648 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Struggling with WAN incremental replication

Post by Gostev »

Iain, in case of ESXi I recommend installing Veeam Backup in the target site (for replication jobs). This way, VMDK rebuilding will be local in the target site. Right now it is done over the network, and your target site send speed is being bottleneck.

Obviously, another option would be to increase your target site send speed, then you will be able to keep your current deployment unchanged.

Hope this helps,
Thanks!
mcwill
Enthusiast
Posts: 64
Liked: 10 times
Joined: Jan 16, 2010 9:47 am
Full Name: Iain McWilliams
Contact:

Re: Struggling with WAN incremental replication

Post by mcwill »

Anton,

Thanks for the suggestion but unfortunately it didn't work.

After setting up a Veeam server at the DR site it is only able to pull data at a max rate of 2Mbs compared to 4Mbs when pushing the same data.

There is a 33-38ms ping latency between the two sites, is that too high for the ESXi management network to handle?

Thanks,
Iain
Gostev
Chief Product Officer
Posts: 31460
Liked: 6648 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Struggling with WAN incremental replication

Post by Gostev »

Hmm, I did not expect this issue... but yes, from your testing it sounds like VMware NBD is not optimized well to such conditions. Are you sure that the link was at full speed, not busy with anything else? Do you have some traffic stats for this scenario, similar to screenshot you have posted above? I want developers to take a look.
mcwill
Enthusiast
Posts: 64
Liked: 10 times
Joined: Jan 16, 2010 9:47 am
Full Name: Iain McWilliams
Contact:

Re: Struggling with WAN incremental replication

Post by mcwill »

Hi Anton,

The link was running at 50% of its capacity, see...

Image

This is the same link that happily reached 500kBps in my first post. The above image is 2 runs, I didn't have the patience to wait for the first run to complete, so stopped it and tried rebooting the Veeam server to see if that would make any difference.

Just to make sure there is no congestion at the receiving end I've just pulled down a ISO via http over the same link and was able to peg the transmit at 4Mbs.

Regards,
Iain
Post Reply

Who is online

Users browsing this forum: Baidu [Spider], Semrush [Bot] and 272 guests