Ideal config for replication speed?

[colohost]

Hey all, I'm trying to figure out if there is an ideal configuration for replication speed across WAN.

I've got vcenter/vsphere clusters in two locations. I've got VBR11 with both Windows and Linux proxies on both sides, as well as proxies on each remote side attached to a local master server. The proxies have direct storage access and of course vsphere/vcenter access.

Between data centers we have a 10gig point to point, but it's Cogent so it's packetized and you can only get the full 10gig via parallel streams. For example, iperf running between veeam proxies on both sides will typically see about 1.5 Gbit/sec, but if I set it to parallel streams of 3 or 4, it will be in the 6-7 Gbps range, 5+ gets it to saturate the 10Gbps. The link also supports jumbo MTU.

If I set up a replication job and set it to "Auto" for both source and target proxy selection, Veeam chooses a local proxy on the source side, it chooses to use storage snapshot (great), but then it seems to directly target a vsphere host across the wan link to replicate data to directly so only that one local proxy is involved in everything. This goes very slowly, as I suspect the latency/jitter and lack of jumbo frames hurts it. With tcpdump, I also suspect it's using a single stream because the target vSphere server likely has no concept of parallel streams. I let it do optimal compression, and for a 46 GB VM it transfers 18.5 GB and reports a 51 MB/sec transfer rate. Obviously less than ideal on a 10gig circuit; took 29 minutes for the job to complete.

So, next config attempt is I go into the replication job and force it to choose a local proxy for source, and force one of the remote proxies attached to the local master server as the target. The local proxy and remote proxy have a jumbo MTU, and hopefully also use parallel streams. This definitely works better. The job does consume about 5 minutes of storage snapshot prep plus "processing" before any real data flows, but it definitely gets going, the same job takes 6:33 total down from 29 minutes; 547 MB/sec effective rate. A re-run only needs to transfer 316 MB but still takes 5 minutes due mostly to the time to create the snapshot, "process" it, etc.

Now that I know my influencing of the settings can have such a dramatic effect, I'm wondering if there are any techniques to further optimize this? Before I spend too much time trying a million scenarios, was hoping to see if anyone had optimal configs for replication throughput. Perhaps I should go back to vm snapshots since those seem to get taken so much quicker, but wasn't sure if there is an optimal proxy config on each end to best optimize.

Thanks!

[Mildur]

Hi Colo

For a single VM replication from a Storage Snapshot is not faster than Replication from a VM Snapshot on vSphere.
A Storage Snapshot is created after all VMs are prepared for the Storage Snapshot. If you have a database server with heavy load in your replication job, all other VMs need to wait till this database server has a VM snapshot. Only then the storage snapshot will be created.
Also please don't forget, our proxy server will do some tasks before it sends the data to the target proxy server:
- Filter out: Zero Data Blocks, SWAP Files, Excluded VM guest OS Files
- Compressing

What is the shown bottleneck in your replication job? Do the target vSphere hosts have a 1 Gbit or 10 Gbit management interface?

Best,
Fabian

[colohost]

Thanks! The proxies are VM's on each side, and the vsphere hosts are 10gig data+storage on the source side, 25gig data+storage on the target. The bottleneck shows as being the proxy both before and after my change of adding in a forced target proxy. I'm guessing that limit though is how fast the proxy can push data into vsphere, so that's where I was hoping maybe there are some secrets to increase the speed.

We do two types of replication; one is normal with lots of VM's as part of regular disaster recovery, but we also do one-off replication jobs for migrating customers between data centers. I'll switch the one-off jobs to just use vm snaps since those are only one or two vm's at a time and it sounds like in no way benefit from storage snaps.