Replication to iLand limited to 6-8 MB/s from Los Angeles

[jroback]

We've found that our transfer speeds from Los Angeles are severely limited to iLand's Dallas and Virginia datacenters. We have 300-500 Mbps internet links with different ISPs at different customer locations, but our Internet links never show usage beyond 60-100 Mbps utilization. The replication jobs to Virginia show about 8 MB/sec in Veeam. Veeam consistently reports bottleneck network: (1% source, 3% proxy, 96% network and 57% busy on target). Jobs to Dallas are about 20% faster.

I've been working with iland support for 4 weeks on this, and I've hit a roadblock where they can't explain it and don't seem interested in fixing it. They looked through my veeam configuration in detail and agreed that everything was configured correctly. They then told me I needed to contact their ISP (Zayo) and ask why all four of the ISPs I'm using are having trouble peering at full speeds. After I pushed back they agreed to talk to Zayo, but now they've come back and said that Zayo can't find a problem, so everything is working as it should.

When I had this issue at one customer, I believed that it was something wrong at my end. But now that I can duplicate the issue at four different customer locations with four different ISPs, I'm wondering if all of iLand's Los Angeles area are having the same issue and are just putting up with it.

For my smaller customers this is workable, but for larger sites with bigger VM's, the replication times are just unworkable. And I'm terrified to think what a failback would look like. (The initial replications took 5 weeks).

So I'd love to hear feedback from others:

1) if you use iLand for DRAAS services, what transfer rates are you able to get?
2) What speeds are other providers able to reach?
3) Does anyone have positive experience with other DRAAS providers that can replicate at more reasonable speeds that they would recommend?

Thanks!
Jeff

[Andreas Neufert]

Hello Jeff,

we are in close contact with iLand and informed them about this request and the reddit one.
Can you please reach out to iLand support to discuss test options. From our Software side it is not limited.

[jroback]

Thank you for looking into this, I appreciate the assistance! I'll PM you over a few more specifics. I started doing some packet captures on my own and I am seeing small window sizes in traffic from their servers, which makes this seem like a classic example of the Long Fat Pipe scenario. So it seems that something is causing iLand's side to use the smaller windows which limit the bandwidth, most likely seems a software configuration or a buffer problem in an interim device.

Out of curiosity, does Veeam do anything with TCP Receive Window Size on either side, or does it leave that to the Windows TCP stack?

[Andreas Neufert]

We leave this to the Windows/Linux stack and your Internet Routers...

There is TCP optimizations where we open by default 5 TCP connections to optimize transport and workaround latency.
Maybe it helps if you increase that value to 10 or 20 in the Network rules. It is just a potential optimization workaround not a solution.

[k00laid]

Hi Jeff,

I'm a Cloud Architect here at iland. I'm happy to follow up but can you please DM me some information on your organization and some case numbers?

[jroback]

k00laid, Thanks for reaching out! I'll send over a DM with more info.

At this point I'm fairly certain there aren't any Veeam issues directly, as I've been able to duplicate the performance issues with one of our VM's in an iLand data center. It appears to be either something with the network configurations at the iLand or iLand's ISP.

Andreas, thanks for the thoughts on TCP connections, We did find a small increase with increasing the connection to 10 TCP connections, but I believe iLand would need to do something on their end for this as well, wouldn't they?

Interestingly, I've been able to replicate the behavior using iPerf between Los Angeles and a non-veeam virtual machine in iLand's data center. What's very interesting is that my initial results showed that the problem is significantly worse for traffic going into iLand's data center. But what's even more interesting is that if I use multiple slower streams (20 10 mb streams, for example) I can get significantly higher overall speeds. When doing UDP tests, I see significant packet loss start at 10 Mbps for inbound traffic. So my current theories are that perhaps there's some sort of router or firewall (or NSX) at the iLand or ISP side doing shaping on inbound traffic, there's packet-per-packet load balancing across uneven links, or there's asymetric routing somewhere. But those are just guesses.

Hoping k00laid can help make some sense of this.