Long distance and packet size

[unsichtbarre]

I have a really long distance replication (Middle East to East Coast USA) over a Cisco IPsec VPN. Both sides have 100Mbps WAN, but replication is just trickling over the wire.

On a hunch I did this from Veeam Proxy to Veeam Proxy: ping -l 1472 -d <IP of other Veeam Proxy>

Turns out the largest ping was 1410. Does this mean that every frame produced by Veeam is being broken up?

I tried some iperf while adjusting the frame size in Windows, but was unable to produce any better result than about 30Mbps by reducing the frame size in the windows driver to 1400.

Moreover, when increasing the number of threads in iperf beyond about 10 (-P 10), the additional threads would error out.

Any suggestions?

[tsightler]

In general, the MTU isn't a major concern as Veeam transmit data larger than the packet size anyway so the data is always chunked up by the network layer. With a VPN you're going to loose a little so this isn't a big deal UNLESS the VPN or some other network gear is blocking ICMP so that the TCP stack can discover the MTU. Normally, if a packet is sent that is larger than the MTU, and Path MTU Discovery is enabled in the stack (pretty much any modern OS defaults) then the network gear is expected to respond with an ICMP that fragementation was needed but not allowed. This tells the sending device that some device in the network path has a smaller MTU and lets it automatically adjust. However, I've seen quite a few cases where network admins block all ICMP in the name of protecting from DoS attacks, and break Path MTU discovery. This is easy to check with ping or you can just change the MTU in Windows to the size you know works.

That being said, I'd think this is unlikely to be the issue in your case.

What is the latency on your link? The core issue here is likely that you have what is known as a "long-flat network", that is, you have plenty of bandwidth, but very high latency that requires a large TCP window. When you're testing with iperf, have you tried some large -w values to force a larger TCP window? Assuming you around 200ms latency (perhaps optimistic), you'd need a TCP window of around 2500KB, to fully utilize 100Mb. The typical default TCP window size is 64K, so that might explain it.

If you run more threads in iperf, say 4, do you get better performance? The fact that you seeing issues with >10 threads is concerning, but, in an ideal world, you wouldn't need that many as you should be able to tune the streams to use the available bandwidth by growing the TCP window. Any modern Windows server should have TCP window scaling on by default, but I've seen it misbehave in some extreme scenarios.

I'd probably look at grabbing some packet traces of iperf and, perhaps Veeam traffic as well, and see what TCP windows you are seeing and if you're seeing lots of dropped packets, etc. The fact that you can only get 30Mbps with iperf is concerning and makes me think something else is going on.

[unsichtbarre]

I went up to -w 256K and then iperf started failing
I could go up to 10 threads which improved performance (only 6Mbps on a single thread), but after 10 threads, the individual thread would error.

[unsichtbarre]

Does anyone think disabling MSS Blocking on Cisco ASA coule help?

More data:

Ping RTT is 177ms - not that bad.

THX

[gmajestix]

We had a site to site IPsec but icmp was blocked. We were doing a Veeam replication over IPsec. When we enabled ICMP on both ends throughput went considerably higher.

[unsichtbarre]

In our case, ICMP is clearly allowed, so that's not the issue.