Hey All,
This response is not a solution to the OP's problem.
Wasabi has a limit of 250 TCP connections initiated in 1 minute window period from a single source IP address; this used to be 100 but was raised within the past 6 months. That is the only type of "throttling" on the Wasabi side. If you are being throttled, you will receive a response code that says you are being throttled. When customers ask to remove throttling, their account is checked to see if they are hitting the TCP connection limit. There is no message(GET/PUT/POST/etc...) throttling. For example, if a client opens as many TCP connections as needed from the start and then uses TCP connection reuse to keep them up, you can send as many S3 API requests as possible. This also means you can open more than 250 TCP connections from a single source IP address as long as you do not attempt to open more than 250 within the 1 minute window.
Most of the time, this TCP connection limit isn't the issue; however, it is more frequently seen for larger MSPs / VCSPs sending many backups at once. As we know, Veeam, by default, opens up 64 connections for each configured task, which can quickly run into the TCP connection limit for larger jobs. If you send 1TB, we equate this to ~1 million objects. Veeam will open 64 connections to start copying 64 objects. Each thread is independent, so after a single object has finished copying, the TCP connection is closed, and a new one is opened. These transfers happen fairly quickly since the default block size is 1MB. So what is seen is a ring of opening and closing 64 TCP connections until the ~1 million objects are finished copying. With a single VM attempting to be transferred simultaneously, you should never encounter any TCP connection limit. You shouldn't encounter this limit even with 3 VMs copying simultaneously. After that, you will open and close too many TCP connections in the 1-minute window period and start encountering errors with connection limits.
Now, attempting to send many VMs simultaneously to achieve a higher throughput is not necessarily what you want to do. For every VM, 64 connections are opening; at the 3 VM point, you will be at 192 connections opening, closing, and sending simultaneously. I equate this to a similar scenario, the ISO downloading issue. Back in the day, you want to download an ISO. The ISO is 4GB in size. Nothing happens when you click on a browser link, so you click on it a few more times. Now, it finally responds, and you download eight instances of the same ISO simultaneously. But instead of getting the total throughput of a single download, your download per ISO is <available speed> / 8. Instead of completing in 25 minutes, you have 8 running, each taking 4 hours. When backing up your VM's, would you rather have a single VM complete faster, then move on to the next VM, or the alternative would be trying to send 3 VM's, saturate the connection, and have them take 24 hours to complete a single VM.
In addition to using more CPU/Memory that needs to be accounted for on the VM/Veeam itself, your per-object transfer speed is also reduced. If you have a 1Gbit internet connection, assuming you can reach full speed to the Wasabi Vault, minus 10% for networking overhead. Copying 3 VMs simultaneously reduces the per object transfer speed to ~4.5Mbit. Chances are, with a speed test and routing, your full speed to the Vault will not be at the line rate. In addition, the more work you attempt to do at one time, the chance for the network/route to be saturated, causing packet loss, retransmission, latency, etc...This will make your backups take even longer.
Again, this is not a solution to OPs question, with 10Gb you are more likely to hit into these types of scenarios because you increase the tasks to achieve higher performance, thus hitting the TCP limit. Multiple proxies, each with a different source IP, are one solution to the problem; the other could be to get your TCP connection limit raised.
Either way, if you are hitting the TCP connection limit, you should know that you hit it. Veeam will show it in the response.
In OP's case, it looks like it's a single VM transfer of ~94GB, which isn't too much. I have recently seen speed tests from customers achieving 30Gbit over the public internet.
@Kazz
If you want to message me your Wasabi support ticket, I will take a look.
Thanks,
@ChrisWasabi
