Offloading (with a Gateway Server) + Network Throttling Behavior

[ddasal]

I am trying to understand what behavior I should expect when running the following configuration.

• Linux based SOBR extents

• Linux based Gateway Server (for offloading use only)

• Network Throttling enabled with various speeds to the object storage provider based on the time of day

I end up seeing a lot of failed attempts during the offloading jobs (sample error below), while other tasks of the same offload job continue to run and will succeed (most of the time, but might also fail if there are multiple disks/tasks per VM/machine).

I am looking for clarity if this is normal behavior. I assume when the next offload job runs, it will just pickup and keep going where things left off (or continue the task if a timeout value is not met), but I am trying to understand why the "failed to connect" to the gateway-server from the repository is happening (even if it is expected behavior).

My uneducated guess is that while we have plenty of cores/tasks available to start lots of tasks between the linux repositories and the linux gateway server we are using for offloading, once connections are made, network throttling is preventing or forcing failures as other tasks are currently consuming the allowed bandwidth…. am I on to something here?

The error in the console:
Failed to offload backup. Error: Failed to connect to the port [<gatewayserverhostname>:2527].
--tr:Failed to connect to target endpoint.
--tr:Client failed to process the command. Command: [connectByIPs].
--tr:event:1:
--tr:event:3:

I can open a support ticket if what I am seeing is not expected, but I thought I'd start here just to gain an understand of what I should or should not be seeing in this case.

[PetrM]

Hi Dustin,

The behavior is not normal, I think the issue occurs due to extra load on repository/gateway itself or on network channel between repository and gateway. Maybe it would make sense to decrease the number of concurrent tasks in repository settings. One more idea is to collect and examine network traffic dump from both nodes. Anyway, our support team will find out the best approach for troubleshooting, please open a support case and share ID with us.

Thanks!

[ddasal]

Case #05266477

Are you referring to the object storage repository concurrent tasks setting or on other infrastructure components? I believe that is what you mean... I adjusted it to 32 for now to see if that will make any difference. It looks like the default is 64 with a recommended maximum of 94 for cloud providers. 32 was the recommendation for an on-premises object store. We are using a cloud provider, but I thought I would start there.

[veremin]

Petr was talking about these settings. Can you elaborate on source of information regarding having 64 concurrent tasks as default values? Thanks!

[ddasal]

I found this article as a reference on setting limits.
https://www.veeam.com/blog/sobr-architecture-guide.html

"Veeam’s default setting is to have a maximum number of 64 S3/BLOB operations per Repository Task slot" from below.

Performance tuning for SOBR Offload
Another factor to consider when designing your use of a Capacity Tier is the number of concurrent S3/BLOB operations connections that Veeam Backup & Replication will use to transfer data to the Capacity Tier via the SOBR Offload jobs. Veeam’s default setting is to have a maximum number of 64 S3/BLOB operations per Repository Task slot. When using on-prem object storage, we recommend keeping the maximum number of parallel to be less than 2048 which is achieved by utilizing 32 Repository Task slots. For public cloud providers like AWS S3, Microsoft Azure BLOB and Google Cloud Storage, we recommend keeping the number of connections below 6016, which means utilizing up to 94 Repository Task slots.

[veremin]

Got it, but in your case you'd rather start with limiting repository task limit first - the setting we have mentioned above - and see how it goes. Thanks!