intermittent failures to wasabi

[badabing888]

Hi all,

latest veeam v12 patched 2 different installs.

We have recently been hitting an issue in the last week with a few backup copy jobs direct to wasabi object storage.

Error: REST API error: 'S3 error: Account Connection Limited

Is the most common error

on prem Backup repo is xfs immutable
Wasabi side is also immutable.

Its only for some VM’s within the one job and usually retry will suceed after a few retries.

case ID 06358590

[veremin]

No, this is not a known issue. So, keep working with the support team. Thanks!

[fillerbunnie]

Yep we are seeing exactly this across 95% of our customers that use Veeam & Wasabi. Noticed it on the 12th.

Have logged to Wasabi, they are saying we are exceeding the connection limit of 200 connections per IP per minute per region, but something else must be happening here as it is virtually ALL our veeam customers and nothing has changed at our end...

[badabing888]

Yeah i fixed it but lowering the concurrent connection on the bucket for this use case it will be ok.

Thanks for the input

[badabing888]

forgot to add yes its has been from the 12th onwards.

IMO they have changed or are enforcing rate limits

[mptalaga]

We are experiencing this issue as well. Today, while troubleshooting it, I received the AccessDenied - Account Connection Limited when trying to login to the Wasabi Admin Console. Hard to troubleshoot anything when they are limiting your console access as well.

Our concurrent connection to the bucket is only set to 2, so we can't set it any lower at this point. We've had nothing but performance issues with Veeam v12 since we've upgraded to it. But we did that months ago, and now we can't go back. It seems like Veeam changed how it was communicating to S3 with the upgrade to v12 that requires more API calls?

[veremin]

Hi, Matt,

Can you provide us with the case numbers so we can verify what was the root cause of the experienced behavior?

Thanks!

[Gostev]

More API calls are certainly required with V12 as the new object storage format required for Direct to Object backup no longer leverages local cache, which we had when our object storage integration was limited to copying existing local backups to object storage. However, it's only a marginal increase comparing to how many calls are issued during actual backup, due to the sheer number of objects to be processed and each requiring own API call to upload, set/prolong immutability, etc.

Nevertheless, 12.1 does bring some further optimizations in this department, reducing those "extra" calls caused by the new V12 format as much as possible.