Recent issue connecting to Wasabi

[Andy24v]

We've been using Veeam to back up our servers to Wasabi Cloud storage for over 6 months with no issues. On the 28th September, all backups failed. Further investigation showed that I was no longer able to look at the properties of the Wasabi Cloud provider. Wasabi had issued a new certificate, and I following KB 3215 I added the URLs for the certificate revocation lists, and I was able to view the properties of the Cloud service.

I retried the backups and they now work, but only about 80% of the time. I have logged a ticket with Veeam support - 06337106, however every thing they have suggested hasn't worked. I have logged a case with Wasabi, and am waiting to hear back.

The error log shows the following:
[13.10.2023 13:28:24.787] < 852> aws | Creating HTTP client. API URI: [https://s3.eu-west-1.wasabisys.com/]
[13.10.2023 13:30:10.001] < 9408> aws | WARN|HTTP request failed, no retry attempts left (total timeout exceeded)
[13.10.2023 13:30:10.001] < 9408> aws | >> |WinHttpSendRequest: 12029: A connection with the server could not be established

The strange thing is that this isn't a hard error, and the issue occurs randomly.

Has anyone else had a similar issue?

Andy

[Andreas Neufert]

Maybe you are running in the throttling algorithm of Wasabi:
https://docs.wasabi.com/docs/setup-limi ... e-limiting
but it is usually the wrong error code for this. Do you use the Object Repository maximum number of task slots for offloading setting.
Maybe experiment with this.

As you checked the logs yourself. There is the RTS log in the root of the VBR logs. It lists the number of task slots used frequently.
Check what happened at 13:28 if there were more than usual task slots used?

Did anything change on the internet connection or firewall? I saw as well some issues with this at some other customers with the same error code. So it is worth checking firewall logs at this point in time.

You can escalate the support case by "contact a manager" button in the Veeam support system.

[Andy24v]

I have checked the4 RTS logs, and can see no difference in average connection pre and post 27th September (the day the problem started).

I have checked firewall logs during the failed backups, and I can't see any dropped traffic from the VBR server. I have even temporarily allowed all http and https traffic, and the backup has still failed.

Andy

[veremin]

It seems that the issues started when a new certificate was issued on September, 27th. Based on the support correspondence, the shared registry key solved the issue and backups are now running smoothly.

Let's wait for some time and see whether the issue reappears.

Thanks!

[timlr80]

I am having the same issue. What is this shared registry key you are talking about that solved the issue?

[veremin]

The registry key doesn't exactly solve all problems; rather, it disables certificate revocation checks. The key is named ObjectStorageTlsRevocationCheck, but I wouldn't recommend using it before contacting support to verify the true cause of the errors you're encountering.

Thanks!