Wasabi issues, thoughts

[ahoyle]

With Wasabi's semi recent data loss issue in the Central US region. We thought things were ok then recently started getting 500 connection errors when running O365 backups to the Wasabi bucket. "The remote server returned an error: (500) Internal Server Error."

Currently working with Wasabi support to see if this is related to the same data loss or an ironic separate issue.

Has anyone seen any lasting issues with data loss from Wasabi storage?

Thanks!

[david.domask]

Hi Allen,

Sorry to hear about the troubles; I vaguely remembered the Wasabi incident in question, but was not aware how extensive it was.

I'm not aware of any specific fallout from this incident that would persist to today; Internal Server Error can be a response from the Wasabi side if the S3 client (Veeam in this case) is too aggressive with API calls.

I would advise open a Support Case and let Support review the situation as well, as if it's about being too aggressive for the API, Support can provide some tweaks which may help.

[ahoyle]

Hi David,

I currently have a case open with support, thinking its going to be the same course of action now that Wasabi has confirmed that the data has been lost and is unrecoverable.

Which, unfortunately, means losing any data that is currently in the bucket and starting over.

The response from Wasabi about the issue is:
"On 30 August 2024, an incident occurred in our us-central-1 region where a storage system I/O module became inoperable and prevented access to some disks that it served. Simultaneously, the software managing these disks improperly took multiple other disks offline. As a result of this incident, some objects on these disks are no longer accessible. The software problem that triggered this event has now been addressed."

Found here: https://status.wasabi.com/incidents/b7jjmvl8yw0r

So maybe not that recent, but still giving us some grief.

Thanks!