Wasabi US-East-2 Issues

[massimiliano.rizzi]

Hello Community and good day,

sorry for jumping straight into the thread.

I just opened Case # 05681023 in order to troubleshoot the warning message below that we have been observing for the last weeks as part of offloading to Wasabi EU Central 1 as well and which is similar to some of the errors reported in the OP:

==================================================
Checkpoint cleanup failed Details: Stopped by <Backup Job name>
==================================================

Although offload jobs are usually completing successfully in our scenario, they are frequently throwing the warning message above.

Thanks and Regards,

Massimiliano

[veremin]

Have you reached the Wasabi support team as well? As noted in this thread, the problem is not related to Veeam Backup & Replication, and the only thing our support team can provide right now is temporary workarounds. Thanks!

[massimiliano.rizzi]

Have you reached the Wasabi support team as well? As noted in this thread, the problem is not related to Veeam Backup & Replication, and the only thing our support team can provide right now is temporary workarounds. Thanks!

I have not but I will reach out to them in the afternoon.

I decided to reach out to Veeam support team as well in order to apply the temporary workarounds while waiting for Wasabi support team to provide a permanent fix for this.

Thanks !

Massimiliano

[chrisWasabi]

Hi All,

We are currently investigating the cause of this issue. If anyone is affected, please open a ticket with our support team at support@wasabi.com so we can further investigate. Our support team and engineering team are fully engaged in discovering the root cause.

Thanks,
chrisWasabi

[RobMiller86]

I did receive word yesterday from Wasabi that:

We have made some adjustments to our infrastructure and see that processing times improved last night.

I stopped my copy jobs to the SOBR with Wasabi as the capacity tier and allowed it to catch up. I have noticed improved performance since receiving the above email from Wasabi yesterday. Everything caught up, I re-enabled the copy jobs, and it now seems to be humming along much better. I still have the API deletes limited to 500. It's not as fast as before, but it does seem to be improved and at least functional now. I'll keep an eye on it.

[veremin]

Good to know you have received an answer from the Wasabi team. Assuming everything goes well, you can gradually increase the MultipleDeleteObjects regkey value. Thanks!

[fillerbunnie]

We're seeing exactly the same thing to ap-southeast-2 (Sydney) so definitely not isolated.

It's just an annoyance at this point but our plan was to move all our archive data into Wasabi, so if it doesn't get fixed soon that plan will certainly be going in the bin!

[veremin]

Have you already contacted the Wasabi support team? The more information they have, the better. Plus, they seem to be aware of the issue and providing solutions. Thanks!

[chrisWasabi]

We're seeing exactly the same thing to ap-southeast-2 (Sydney) so definitely not isolated.

It's just an annoyance at this point but our plan was to move all our archive data into Wasabi, so if it doesn't get fixed soon that plan will certainly be going in the bin!

Hi fillerbunnie,

I checked the logs for Syndey, and we do not see any instances of this issue occurring. We are able to determine if this is happening because the specific problem is related to a particular S3 API call, the "MultiObjectDelete". I would urge you to contact Wasabi Support (support@wasabi.com) if you are seeing an issue that is preventing backups from working.

Thanks,
wasabiChris

[Regnor]

An update from our side. We were able to solve the Access denied errors; those were neither Veeam nor Wasabi related.
Now we're seeing errors for the cleanups or when trying to delete backups:
"Failed to delete backup Error: HTTP exception: WinHttpQueryDataAvaliable: 12152: The server returned an invalid or unrecognized response, error code: 12152"

Not sure if this is still related to the problems mentioned in this topic.
I have contacted Wasabi Support and will see if they can find anything.