Discussions related to using object storage as a backup target.
massimiliano.rizzi
Service Provider
Posts: 204
Liked: 25 times
Joined: Jan 24, 2012 7:56 am
Full Name: Massimiliano Rizzi
Contact:

Re: Wasabi US-East-2 Issues

Post by massimiliano.rizzi »

Hello Community and good day,

sorry for jumping straight into the thread.

I just opened Case # 05681023 in order to troubleshoot the warning message below that we have been observing for the last weeks as part of offloading to Wasabi EU Central 1 as well and which is similar to some of the errors reported in the OP:

==================================================
Checkpoint cleanup failed Details: Stopped by <Backup Job name>
==================================================

Although offload jobs are usually completing successfully in our scenario, they are frequently throwing the warning message above.

Thanks and Regards,

Massimiliano
veremin
Product Manager
Posts: 20270
Liked: 2252 times
Joined: Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin
Contact:

Re: Wasabi US-East-2 Issues

Post by veremin »

Have you reached the Wasabi support team as well? As noted in this thread, the problem is not related to Veeam Backup & Replication, and the only thing our support team can provide right now is temporary workarounds. Thanks!
massimiliano.rizzi
Service Provider
Posts: 204
Liked: 25 times
Joined: Jan 24, 2012 7:56 am
Full Name: Massimiliano Rizzi
Contact:

Re: Wasabi US-East-2 Issues

Post by massimiliano.rizzi »

Have you reached the Wasabi support team as well? As noted in this thread, the problem is not related to Veeam Backup & Replication, and the only thing our support team can provide right now is temporary workarounds. Thanks!
I have not but I will reach out to them in the afternoon.

I decided to reach out to Veeam support team as well in order to apply the temporary workarounds while waiting for Wasabi support team to provide a permanent fix for this.

Thanks !

Massimiliano
chrisWasabi
Technology Partner
Posts: 19
Liked: 33 times
Joined: Feb 23, 2021 3:42 pm
Contact:

Re: Wasabi US-East-2 Issues

Post by chrisWasabi » 3 people like this post

Hi All,

We are currently investigating the cause of this issue. If anyone is affected, please open a ticket with our support team at support@wasabi.com so we can further investigate. Our support team and engineering team are fully engaged in discovering the root cause.

Thanks,
chrisWasabi
RobMiller86
Service Provider
Posts: 138
Liked: 21 times
Joined: Oct 28, 2019 7:10 pm
Full Name: Rob Miller
Contact:

Re: Wasabi US-East-2 Issues

Post by RobMiller86 »

I did receive word yesterday from Wasabi that:
We have made some adjustments to our infrastructure and see that processing times improved last night.
I stopped my copy jobs to the SOBR with Wasabi as the capacity tier and allowed it to catch up. I have noticed improved performance since receiving the above email from Wasabi yesterday. Everything caught up, I re-enabled the copy jobs, and it now seems to be humming along much better. I still have the API deletes limited to 500. It's not as fast as before, but it does seem to be improved and at least functional now. I'll keep an eye on it.
veremin
Product Manager
Posts: 20270
Liked: 2252 times
Joined: Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin
Contact:

Re: Wasabi US-East-2 Issues

Post by veremin »

Good to know you have received an answer from the Wasabi team. Assuming everything goes well, you can gradually increase the MultipleDeleteObjects regkey value. Thanks!
fillerbunnie
Novice
Posts: 4
Liked: never
Joined: Apr 18, 2021 10:22 pm
Full Name: Elage
Contact:

Re: Wasabi US-East-2 Issues

Post by fillerbunnie »

We're seeing exactly the same thing to ap-southeast-2 (Sydney) so definitely not isolated.

It's just an annoyance at this point but our plan was to move all our archive data into Wasabi, so if it doesn't get fixed soon that plan will certainly be going in the bin!
veremin
Product Manager
Posts: 20270
Liked: 2252 times
Joined: Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin
Contact:

Re: Wasabi US-East-2 Issues

Post by veremin »

Have you already contacted the Wasabi support team? The more information they have, the better. Plus, they seem to be aware of the issue and providing solutions. Thanks!
chrisWasabi
Technology Partner
Posts: 19
Liked: 33 times
Joined: Feb 23, 2021 3:42 pm
Contact:

Re: Wasabi US-East-2 Issues

Post by chrisWasabi » 2 people like this post

fillerbunnie wrote: Oct 26, 2022 9:24 am We're seeing exactly the same thing to ap-southeast-2 (Sydney) so definitely not isolated.

It's just an annoyance at this point but our plan was to move all our archive data into Wasabi, so if it doesn't get fixed soon that plan will certainly be going in the bin!
Hi fillerbunnie,

I checked the logs for Syndey, and we do not see any instances of this issue occurring. We are able to determine if this is happening because the specific problem is related to a particular S3 API call, the "MultiObjectDelete". I would urge you to contact Wasabi Support (support@wasabi.com) if you are seeing an issue that is preventing backups from working.

Thanks,
wasabiChris
Regnor
Veeam Software
Posts: 934
Liked: 287 times
Joined: Jan 31, 2011 11:17 am
Full Name: Max
Contact:

Re: Wasabi US-East-2 Issues

Post by Regnor » 1 person likes this post

An update from our side. We were able to solve the Access denied errors; those were neither Veeam nor Wasabi related.
Now we're seeing errors for the cleanups or when trying to delete backups:
"Failed to delete backup Error: HTTP exception: WinHttpQueryDataAvaliable: 12152: The server returned an invalid or unrecognized response, error code: 12152"

Not sure if this is still related to the problems mentioned in this topic.
I have contacted Wasabi Support and will see if they can find anything.
Post Reply

Who is online

Users browsing this forum: Stupots and 12 guests