Wasabi Errors

[ctg49]

We're using Wasabi for S3 object storage, and starting maybe a week ago we began getting some errors with our offloads, hoping someone's seen it before I start opening tickets.


10/6/2019 9:47:43 AM :: Failed to offload backup Error: Amazon REST error: '<ErrorResponse xmlns="https://iam.amazonaws.com/doc/2010-05-0 ... ge>Maximum number of server active requests exceeded</Message></Error></ErrorResponse>', error code: 503
10/6/2019 9:47:43 AM :: Amazon REST error: '<ErrorResponse xmlns="https://iam.amazonaws.com/doc/2010-05-0 ... ge>Maximum number of server active requests exceeded</Message></Error></ErrorResponse>', error code: 503

On some jobs, these messages appear during the index preparation, in other it occurs during the actual traffic verification/just prior to actual offloads. Occasionally we'll also get jobs that get to the point of offloading say, one of three files, but after the first one offloads, it just perpetually sits without offloading the remaining two. When that happens it'll never proceed (let it sit for a few days, nothing changed), and stopping the job just sits it in a stopped state, also for a couple days. Requires a reboot of the server to clear.

[Gostev]

These are clearly Wasabi S3 errors.

[wishr]

Hi Chris,

According to Wasabi API manual, 503 error indicates that the server or service is temporarily unavailable. Also, that page says, there were some issues across both EU and US DCs and the ingest rate was temporary tuned down.

Anyway, I'd suggest raising a support case with Wasabi to confirm this is the case.

Thanks

[ctg49]

Perfect, I figured these were Wasabi errors based on the error message. Glad I wasn't wrong about that, and the Wasabi status messages is something I'll keep a closer eye on in the future. Thank you!

A question though, has there been any consideration placed on how VEEAM handles things like this? As an example, since making this post I had an offload job that was in the middle of an object repository cleanup, which has errored (with the same error code) but the job hasn't actually halted yet. Does that mean the job is still performing functions? Or does that mean it's now hung? Does the job have the capacity to identify errors in the offload process then recover gracefully or kill gracefully?

[Gostev]

The offload job will keep retrying for quite a while, but eventually it should give up.

[asdffdsa6131]

hello,
i am a long time wasabi user and i can tell you that wasabi has had serious growing pains.
dozens of problems in the last two months.
in fact if you goto https://status.wasabi.com/ right now, you will see that there are problems.

you should click the subscribe button and you will receive updates via email.

i give credit to wasabi, as they have been most forthright about the problems.
and they are adding an entire new data center expansion for the us-east.
i gave up using us-east and now us-west instead.

i use VBAR community edition and rclone for cloud backup.

the good news, is that i have had zero data corruptions despite the growing pains

hope this helps.

[mcz]

Unfortunately, we are also a part of this wasabi-nightmare... It started 2 weeks ago while browsing the buckets and we received these "500 internal error" messages. Now we cannot rescan our SOBR which has to be done to do further offloads. So we're stuck and wasabi tells me "engineers are working on it" but it doesn't help us at the moment. We're using EU data centers but the statuspage doesn't show any issues - allthough there are.

So frustrating this whole story... I'm always asking myself what would happen if you'd really needed your backups from S3 in case of an emergency (think about veeam's v10 feature, which would copy the latest restore points automatically to S3). You just cannot simply rely on wasabi...

[asdffdsa6131]

yes, i share your frustration about wasabi.
wasabi has led to me to have a broken heart just like this.
https://www.washingtonpost.com/health/2 ... -syndrome/

and i have been super-frustrated about veeam and its inability to copy the latest restore point to s3, really we had to wait until version 10 for this to happen!

[wasabi-jim]

Hi Folks - This is Jim from Wasabi's product management team. Below is a statement from our product development team that provides some more detail beyond what the status page conveys. Thank you to everyone for your patience. Regards, Jim

https://wasabi-support.zendesk.com/hc/e ... tober-2019

[asdffdsa6131]

shame on wasabi, for over two months, unreliable service!!!
what a scam, offer service at a price they cannot deliver and pretend to care.

yet wasabi has not refunded any money to me or to you.
wasabi falsely advertised their services, took our money and still cannot deliver many months later.

wasabi, either charge us a realistic fee and deliver documented service
or
return our money so we can goto amazon and save money with glacier.

what a scam!

[mcz]

Hi Jim,

thanks for this update. Allthough I understand the situation, I still got the feeling that our case does not really belong to the current issues in US-regions. We're using the EU location and I'm receiving "500 internal error" messages all the time. It was fine yesterday for let's say 8 hours but now I cannot use the service again.

My veeam has to do a rescan of the whole bucket but if we always receive this messages, it's impossible to finish. My rescan takes many days and until I haven't finished, I cannot offload which filled up my repository and now I cannot even backup because I do not have enough disk space. I'm totally trapped! I really need a fix from wasabi ASAP!!! Please, please don't let me down.

The request # on wasabi-side is 13002.

[vbelagodu]

Hi Folks,

My name is Vilas Belagodu, I am a Director of Partner Engineering at Wasabi. I can be reached at vilas@wasabi.com.

asdffdsa6131 - Please share your Wasabi ticket number - I will have the billing issue resolved immediately.

mcz, I am working with engineering on resolving your issue as well. Please stay tuned for more info via ticket 13002.

ctg49 - Are you still facing issue with “Maximum number of server active requests exceeded” - We have resolved those issues with a new software release around 8th Oct. Please share more info and possibly a wasabi ticket ID.

Regards,
Vilas Belagodu

[ctg49]

vbelagodu,

Nope, we've not had any issues in quite a while. We also had some previous issues with automated deletes/pruning failing which now have been resolved (although they still take forever, I assume that's expected). Whatever changes were made in the last week or two made a huge difference, thank you!

[lowlander]

From Veeam perspective : If there is enough space in the non-S3 extent of the SOBR, will the backup succeed if the S3 extent is not available ? Also when performing a rescan on the SOBR is the SOBR available during the rescan for serving backup/copy jobs ?

[wishr]

Hello,

Capacity Tier is an additional layer of SOBR and its availability does not affect operations of jobs, because those are leveraging Performance Tier extents.

Thanks

[mcz]

Hi Vilas,

just a short update: We're still receiving these internal error messages. What is the current status? Is the root cause known but cannot be fixed soon or is it still unknown what's going on?

[vbelagodu]

Hi MCZ,

I have been working your ticket in the background with my support team and also with engineering. After review of the storage profile from a metadata perspective, its likely that the issue is localized to this specific account and we working with engineering to address it ASAP, additional specifics was shared in the ticket on that matter. Thats the best mode of communication. We receive notifications and a method of tracking with it. Could i request you to keep the data updated via the ticket moving forward?

Regards,
Vilas Belagodu

[mcz]

Hi Vilas,

thanks for your post. Just got the following message from support:

"After discussing with the Engg team, we determine that we have a fix that we are targeting for mid-november for this type of error wherein the query for ListObjects performance of such bucket, based on having lots of objects under them will be improved. I really appreciate your patience here"

So far so good but there's so much time that we loose...

[mcz]

Hi Vilas,

just wannted to say thank you for the zoom meeting today. I really appreciate all your efforts and now I've got the feeling that wasabi listens to it's customers and doesn't let them down. I also liked that we were able to talk instead of writing thousands of mails...

Thanks!