Discussions related to using object storage as a backup target.
Post Reply
ctg49
Enthusiast
Posts: 65
Liked: 45 times
Joined: Feb 14, 2018 1:47 pm
Full Name: Chris Garlington
Contact:

Wasabi Errors

Post by ctg49 »

We're using Wasabi for S3 object storage, and starting maybe a week ago we began getting some errors with our offloads, hoping someone's seen it before I start opening tickets.


10/6/2019 9:47:43 AM :: Failed to offload backup Error: Amazon REST error: '<ErrorResponse xmlns="https://iam.amazonaws.com/doc/2010-05-0 ... ge>Maximum number of server active requests exceeded</Message></Error></ErrorResponse>', error code: 503
10/6/2019 9:47:43 AM :: Amazon REST error: '<ErrorResponse xmlns="https://iam.amazonaws.com/doc/2010-05-0 ... ge>Maximum number of server active requests exceeded</Message></Error></ErrorResponse>', error code: 503

On some jobs, these messages appear during the index preparation, in other it occurs during the actual traffic verification/just prior to actual offloads. Occasionally we'll also get jobs that get to the point of offloading say, one of three files, but after the first one offloads, it just perpetually sits without offloading the remaining two. When that happens it'll never proceed (let it sit for a few days, nothing changed), and stopping the job just sits it in a stopped state, also for a couple days. Requires a reboot of the server to clear.
Gostev
Chief Product Officer
Posts: 31460
Liked: 6648 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Wasabi Errors

Post by Gostev »

These are clearly Wasabi S3 errors.
wishr
Veteran
Posts: 3077
Liked: 453 times
Joined: Aug 07, 2018 3:11 pm
Full Name: Fedor Maslov
Contact:

Re: Wasabi Errors

Post by wishr »

Hi Chris,

According to Wasabi API manual, 503 error indicates that the server or service is temporarily unavailable. Also, that page says, there were some issues across both EU and US DCs and the ingest rate was temporary tuned down.

Anyway, I'd suggest raising a support case with Wasabi to confirm this is the case.

Thanks
ctg49
Enthusiast
Posts: 65
Liked: 45 times
Joined: Feb 14, 2018 1:47 pm
Full Name: Chris Garlington
Contact:

Re: Wasabi Errors

Post by ctg49 »

Perfect, I figured these were Wasabi errors based on the error message. Glad I wasn't wrong about that, and the Wasabi status messages is something I'll keep a closer eye on in the future. Thank you!

A question though, has there been any consideration placed on how VEEAM handles things like this? As an example, since making this post I had an offload job that was in the middle of an object repository cleanup, which has errored (with the same error code) but the job hasn't actually halted yet. Does that mean the job is still performing functions? Or does that mean it's now hung? Does the job have the capacity to identify errors in the offload process then recover gracefully or kill gracefully?
Gostev
Chief Product Officer
Posts: 31460
Liked: 6648 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Wasabi Errors

Post by Gostev »

The offload job will keep retrying for quite a while, but eventually it should give up.
asdffdsa6131
Expert
Posts: 121
Liked: 24 times
Joined: Sep 30, 2018 9:03 pm
Contact:

Re: Wasabi Errors

Post by asdffdsa6131 » 2 people like this post

hello,
i am a long time wasabi user and i can tell you that wasabi has had serious growing pains.
dozens of problems in the last two months.
in fact if you goto https://status.wasabi.com/ right now, you will see that there are problems.

you should click the subscribe button and you will receive updates via email.

i give credit to wasabi, as they have been most forthright about the problems.
and they are adding an entire new data center expansion for the us-east.
i gave up using us-east and now us-west instead.

i use VBAR community edition and rclone for cloud backup.

the good news, is that i have had zero data corruptions despite the growing pains

hope this helps.
mcz
Veeam Legend
Posts: 835
Liked: 172 times
Joined: Jul 19, 2016 8:39 am
Full Name: Michael
Location: Rheintal, Austria
Contact:

Re: Wasabi Errors

Post by mcz »

Unfortunately, we are also a part of this wasabi-nightmare... It started 2 weeks ago while browsing the buckets and we received these "500 internal error" messages. Now we cannot rescan our SOBR which has to be done to do further offloads. So we're stuck and wasabi tells me "engineers are working on it" but it doesn't help us at the moment. We're using EU data centers but the statuspage doesn't show any issues - allthough there are.

So frustrating this whole story... I'm always asking myself what would happen if you'd really needed your backups from S3 in case of an emergency (think about veeam's v10 feature, which would copy the latest restore points automatically to S3). You just cannot simply rely on wasabi...
asdffdsa6131
Expert
Posts: 121
Liked: 24 times
Joined: Sep 30, 2018 9:03 pm
Contact:

Re: Wasabi Errors

Post by asdffdsa6131 »

yes, i share your frustration about wasabi.
wasabi has led to me to have a broken heart just like this.
https://www.washingtonpost.com/health/2 ... -syndrome/

and i have been super-frustrated about veeam and its inability to copy the latest restore point to s3, really we had to wait until version 10 for this to happen!
wasabi-jim
Technology Partner
Posts: 3
Liked: 9 times
Joined: May 08, 2017 8:43 pm
Full Name: Jim Donovan
Location: Boston, MA USA
Contact:

Re: Wasabi Errors

Post by wasabi-jim » 2 people like this post

Hi Folks - This is Jim from Wasabi's product management team. Below is a statement from our product development team that provides some more detail beyond what the status page conveys. Thank you to everyone for your patience. Regards, Jim

https://wasabi-support.zendesk.com/hc/e ... tober-2019
asdffdsa6131
Expert
Posts: 121
Liked: 24 times
Joined: Sep 30, 2018 9:03 pm
Contact:

Re: Wasabi Errors

Post by asdffdsa6131 »

shame on wasabi, for over two months, unreliable service!!!
what a scam, offer service at a price they cannot deliver and pretend to care.

yet wasabi has not refunded any money to me or to you.
wasabi falsely advertised their services, took our money and still cannot deliver many months later.

wasabi, either charge us a realistic fee and deliver documented service
or
return our money so we can goto amazon and save money with glacier.

what a scam!
mcz
Veeam Legend
Posts: 835
Liked: 172 times
Joined: Jul 19, 2016 8:39 am
Full Name: Michael
Location: Rheintal, Austria
Contact:

Re: Wasabi Errors

Post by mcz »

Hi Jim,

thanks for this update. Allthough I understand the situation, I still got the feeling that our case does not really belong to the current issues in US-regions. We're using the EU location and I'm receiving "500 internal error" messages all the time. It was fine yesterday for let's say 8 hours but now I cannot use the service again.

My veeam has to do a rescan of the whole bucket but if we always receive this messages, it's impossible to finish. My rescan takes many days and until I haven't finished, I cannot offload which filled up my repository and now I cannot even backup because I do not have enough disk space. I'm totally trapped! I really need a fix from wasabi ASAP!!! Please, please don't let me down.

The request # on wasabi-side is 13002.
vbelagodu
Technology Partner
Posts: 6
Liked: 16 times
Joined: Oct 21, 2019 1:13 pm
Full Name: Vilas Belagodu
Contact:

Re: Wasabi Errors

Post by vbelagodu » 5 people like this post

Hi Folks,

My name is Vilas Belagodu, I am a Director of Partner Engineering at Wasabi. I can be reached at vilas@wasabi.com.

asdffdsa6131 - Please share your Wasabi ticket number - I will have the billing issue resolved immediately.

mcz, I am working with engineering on resolving your issue as well. Please stay tuned for more info via ticket 13002.

ctg49 - Are you still facing issue with “Maximum number of server active requests exceeded” - We have resolved those issues with a new software release around 8th Oct. Please share more info and possibly a wasabi ticket ID.

Regards,
Vilas Belagodu
ctg49
Enthusiast
Posts: 65
Liked: 45 times
Joined: Feb 14, 2018 1:47 pm
Full Name: Chris Garlington
Contact:

Re: Wasabi Errors

Post by ctg49 » 5 people like this post

vbelagodu,

Nope, we've not had any issues in quite a while. We also had some previous issues with automated deletes/pruning failing which now have been resolved (although they still take forever, I assume that's expected). Whatever changes were made in the last week or two made a huge difference, thank you!
lowlander
Service Provider
Posts: 450
Liked: 30 times
Joined: Dec 28, 2014 11:48 am
Location: The Netherlands
Contact:

Re: Wasabi Errors

Post by lowlander »

From Veeam perspective : If there is enough space in the non-S3 extent of the SOBR, will the backup succeed if the S3 extent is not available ? Also when performing a rescan on the SOBR is the SOBR available during the rescan for serving backup/copy jobs ?
wishr
Veteran
Posts: 3077
Liked: 453 times
Joined: Aug 07, 2018 3:11 pm
Full Name: Fedor Maslov
Contact:

Re: Wasabi Errors

Post by wishr »

Hello,

Capacity Tier is an additional layer of SOBR and its availability does not affect operations of jobs, because those are leveraging Performance Tier extents.

Thanks
mcz
Veeam Legend
Posts: 835
Liked: 172 times
Joined: Jul 19, 2016 8:39 am
Full Name: Michael
Location: Rheintal, Austria
Contact:

Re: Wasabi Errors

Post by mcz »

Hi Vilas,

just a short update: We're still receiving these internal error messages. What is the current status? Is the root cause known but cannot be fixed soon or is it still unknown what's going on?
vbelagodu
Technology Partner
Posts: 6
Liked: 16 times
Joined: Oct 21, 2019 1:13 pm
Full Name: Vilas Belagodu
Contact:

Re: Wasabi Errors

Post by vbelagodu » 1 person likes this post

Hi MCZ,

I have been working your ticket in the background with my support team and also with engineering. After review of the storage profile from a metadata perspective, its likely that the issue is localized to this specific account and we working with engineering to address it ASAP, additional specifics was shared in the ticket on that matter. Thats the best mode of communication. We receive notifications and a method of tracking with it. Could i request you to keep the data updated via the ticket moving forward?

Regards,
Vilas Belagodu
mcz
Veeam Legend
Posts: 835
Liked: 172 times
Joined: Jul 19, 2016 8:39 am
Full Name: Michael
Location: Rheintal, Austria
Contact:

Re: Wasabi Errors

Post by mcz »

Hi Vilas,

thanks for your post. Just got the following message from support:

"After discussing with the Engg team, we determine that we have a fix that we are targeting for mid-november for this type of error wherein the query for ListObjects performance of such bucket, based on having lots of objects under them will be improved. I really appreciate your patience here"

So far so good but there's so much time that we loose...
mcz
Veeam Legend
Posts: 835
Liked: 172 times
Joined: Jul 19, 2016 8:39 am
Full Name: Michael
Location: Rheintal, Austria
Contact:

Re: Wasabi Errors

Post by mcz » 3 people like this post

Hi Vilas,

just wannted to say thank you for the zoom meeting today. I really appreciate all your efforts and now I've got the feeling that wasabi listens to it's customers and doesn't let them down. I also liked that we were able to talk instead of writing thousands of mails...

Thanks!
Post Reply

Who is online

Users browsing this forum: No registered users and 7 guests