SOBR Wasabi Issues

[lukejf]

Hi All,
We have been happily using wasabi for a while now and have recently tried their object lock which appears to be working well. Then suddenly we started getting the following error.

Rest API error: S3 error: Access deniend.

I thought it may have been an issue with our polices so set all our polices to S3Full access yet still get the error. Any ideas?

[PetrM]

Hi Luke,

Please share a support case ID for this issue as requested when you click New Topic. Also, it would probably make sense to contact storage vendor support as well, especially if the error suddenly appeared without any kind of changes on Veeam side.

Thanks!

[GreenNom]

Not to hijack this, but I have the same issue and have an open case with Wasabi and just opened a case with Veeam #04824320 at Wasabi's request.

Everything was working well for a couple of days but now I can't create SOBR repos with encryption enabled as I received the Access Denied error, and if I don't enable encryption the Repo will create but the backup Offload will fail with the same error.
Also existing jobs that were ticking along nicely all fail the offload with the Access Denied error now.

[NightBird]

Same error than here ?
object-storage-f52/scaleway-s3-object-s ... 73619.html

« 10.05.2021 12:23:43 :: REST API error: 'S3 error: COMPLIANCE mode, override denied
Code: AccessDenied', error code: 403 »

[veremin]

We have been happily using wasabi for a while now and have recently tried their object lock which appears to be working well. Then suddenly we started getting the following error.

Can you tell us how you have enabled object lock (created a bucket, enabled object lock and versioning, etc.) and how you have then added this entity to backup console? Also, can you share a ticket ID with us, so we can bring the issue internally?

Not to hijack this, but I have the same issue and have an open case with Wasabi and just opened a case with Veeam #04824320 at Wasabi's request.

Thank you for sharing the case number - we will investigate the problem.

[GreenNom]

Same error than here ?

« 10.05.2021 12:23:43 :: REST API error: 'S3 error: COMPLIANCE mode, override denied
Code: AccessDenied', error code: 403 »

The error I've been getting doesn't say anything regarding Compliance mode, just the Access Denied.

Setup of the bucket was as described in the Wasabi KB https://wasabi-support.zendesk.com/hc/e ... 0059655831

Bucket was added to Veeam as a S3 compatible storage as described in the rest of Wasabi KB https://wasabi-support.zendesk.com/hc/e ... ntegration

[bytewiseits]

We have also been testing Object Lock Immutability with Wasabi (us-west-1 region) from our lab VBR and it had been working very well for a month up until last Friday [21/05/2021] when the uploads just stopped all of a sudden (coincidentally at the same time as they had an outage). After the outage was resolved we then got 'REST API error: 'S3 error: Access Denied, Code: AccessDenied', error code: 403' whenever trying to offload. We hadn't made any changes on the Veeam/VBR side of things or in the Wasabi Console.

Being an S3 based error I waited a few days to see if it would maybe resolve and also tried more permissive policies (all s3 permissions) but still the same currently so have just opened up a ticket with Wasabi. Incidentally they just specifically asked us if encryption is enabled and object lock used of which is yes to both. (bucket was created with object versioning/object lock enabled, then object repository created with immutability set to 3 days)

Interestingly though, Veeam can still read/download data from the capacity tier without issue (put performance tier in maintenance then force restore from S3) so it does have read access at least.

On our production VBR we still have an SOBR happily offloading to Wasabi us-west-1 region daily with a similar setup, however it does not use object lock at all, only encryption on the object repo.

(sorry to thread jack but it does seems related - hopefully its just a small fixable glitch )

[veremin]

Dion, and what was the ticket where the issue was investigated - should help us to collect as much information about the root cause as possible? Thanks!

[bytewiseits]

Hi Vladamir, I have not long opened up ticket 44574 with Wasabi being that it was an S3 based error we were seeing. I haven't logged a ticket with Veeam yet as it seemed to be a storage provider/S3 based issue and it is just my test lab VBR affected. I can raise a Veeam ticket if you wish, but was just waiting to hear back from Wasabi if they found anything on their end of things first before annoying your support team.

[veremin]

Since several users have reported the similar issue already, we would also like to participate in the investigation process. To do this proficiently we need to gather more information. Having opened ticket with debug logs attached will surely help us to speed up the process.

So, we would appreciate, if you reach our support team and share the ticket number, once it's opened.

Thanks!

[shukerr]

Same issue here. All went wrong on the 21/05/2021 like the other comments.

[veremin]

And like I mentioned above, we would be glad, if you can submit the support case for this issue Thanks!

[dalbertson]

Hi all, as @veremin has stated, please open a case with wasabi support and Veeam support. I am working directly with Wasabi and they are aware of some reasoning for the issue, but instead of just assuming its all the same issue we will look into them each to make sure.

Thank you.

[lukejf]

Hi All i have a work around from Wasabi support which i have confirmed resolved my issue. We are using non root keys.

`````````````````````````````````````````````````````````````````````````````````````````````

If you are using ROOT API keys for the user and plan to use the same keys for your subsequent jobs, then do not need any changes done on the IAM end.

If you are using NON-ROOT API keys for your sub-user, you may
Change the "Action": "s3:*", to "Action": "*", ., since we have a resource tied to that action it would still be applied to just that bucket.
For instance, here's how your policy should look like wherein you are allowing ALL actions for that particularly defined bucket(s).

{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": "s3:ListAllMyBuckets",
"Resource": "arn:aws:s3:::*"
},
{
"Effect": "Allow",
"Action": "*",
"Resource": [
"arn:aws:s3:::bucket-name-here",
"arn:aws:s3:::bucket-name-here/*"
]
}
]
}

Thanks Heaps Wasabi support!

[bytewiseits]

Can confirm changing the Allow Action from 's3:*' to just '*' is allowing offloads with object lock to work again for us too. Seems like Wasabi made an undocumented change last Friday. Will now wait and see what their support comes back with in terms of the root cause.

[lukejf]

Correct, however it is still Beta right?

[bytewiseits]

Seems to depend on whether you ask Wasabi marketing or technical departments as to what answer you get about the beta status.
Marketing let me know last week with much fanfare that object lock support had been officially announced and there were webinars etc, however in our production Wasabi account I see that object lock is still not available there. (we use a separate Wasabi account for our lab/test VBR which was in the Wasabi OL beta)

My guess is that they are still in the process of launching it and rolling it out to all accounts so shouldn't be too far off when marketing and technical can both agree again

[SchaeferTWS]

Hi All i have a work around from Wasabi support which i have confirmed resolved my issue. We are using non root keys.

`````````````````````````````````````````````````````````````````````````````````````````````

If you are using ROOT API keys for the user and plan to use the same keys for your subsequent jobs, then do not need any changes done on the IAM end.

If you are using NON-ROOT API keys for your sub-user, you may
Change the "Action": "s3:*", to "Action": "*", ., since we have a resource tied to that action it would still be applied to just that bucket.
For instance, here's how your policy should look like wherein you are allowing ALL actions for that particularly defined bucket(s).

That did the trick for us - even if we are using the ROOT API key.
Works like a charm now, thanks!

[shukerr]

Nice one! I'm using non-root keys also & changing "S3:*" to "*" has fixed it for me also.
Let's wait for the RCA from Wasabi and see if this change is going to be permanent and any implications of the change.

[veremin]

Sometimes ago we described a secure IAM policy for connection to s3 object storage. The policy consists of necessary permissions, including ones needed for creating and retaining object (s3: PutObject and s3:PutObjectRetention).

These APIs do not seem to work correctly in case of immutable Wasabi storage.

So to work around the issue, you need to allow all s3 actions in the IAM policy – see Luke’s response.

In the meantime we are opening a case with Wasabi to report this behavior.

Thanks!

[cwhitehbllp]

Another confirmation changing from S3: to * has resolved the issues.

[AlexHeylin]

I've just had notification from Wasabi:
===
Hi Team -

We have fixed the IAM interaction issue that was earlier notified to you for Object Lock bucket(s) using NON-ROOT Keys. The new change and code has been deployed successfully. Please go ahead and use the appropriate IAM Policy for your sub-user to give them correct permissions necessary to be able to access the S3 to save/load data to/from Wasabi bucket that has Object Lock feature enabled to it. You may use the minimal permissions rule and create your own IAM policy and attach to your sub-user(s) if you wish and as per your requirements. One example of how this would look like is shown below:

Code: Select all

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "VisualEditor0",
      "Effect": "Allow",
      "Action": [
        "s3:ListBucket",
        "s3:GetBucketLocation",
        "s3:GetObject",
        "s3:PutObject",
        "s3:DeleteObject",
        "s3:GetBucketVersioning",
        "s3:GetBucketObjectLockConfiguration",
        "s3:ListBucketVersions",
        "s3:GetObjectVersion",
        "s3:GetObjectRetention",
        "s3:GetObjectLegalHold",
        "s3:PutObjectRetention",
        "s3:PutObjectLegalHold",
        "s3:DeleteObjectVersion"
      ],
      "Resource": [
        "arn:aws:s3:::my-bucket/*",
        "arn:aws:s3:::my-bucket"
      ]
    },
    {
      "Sid": "VisualEditor1",
      "Effect": "Allow",
      "Action": "s3:ListAllMyBuckets",
      "Resource": "*"
    }
  ]
}

NOTE: Please be sure to use your own bucket-names in the policy in place of 'my-bucket' shown above.

Please go ahead and test this change and send us a note of what you see on your end. Once again, I really appreciate your patience and cooperation with this matter. Thanks

[AlexHeylin]

For what it's worth - this same fix from Wasabi has also addressed the occasional 403 errors we were seeing when using a non-root user with an old mutable (non-object-lock) S3 repo as capacity tier. That's not a known or intended effect - but that's what seems to have happened.