[Wasabi] 403 Access Denied error when using limited IAM policy

[ReKe]

Hello,

I run a backup copy job with object lock to wasabi in v12 without any issues. IAM Policy is configured as mentioned in the follwing KB.

https://www.veeam.com/kb3151

After upgrading backups also runs fine. But if I try to rescan the repository I get the follwing error.

Code: Select all

Failed to synchronize Wasabi Details: S3 error: Access Denied
Code: AccessDenied
Agent failed to process method {Cloud.SetRepositoryOwner}.

Are there any changes needed in the IAM Policy for v12.1?

[mbroaders]

I am seeing the same issue but for clients on v12 when storage management tasks run for any Wasabi storage that is set up with a custom IAM profile. I have an existing ticket open with both Veeam and Wasabi. Veeam case : 07039952. It is proving difficult as Veeam are asking me to talk to Wasabi and Wasabi are asking me to speak to Veeam. From what i can gather so far it seems Veeam is trying to delete items which are still under retention which causes the warning.

Setting the policy to AdministratorAccess stops the warnings from occurring but also gives the user rights to all buckets which is not something we wish to do.

[haslund]

We have posted the new specific permissions required directly in the user guide from 12.1. We'll make sure to update the KB. Please see if https://helpcenter.veeam.com/docs/backu ... positories matches your current policy.

[pirx]

Just switched created our first SOBR with Wasabi bucket as capacity extent - with immutability / object lock. We are mostly using AWS S3. First offload, data is transferred but cleanup fails with 403. Anything obvious that I could try? Should I open a Veeam or Wasabi case?

Code: Select all

13.12.2023 18:22:08 :: Checkpoint cleanup failed Details: REST API error: 'S3 error: Access Denied
Code: AccessDenied', error code: 403
Other: HostId: 'uh7CQ2fDVbf5gdJOxE2m2Lmc0d+cy74RTgMrWMwvXLXV4ygAS2KnD2beVKsjNbcVVX58emKMrOj8', CMReferenceId: 'MTcwMjQ4ODA2ODg4NyAxNTQuNDkuMjE1LjEwNSBDb25JRDozMDA5MzE4NzEvRW5naW5lQ29uSUQ6Mzg1MTQ4MS9Db3JlOjYx'

I already added other arn like arn:aws:s3:::xxx-veeam/* but this makes no difference.

Code: Select all

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "VisualEditor0",
      "Effect": "Allow",
      "Action": [
        "s3:GetBucketLocation",
        "s3:GetObject",
        "s3:PutObject",
        "s3:DeleteObject",
        "s3:GetBucketVersioning",
        "s3:GetBucketObjectLockConfiguration",
        "s3:ListBucketVersions",
        "s3:GetObjectVersion",
        "s3:GetObjectRetention",
        "s3:GetObjectLegalHold",
        "s3:PutObjectRetention",
        "s3:PutObjectLegalHold",
        "s3:DeleteObjectVersion"
      ],
      "Resource": "arn:aws:s3:::xxx-veeam*"
    },
    {
      "Sid": "VisualEditor1",
      "Effect": "Allow",
      "Action": [
        "s3:ListAllMyBuckets",
        "s3:ListBucket"
      ],
      "Resource": "*"
    }
  ]
}

Code: Select all

2023 17:39:17.147] <26> Info         [BackupArchiveIndex] Setting LastCheckpointCleanupDateUtc 13.12.2023 16:39:17 for index [Id: cd6a719d-e1df-4e7f-9f7b-dc62c4c91d19, ArchiveIndexId: 88f9bc4f-0b5d-4483-aa3f-db8134874dbd, BackupId: 8d847350-c305-42d0-b614-165bda938d09, ObjectId: 4302f3c1-4a08-4e92-a9cc-1cbc76b55d48, Path: , LastCheckpointId: 1, IsUpgraded: True, RescanNeeded: False]
[13.12.2023 17:39:17.162] <26> Info         Simple query 'UpdateIncrementUsn' to table [dbo].[Backup.Model.BackupArchiveIndices] [?@id: cd6a719d-e1df-4e7f-9f7b-dc62c4c91d19, @last_checkpoint_cleanup_date_utc: 13.12.2023 16:39:17] executed.
[13.12.2023 17:39:17.162] <26> Info                 [AP] (e34f5eea) command: 'Invoke: ArchRepo.GetCheckpointsForDelete { (EString) ArchRepoId = 32a0cf42-c07d-41a8-8fa0-120589e109ac; (EString) BackupId = a5094b90-013c-426b-bb5b-4b5a06d84f15; (EString) IndexId = 88f9bc4f-0b5d-4483-aa3f-db8134874dbd; (EDateTime) CurrentDate = 2023-12-13 16:39:17; (EBoolean) IncludeLastCheckpoint = false; }'
[13.12.2023 17:39:17.912] <94> Info                   [AP] (e34f5eea) output: <VCPCommandResult result="true" exception="" />
[13.12.2023 17:39:18.022] <83> Info                   [AP] (e34f5eea) output: <VCPCommandArgs><Item key="Checkpoints" type="EUInt64Array" value="0~n" /></VCPCommandArgs>
[13.12.2023 17:39:18.022] <83> Info                   [AP] (e34f5eea) output: >
[13.12.2023 17:39:18.022] <26> Info                 [AP] (e34f5eea) command: 'Invoke: ArchRepo.DeleteCheckpoint { (EString) ArchRepoId = 32a0cf42-c07d-41a8-8fa0-120589e109ac; (EString) BackupId = a5094b90-013c-426b-bb5b-4b5a06d84f15; (EString) IndexId = 88f9bc4f-0b5d-4483-aa3f-db8134874dbd; (EUInt64) CheckpointId = 0; }'
[13.12.2023 17:39:18.241] <92> Info                   [AP] (e34f5eea) output: <VCPCommandResult result="false" exception="REST API error: &apos;S3 error: Access Denied&#x0A;Code: AccessDenied&apos;, error code: 403&#x0A;--tr:Request ID: 437C096BB283EEEF:A&#x0A;Other: HostId: &apos;P+HON/BePlYNZbvgEkGYG6uhdFvWn/QWL35Fv53LJQXq2lYhbZ6lQmJUP6xA1OTtocu7nGoxrKi4&apos;, CMReferenceId: &apos;MTcwMjQ4NTM5NjczMCAxNTQuNDkuMjE1LjEwNiBDb25JRDoxOTQ4ODY0MS9FbmdpbmVDb25JRDoyNTEyOTcvQ29yZTo2Mg==&apos;&#x0A;--tr:CS3VersioningUtils::DeleteFileVersionAsync async task has failed, path [/Veeam/Archive/TR-Manisa-B01/a5094b90-013c-426b-bb5b-4b5a06d84f15/88f9bc4f-0b5d-4483-aa3f-db8134874dbd/checkpoints/checkpoint.0], version [001702484988043393505-mtqR68p0QV]&#x0A;--tr:Failed to DeleteObject from CloudArchRepo, object name [checkpoint.0], backup/index id [a5094b90-013c-426b-bb5b-4b5a06d84f15]/[88f9bc4f-0b5d-4483-aa3f-db8134874dbd], object type [1]&#x0A;--tr:Failed to delete V2 checkpoint&#x0A;--tr:Failed to process method &apos;ArchRepo.DeleteCheckpoint&apos;" />
[13.12.2023 17:39:18.350] <87> Info                   [AP] (e34f5eea) output: >
[13.12.2023 17:39:18.366] <26> Error        REST API error: 'S3 error: Access Denied (Veeam.Backup.Common.CCppComponentException)
[13.12.2023 17:39:18.366] <26> Error        Code: AccessDenied', error code: 403 (Veeam.Backup.Common.CCppComponentException)
[13.12.2023 17:39:18.366] <26> Error           in c++: Request ID: 437C096BB283EEEF:A

[tyler.jurgens]

Just switched created our first SOBR with Wasabi bucket as capacity extent - with immutability / object lock. We are mostly using AWS S3. First offload, data is transferred but cleanup fails with 403.

Try fixing your IAM policy:

Code: Select all

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "VisualEditor0",
      "Effect": "Allow",
      "Action": [
        "s3:GetBucketLocation",
        "s3:GetObject",
        "s3:PutObject",
        "s3:DeleteObject",
        "s3:GetBucketVersioning",
        "s3:GetBucketObjectLockConfiguration",
        "s3:ListBucketVersions",
        "s3:GetObjectVersion",
        "s3:GetObjectRetention",
        "s3:GetObjectLegalHold",
        "s3:PutObjectRetention",
        "s3:PutObjectLegalHold",
        "s3:DeleteObjectVersion"
      ],
      "Resource": "arn:aws:s3:::xxx-veeam/*",
      "Resource": "arn:aws:s3:::xxx-veeam"
    },
    {
      "Sid": "VisualEditor1",
      "Effect": "Allow",
      "Action": [
        "s3:ListAllMyBuckets",
        "s3:ListBucket"
      ],
      "Resource": "*"
    }
  ]
}

[pirx]

Try fixing your IAM policy:

Not much difference than the Resource part as much as I can see. And If I take your code ans safe it, its somehow modified to this. And I still get same error.

I think the issue is with the bucket names. We have many buckets that start with xxx-veeam (xxx = company) and then there are different prefixes. Like xxx-veeam-<location>-<SOBR><region>. Not sure if the Ressource part covers this. But I don't understand why offload is possible just not cleanup if the IAM policy is the issue.

I've now taken the example from https://www.veeam.com/kb3151, lets see if this works with the modified Ressouces.

Edit: still 403

[ReKe]

We have posted the new specific permissions required directly in the user guide from 12.1. We'll make sure to update the KB. Please see if https://helpcenter.veeam.com/docs/backu ... positories matches your current policy.

Hi,

I have compare the old KB to your link for Imuutability enabeld and Helper Apllinaces not used, there are no difference. Is this correct? So looks like we had to open a support case.

[pirx]

Just got feedback from Wasabi support. I'm not happy about this but first lets try it out.

Thank you for reaching out to us. My apologies for delay but I am only now getting to your message.
We started to see this error Access Denied happening recently with our Veeam customers. We believe this has to do with how we are evaluating policies and the situation has been escalated to our Engineering team. The workaround for now would be to use the 'AdministratorAccess' policy in place of the S3 secure policy you had in place for these two buckets. Please try this and let me know the result or if the Admin permission policy does not help. I will make sure to update you on progress of the issue separately as soon as there is new information to share.

[pirx]

Not sure if this is related but I'm on v12 and just added my first Wasabi SOBR buckets and get 403 / AccessDenied during cleanup (offload works). Feedback from Wasabi support:

Code: Select all

Thank you for reaching out to us. My apologies for delay but I am only now getting to your message.
We started to see this error Access Denied happening recently with our Veeam customers. We believe this has to do with how we are evaluating policies and the situation has been escalated to our Engineering team. The workaround for now would be to use the 'AdministratorAccess' policy in place of the S3 secure policy you had in place for these two buckets. Please try this and let me know the result or if the Admin permission policy does not help. I will make sure to update you on progress of the issue separately as soon as there is new information to share.

[massimiliano.rizzi]

Not sure if this is related but I'm on v12 and just added my first Wasabi SOBR buckets and get 403 / AccessDenied during cleanup (offload works). Feedback from Wasabi support

A quick update here to confirm that are in the same boat with several customers on v12 affected and that Wasabi Technical Support has acknowledged this issue has to do with how they are evaluating policies and has been escalated to their Engineering team.

If my understanding is correct, I expect to see this same exact issue on v12.1 as well as it originates on the Wasabi side. Is my understanding correct ?

Apart from the Access Denied errors we are observing with our Veeam customers from within their Veeam Backup & Replication consoles, what are the consequences on the Wasabi S3 buckets side (for example, objects not being deleted and thus additional storage consumption) ?

Thanks !
Massimiliano

[veremin]

Thanks guys for letting us know about the problems experienced with the custom IAM policy. We will re-verify the provided configuration and post back. Thanks!

[massimiliano.rizzi]

Thanks guys for letting us know about the problems experienced with the custom IAM policy. We will re-verify the provided configuration and post back. Thanks!

Hi there Vladimir,

thank you for your reply.

Regarding the problems experienced with the custom IAM policy using Wasabi, we have configurations that have been working for several months now using the same custom IAM policy, and then started showing the same exact issue across different customers over the last few weeks.

As soon as we started seeing this issue lately I felt like it originates on the Wasabi side and not on the Veeam side and that, as a result, I believe the onus is on Wasabi to fix the issue. I would really appreciate if Veeam could kindly put some pressure on Wasabi in order to fix the issue as I expect many Veeam customers in the same boat.

Thanks !
Massimiliano

[veremin]

Understood, we've already shared the detailed information with the Wasabi team regarding the problem (and suspected root cause) and started a joint investigation with them.

We will keep the thread updated with the results of the findings.

Thanks!

[ReKe]

Thanks for all the information here. Adding AdminAcess brings it back to work. But this is relay something that need to be fixed by wasabi soon.

[veremin]

Correct, Wasabi has confirmed the issue is on their side and are planning to fix it. We will post back, once more information is available. Thanks!

[sascha.miloradovic]

Hi all!
I have exactly the same issue across 50-60 customers. Issue happened over night. I have a case open since 15th november (#07013518). Issue appearing on V11 and V12. Same situation like @mbroaders: veeam -> wasabi -> veeam -> wasabi -> veeam and so on.. very tedious. I am happy (but not happy for the customers) to read that I am not the only one having this issue.

[pirx]

Wasabi support gave me information that it should be fixed, but it's still not. Does it work for anyone else without Admin permissions now?

[massimiliano.rizzi]

Wasabi support gave me information that it should be fixed, but it's still not. Does it work for anyone else without Admin permissions now?

Hi there,

we received the reply below from Wasabi Tech Support last Friday regarding the error message:

==================================================
Hi Massimiliano,

The policy issue should be fixed on our end now. Would you mind confirming that you are no longer receiving the '403 Access Denied' errors on the cleanup process?
Regards,

Wasabi Technologies
==================================================

After checking from our end the offload sessions results that came through over the course of the weekend from the various customers I can confirm that we are still seeing the same "Checkpoint cleanup failed Details: REST API error: 'S3 error: Access Denied Code: AccessDenied', error code: 403” as well.

Thanks !

Massimiliano

[veremin]

We'd recommend sharing the same information with the Wasabi support team so that they have more data to rely on during their investigation. Thanks!

[massimiliano.rizzi]

We'd recommend sharing the same information with the Wasabi support team so that they have more data to rely on during their investigation. Thanks!

I sent them earlier this morning in order to share the same information.

Out of curiosity, Has anybody seen any improvements using v12.1 ? If my understanding is correct, I expect to see this same exact issue on v12.1 as well as it originates on the Wasabi side and not on the Veeam side. Is my understanding correct ?

Thanks !

Massimiliano

[sascha.miloradovic]

Wasabi support gave me information that it should be fixed, but it's still not. Does it work for anyone else without Admin permissions now?

It is still not fixed. Still same error.

[pirx]

We are new to Wasabi and it feels a bit that they are not really testing anything.

[sascha.miloradovic]

@prix
We have nearly a PB of data in wasabi across 70-80 customers. In the last two years, there have been no major issues I can think of. This is the first major issue which takes > 1 month to resolve. Let's see how long they will take to resolve this issue.

[massimiliano.rizzi]

Hi there,

I've just received an update from Wasabi Technical Support. They are still investigating the issue and will provide an update on this matter as soon as new information is available.

Regards,

Massimiliano

[chrisWasabi]

Hi All,

Our engineering team has been able to replicate this issue and is working on a fix. It is being loaded into QA for verification as I type this. We hope to have a fix deployed soon for it. There will be more details to come as the fix is verified. We understand this is affecting your backups and are going to do our best to resolve it ASAP. Thank you all for your patience.

-ChrisWasabi

[veremin]

Thank you, Chris, for chiming in and updating the thread with the results of your team's findings. Appreciate the help and assistance provided.

[QSI_Team]

Any update on this? We are either going to have to allow the admin access or increase our SOBR limits in all our customer's Veeam servers since without the offloads they are expanding rapidly. Is Wasabi going to reimburse us for all the extra space that is being taken up that is due to their API issue?

[pirx]

Wasabi support last updated my case on 21.12. Nothing new since then.

[pirx]

I asked for an update and there is still no solution and AdministratorAccess is the workaround. This is very unfortunate as this violates our security policy and as we just started to migrate to Wasabi it creates certain questions internally.

[AlexHeylin]

@pirx - I understand the concern, but better a good QA process which is slower in December due to the holidays etc, than a rushed "fix" that potentially causes another problem. That should be a reasonable explanation to address those concerns internally.