IAM JSON for AWS S3 Immutable (Object-Lock)

[Skyview]

Hello,

I see there is an existing KB that details a least-priviledge access IAM policy for Veeam S3 Object storage here:
https://www.veeam.com/kb3033

However, that does not include the new S3 permissions needed to do object-lock (immutablity features). Is there a KB with the permissions for that ability?

I see there are a few to pick from, such as
s3:GetObjectLegalHold
s3:PutObjectLegalHold
s3:BypassGovernanceRetention
s3:GetObjectRetention
s3:PutObjectRetention

to name a few. I want to only include the necessary permissions for Veeam to work. Can you please outline what are the minimum S3 permissions needed for Immutable AWS S3 Object Storage please?

[dalbertson]

You will need to add

s3:GetObjectRetention
s3:PutObjectRetention

We use compliance mode so there will be non bypass and legal hold is not used.

I will get a kB issued for the new additions

[Skyview]

So the JSON would look like this then? And also, is that setting (compliance mode vs governance mode) something you have to manually set on your bucket in the console? Thanks, I'll hang up and listen

Code: Select all

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "s3:PutObject",
                "s3:GetObject",
                "s3:DeleteObject"
                "s3:GetObjectRetention"
                "s3:PutObjectRetention"
            ],
            "Resource": "arn:aws:s3:::<yourbucketname>/*"
        },
        {
            "Effect": "Allow",
            "Action": [
                "s3:ListBucket",
                "s3:GetBucketLocation"
            ],
            "Resource": "arn:aws:s3:::<yourbucketname>"
        },
        {
            "Effect": "Allow",
            "Action": "s3:ListAllMyBuckets",
            "Resource": "*"
        }
    ]
}

[dalbertson]

Actually here is a post where I put a sample policy in. Just add

s3:GetObjectRetention
s3:PutObjectRetention

To the upper section and it should be golden

[Skyview]

Not working for me.

Still get an error:

"Failed to enable backup immutability: the selected object storage does not support S3 Object Lock feature"

Is there a global bucket permission that needs set, not just the get/put?

[dalbertson]

Did you not enable object lock on the bucket?

When creating a bucket you need to enable versioning and object lock

[chris.arceneaux]

The policy below will work. Please note this policy assumes AWS KMS encryption is not being used. Added permissions for KMS would be required if that was the case.

Code: Select all

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "VisualEditor0",
            "Effect": "Allow",
            "Action": [
                "s3:GetObjectRetention",
                "s3:ListBucketVersions",
                "s3:ListBucket",
                "s3:PutObjectLegalHold",
                "s3:GetBucketVersioning",
                "s3:GetObjectLegalHold",
                "s3:GetBucketObjectLockConfiguration",
                "s3:PutObject",
                "s3:GetObject",
                "s3:GetEncryptionConfiguration",
                "s3:PutBucketObjectLockConfiguration",
                "s3:DeleteObject",
                "s3:GetBucketLocation"
            ],
            "Resource": [
                "arn:aws:s3:::<yourbucketname>/*",
                "arn:aws:s3:::<yourbucketname>"
            ]
        },
        {
            "Sid": "VisualEditor1",
            "Effect": "Allow",
            "Action": [
                "s3:ListAllMyBuckets",
                "s3:HeadBucket"
            ],
            "Resource": "*"
        }
    ]
}

[Skyview]

It's enabled. It's definitely something with the bucket settings, because when I enable all bucket-level permissions it works, when I only have:
"Effect": "Allow",
"Action": [
"s3:ListBucket",
"s3:GetBucketLocation"
],
"Resource": "arn:aws:s3:::<yourbucketname>"
},

it does not work.

[Skyview]

The policy below will work. Please note this policy assumes AWS KMS encryption is not being used. Added permissions for KMS would be required if that was the case.

Code: Select all

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "VisualEditor0",
            "Effect": "Allow",
            "Action": [
                "s3:GetObjectRetention",
                "s3:ListBucketVersions",
                "s3:ListBucket",
                "s3:PutObjectLegalHold",
                "s3:GetBucketVersioning",
                "s3:GetObjectLegalHold",
                "s3:GetBucketObjectLockConfiguration",
                "s3:PutObject",
                "s3:GetObject",
                "s3:GetEncryptionConfiguration",
                "s3:PutBucketObjectLockConfiguration",
                "s3:DeleteObject",
                "s3:GetBucketLocation"
            ],
            "Resource": [
                "arn:aws:s3:::<yourbucketname>/*",
                "arn:aws:s3:::<yourbucketname>"
            ]
        },
        {
            "Sid": "VisualEditor1",
            "Effect": "Allow",
            "Action": [
                "s3:ListAllMyBuckets",
                "s3:HeadBucket"
            ],
            "Resource": "*"
        }
    ]
}

This worked (after plugging in my specific bucket name, of course).

But can we get a KB from Veeam please so we have an officially supported config?

[Skyview]

BTW, thank you Chris. You saved me a lot of trial & error removing permissions one by one.

[dalbertson]

I will get a KB created.

[Skyview]

And this is why Veeam is the best.

[chris.arceneaux]

No problem. Glad I could help!

[Skyview]

Ran into another issue I believe is with permissions. I was able to add the AWS S3 Object Repository, however when I include it when making a SOBR, I get this:

Failed to save scale-out backup repository:
Unable to create database records for repository
Amazon REST error: 'S3 error: Access Denied
Code: AccessDenied', error code: 403
Other: HostId: (**REDACTED FOR FORUM POST**)

Amazon REST error: 'S3 error: Access Denied

[Skyview]

Here is the JSON I have currently:

Code: Select all

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "VisualEditor0",
            "Effect": "Allow",
            "Action": [
                "s3:GetObjectRetention",
                "s3:ListBucketVersions",
                "s3:ListBucket",
                "s3:PutObjectLegalHold",
                "s3:GetBucketVersioning",
                "s3:GetObjectLegalHold",
                "s3:GetBucketObjectLockConfiguration",
                "s3:PutObject",
                "s3:GetObject",
                "s3:GetEncryptionConfiguration",
                "s3:PutBucketObjectLockConfiguration",
                "s3:DeleteObject",
                "s3:GetBucketLocation"
            ],
            "Resource": [
                "arn:aws:s3:::bucketnameredacted/*",
                "arn:aws:s3:::butcketnameredacted"
            ]
        },
        {
            "Sid": "VisualEditor1",
            "Effect": "Allow",
            "Action": [
                "s3:ListAllMyBuckets",
                "s3:HeadBucket"
            ],
            "Resource": "*"
        }
    ]
}

[Skyview]

More information on this:

It only errors when I select "Encrypt data uploaded to object storage" option. If I uncheck that, it proceeds normally.

[Skyview]

More More information:
SOBR tiering failed anyways. Similar error:

3/5/2020 3:51:35 PM :: Amazon REST error: 'S3 error: Access Denied
Code: AccessDenied', error code: 403
Other: HostId: (***redacted for forum post***)

[dalbertson]

@Skyview with or without encrpytion....also where are you enabling encryption...veeam or AWS?

[Skyview]

If I check the "Encrypt data uploaded to object storage" option, adding the AWS S3 Object Repo to the SOBR fails.

If I leave it unchecked, it will proceed and add it to the SOBR.

However, SOBR tiering will fail.

I think it's because when you check encryption when adding the Object Repo to SOBR, it tries to write something, hence the failure on add.

But with no encryption, there is no write operation- until SOBR tiering, at which point that fails.

So I still think it's IAM permissions is missing something.

[Skyview]

@Skyview with or without encrpytion....also where are you enabling encryption...veeam or AWS?

veeam

[chris.arceneaux]

I can confirm some of this behavior as I did not have Veeam encryption enabled in my lab. Please leave me some time to further test and I'll report back.

SOBR offloads have been succeeding in my lab though.

[dalbertson]

Open a case on this with support. I want to track this. Encryption from veeam shouldn’t matter with permissions on S3 as it doesn’t change the API calls we are making to S3 since it’s not bucket encryption.

[chris.arceneaux]

I've gotten to the bottom of things. Here's the updated policy:

Code: Select all

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "VisualEditor0",
            "Effect": "Allow",
            "Action": [
                "s3:GetObjectRetention",
                "s3:ListBucketVersions",
                "s3:ListBucket",
                "s3:PutObjectLegalHold",
                "s3:GetBucketVersioning",
                "s3:GetObjectLegalHold",
                "s3:GetBucketObjectLockConfiguration",
                "s3:PutObject*",
                "s3:GetObject*",
                "s3:GetEncryptionConfiguration",
                "s3:PutObjectRetention",
                "s3:PutBucketObjectLockConfiguration",
                "s3:DeleteObject*",
                "s3:GetBucketLocation"
            ],
            "Resource": [
                "arn:aws:s3:::<yourbucketname>/*",
                "arn:aws:s3:::<yourbucketname>"
            ]
        },
        {
            "Sid": "VisualEditor1",
            "Effect": "Allow",
            "Action": [
                "s3:ListAllMyBuckets",
                "s3:HeadBucket"
            ],
            "Resource": "*"
        }
    ]
}

[Skyview]

Working! After using the above policy posted by chris (and of course plugging in the correct bucketname) I was able to enable encryption and SOBR tiering has now kicked off and is seeding. It might take a bit for the initial seed to complete, I'll report back on results. Thank you! (And lets get this typed up into official KB/howto)

[Skyview]

Update:


SOBR Tiering finished! Thanks Chris!

[chris.arceneaux]

Great to hear. Thanks for letting us know!

[tpx]

I'm getting the same message: "Failed to enable backup immutability: the selected object storage does not support S3 Object Lock feature"

I've tried the updated policy from chris.arceneaux. Also, tried an IAM policy with full administrative access. No luck so far.

Confirmed that the S3 bucket has Object Lock set to Compliance mode.

[Gostev]

If you tried an IAM policy with full administrative access, then your issue is completely unrelated to this discussion, so let's not derail or hi-jack this topic. Please open a support case, and create the dedicated topic (if you feel your issue needs to be discussed with the entire community). Thanks!

[Skyview]

Did you make sure to edit the policy and change "yourbucketname" to the bucket name you're using? Does your bucket have Versioning enabled?

Just another update- SOBR tiering has been working successfully for several days now!

Are we sure this is the Least Privilege Access though, and there's no unnecessary permissions in this policy?

Thanks again!

[chris.arceneaux]

Are we sure this is the Least Privilege Access though, and there's no unnecessary permissions in this policy?

This is being further investigated to double-check but, as of now, this is the minimal permissions policy. If the IAM policy gets further reduced, we'll follow up here.