AWS S3 Immutability (Object Lock) error

[tpx]

I was receiving the following error when selecting the Make recent backups immutable checkbox:
"Failed to enable backup immutability: the selected object storage does not support S3 Object Lock feature"

I found the solution in Veeam's documentation:
https://helpcenter.veeam.com/docs/backu ... ml?ver=100

When enabling Object Lock on an S3 bucket, use the None option for the object lock configuration mode. Otherwise, you will not be able to register the bucket with Veeam Backup & Replication. Note that Veeam Backup & Replication will automatically use Compliance object lock mode for each uploaded object.

I had previously selected 'Compliance mode' on the S3 bucket when enabling Object Lock.

It's working now after I changed it to the None option.

[dalbertson]

I will perform some testing in my lab. But there are two ways to create a bucket with object lock. One way is doing it while you are creating the bucket by enabling versioning and then adding the checkbox for object lock. The second way is after the bucket is created and going to the properties page. I am thinking that the issue appears when the second way is being selected.

[robert.vonmehren]

Just to add - I am seeing the same issue with a customer.

Versioning and object lock were enabled and set to compliance mode - this resulted in an error that object lock was not supported for this bucket

We tried with a new bucket with object lock set to "none" and resulted in the same error. Will post support case # when opened

[amarsaudon]

Did you ever get this sorted? Experiencing the same thing, figured I'd ask before opening a ticket.

[dalbertson]

https://helpcenter.veeam.com/docs/backu ... ml?ver=100

When enabling Object Lock on an S3 bucket, use the None option for the object lock configuration mode. Otherwise, you will not be able to register the bucket with Veeam Backup & Replication. Note that Veeam Backup & Replication will automatically use Compliance object lock mode for each uploaded object.

[mrt]

When enabling Object Lock on an S3 bucket, use the None option for the object lock configuration mode.

Struggling with this error. Did AWS change the wording on the bucket creation form? When I create a bucket and enable Object Lock, I'm given no options at all to set "configuration mode" nor is that setting available after bucket creation. Thanks

[gareauk]

I had the same issue, after speaking with our Veeam rep, we found out the Veeam IAM user was missing some permissions. The error went away once we corrected the permissions.

https://helpcenter.veeam.com/docs/backu ... ml?ver=110

[Andreas Neufert]

Thanks for sharing this Kelly

[MrC]

Case #05422675 — Issues adding S3 repository

This subject is very confusing and mostly due to the poor and contradictory documentation provided by Veeam.

This article lists different IAM permissions then the one Kelly linked above.
https://www.veeam.com/kb3151

Which is correct? When you try and paste the JSON code from the article Kelly shared you cant even browse the bucket.

When I use the below JSON code I can see the bucket but not the folder I created.


{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "VisualEditor0",
"Effect": "Allow",
"Action": [
"s3:GetObjectRetention",
"s3:DeleteObjectVersion",
"s3:ListBucketVersions",
"s3:ListBucket",
"s3:PutObjectLegalHold",
"s3:GetBucketVersioning",
"s3:GetObjectLegalHold",
"s3:GetBucketObjectLockConfiguration",
"s3:PutObject",
"s3:GetObject",
"s3:PutObjectRetention",
"s3:DeleteObject",
"s3:GetBucketLocation",
"s3:GetObjectVersion"
],
"Resource": [
"arn:aws:s3:::bucket-name",
"arn:aws:s3:::bucket-name/*"
]
},
{
"Sid": "VisualEditor1",
"Effect": "Allow",
"Action": [
"s3:ListAllMyBuckets",
"s3:ListBucket"
],
"Resource": "*"
}
]
}

Something that seems so simple and easy has so many articles that contradict each other.

So you create the bucket with:
- Bucket Versioning Enabled
- Block all public access enabled (on)
- Object lock enabled
- default retention disabled

Are suppose to use Bucket Policies and if so what are the permissions?

[HannesK]

Hello,
and welcome to the forums.

Something that seems so simple and easy

when I started with AWS, that IAM part was relatively difficult for me to understand

so many articles that contradict each other.

just to confirm: Is that statement about the two documents (helpcenter and KB article), or more? The reason for the two is, that "experts" prefer the "pure information" and people like me (who deal with AWS only from time to time) prefer more "hand on" (the KB article).

So you create the bucket with:
- Bucket Versioning Enabled
- Block all public access enabled (on)
- Object lock enabled
- default retention disabled

correct

Best regards,
Hannes

[sti]

https://helpcenter.veeam.com/docs/backu ... ml?ver=100

When enabling Object Lock on an S3 bucket, use the None option for the object lock configuration mode. Otherwise, you will not be able to register the bucket with Veeam Backup & Replication. Note that Veeam Backup & Replication will automatically use Compliance object lock mode for each uploaded object.

I did that, and I was able to register the S3 bucket with Veeam Backup & Replication. I created a scale-out repository and added this bucket in the capacity tier of it , I configured the bucket with 'object locking enabled' , retention mode 'none' , instead of the non operable compliance/governance retention mode where I failed to register with veeam B&R
Note : Underneath was carried out in a LAB on Scality RING 8.5, (don't do what I did this in your production infra)

I tried to delete the Backupjob from within Veeam B&R and due to the job setting that I have retention in place, I could proceed with the job deletion, but I got orphaned backups.
Lateron I connected an S3 browser to the bucket and with the Bucket Owner account (Access Token/Password S3 AWS IAM identity) I was able to access my bucket and wipe out my backup data despite Veeams retention policy in place.

Why is that you might think. When you open the bucket contents (with e.g. an S3 browser) and login with the owner identity, you will spot that the owner of the bucket has full control ! If you login with these credentials, you can fully delete 'immutable objects' . So there is definately a whole lot more to it then what I have been reading over here.

[Mildur]

Hi Stefan

The objects are not gone. They should have a deletion marker on it which can be removed. Then the objects will be visible again.
You should ask your object storage vendor for such a script to remove the deletion marker on the objects.

Thanks
Fabian

[sti]

You definately need a script there to retrieve every single Object IDs (from files to folders, the whole tree in fact ) Veeam keeps in your bucket.
After reading what deletion markers are, we are only then recovering the last version of those Object IDs you reference. I suppose the script comes down to retrieve a full list of object IDs and then to remove the deletion marker on each of them.

[veremin]

You can find more information regarding the delete marker removal script here. And the good news is in version 12 you will not need to run the script - the backup server will be able to restore data even from objects with delete markers. Thanks!

[sti]

Just want to add , that once you remove the bucket 'root' folder , any subsequent backups remain 'invisible' too.
So for as long as long as your deletion marker is in place on your bucket root , any new subsequent backups beyond the deletion , even though appearing as restore point
cannot be restored , Veeam Backup and replication gives you the following error message :

REST API error : S3 error 'The specified key does not exist'
Code : No suchkey , error code 404

So once your bucket 'root' has been deleted, you are forced to figure out how to script around that problem and regain proper access to your data.
We downloaded Veeam v12 beta. Can a Veeam Moderator figure out if the Veeam 12 beta is already operational at this level ?

[veremin]

What exactly do you mean by operational? Can it be used reliably in production environments? No, as the product is still in the beta stage, during which we're trying to spot and fix technical and logical issues based on the received feedback. Thanks!

[sti]

What i meant was, is Veeam B+R v12 able to restore objects that are hidden due to the deletion marker ?
Has one tested that in a lab with Veeam B+R v12 ?
I got a script from the vendor, ran it but kept on getting
REST API error : S3 error 'The specified key does not exist'
Code : No suchkey , error code 404
On both v11 & v12beta at that time.

[veremin]

Confirmed, the v12 should be able to initiate restores from objects having delete markers assigned (without the need to run the script first). If you are already on the RTM build and having issues with the said scenario, kindly reach our support team directly. Thanks!