Discussions related to using object storage as a backup target.
Post Reply
tpx
Novice
Posts: 7
Liked: 2 times
Joined: Mar 08, 2020 9:26 am
Contact:

AWS S3 Immutability (Object Lock) error

Post by tpx » 1 person likes this post

I was receiving the following error when selecting the Make recent backups immutable checkbox:
"Failed to enable backup immutability: the selected object storage does not support S3 Object Lock feature"

I found the solution in Veeam's documentation:
https://helpcenter.veeam.com/docs/backu ... ml?ver=100

When enabling Object Lock on an S3 bucket, use the None option for the object lock configuration mode. Otherwise, you will not be able to register the bucket with Veeam Backup & Replication. Note that Veeam Backup & Replication will automatically use Compliance object lock mode for each uploaded object.

I had previously selected 'Compliance mode' on the S3 bucket when enabling Object Lock.

It's working now after I changed it to the None option.
dalbertson
Veeam Software
Posts: 492
Liked: 175 times
Joined: Jul 21, 2015 12:38 pm
Full Name: Dustin Albertson
Contact:

Re: AWS S3 Immutability (Object Lock) error

Post by dalbertson »

I will perform some testing in my lab. But there are two ways to create a bucket with object lock. One way is doing it while you are creating the bucket by enabling versioning and then adding the checkbox for object lock. The second way is after the bucket is created and going to the properties page. I am thinking that the issue appears when the second way is being selected.
Dustin Albertson | Director of Product Management - Cloud & Applications | Veeam Product Management, Alliances
robert.vonmehren
Veeam Software
Posts: 157
Liked: 12 times
Joined: Nov 07, 2018 3:44 pm
Contact:

Re: AWS S3 Immutability (Object Lock) error

Post by robert.vonmehren »

Just to add - I am seeing the same issue with a customer.

Versioning and object lock were enabled and set to compliance mode - this resulted in an error that object lock was not supported for this bucket

We tried with a new bucket with object lock set to "none" and resulted in the same error. Will post support case # when opened
amarsaudon
Lurker
Posts: 2
Liked: never
Joined: Mar 10, 2020 5:05 pm
Full Name: Alex Marsaudon
Contact:

Re: AWS S3 Immutability (Object Lock) error

Post by amarsaudon »

Did you ever get this sorted? Experiencing the same thing, figured I'd ask before opening a ticket.
dalbertson
Veeam Software
Posts: 492
Liked: 175 times
Joined: Jul 21, 2015 12:38 pm
Full Name: Dustin Albertson
Contact:

Re: AWS S3 Immutability (Object Lock) error

Post by dalbertson » 3 people like this post

https://helpcenter.veeam.com/docs/backu ... ml?ver=100

When enabling Object Lock on an S3 bucket, use the None option for the object lock configuration mode. Otherwise, you will not be able to register the bucket with Veeam Backup & Replication. Note that Veeam Backup & Replication will automatically use Compliance object lock mode for each uploaded object.
Dustin Albertson | Director of Product Management - Cloud & Applications | Veeam Product Management, Alliances
mrt
Enthusiast
Posts: 53
Liked: 2 times
Joined: Feb 10, 2011 7:27 pm
Contact:

Re: AWS S3 Immutability (Object Lock) error

Post by mrt »

When enabling Object Lock on an S3 bucket, use the None option for the object lock configuration mode.
Struggling with this error. Did AWS change the wording on the bucket creation form? When I create a bucket and enable Object Lock, I'm given no options at all to set "configuration mode" nor is that setting available after bucket creation. Thanks
gareauk
Lurker
Posts: 1
Liked: 1 time
Joined: Dec 09, 2021 7:19 pm
Full Name: Kelly Gareau
Contact:

Re: AWS S3 Immutability (Object Lock) error

Post by gareauk » 1 person likes this post

I had the same issue, after speaking with our Veeam rep, we found out the Veeam IAM user was missing some permissions. The error went away once we corrected the permissions.

https://helpcenter.veeam.com/docs/backu ... ml?ver=110
Andreas Neufert
VP, Product Management
Posts: 6707
Liked: 1401 times
Joined: May 04, 2011 8:36 am
Full Name: Andreas Neufert
Location: Germany
Contact:

Re: AWS S3 Immutability (Object Lock) error

Post by Andreas Neufert »

Thanks for sharing this Kelly
MrC
Lurker
Posts: 1
Liked: never
Joined: Oct 18, 2013 2:44 pm
Full Name: Chris Currell
Contact:

Re: AWS S3 Immutability (Object Lock) error

Post by MrC »

Case #05422675 — Issues adding S3 repository

This subject is very confusing and mostly due to the poor and contradictory documentation provided by Veeam.

This article lists different IAM permissions then the one Kelly linked above.
https://www.veeam.com/kb3151

Which is correct? When you try and paste the JSON code from the article Kelly shared you cant even browse the bucket.

When I use the below JSON code I can see the bucket but not the folder I created.


{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "VisualEditor0",
"Effect": "Allow",
"Action": [
"s3:GetObjectRetention",
"s3:DeleteObjectVersion",
"s3:ListBucketVersions",
"s3:ListBucket",
"s3:PutObjectLegalHold",
"s3:GetBucketVersioning",
"s3:GetObjectLegalHold",
"s3:GetBucketObjectLockConfiguration",
"s3:PutObject",
"s3:GetObject",
"s3:PutObjectRetention",
"s3:DeleteObject",
"s3:GetBucketLocation",
"s3:GetObjectVersion"
],
"Resource": [
"arn:aws:s3:::bucket-name",
"arn:aws:s3:::bucket-name/*"
]
},
{
"Sid": "VisualEditor1",
"Effect": "Allow",
"Action": [
"s3:ListAllMyBuckets",
"s3:ListBucket"
],
"Resource": "*"
}
]
}

Something that seems so simple and easy has so many articles that contradict each other.

So you create the bucket with:
- Bucket Versioning Enabled
- Block all public access enabled (on)
- Object lock enabled
- default retention disabled

Are suppose to use Bucket Policies and if so what are the permissions?
HannesK
Product Manager
Posts: 14287
Liked: 2877 times
Joined: Sep 01, 2014 11:46 am
Full Name: Hannes Kasparick
Location: Austria
Contact:

Re: AWS S3 Immutability (Object Lock) error

Post by HannesK »

Hello,
and welcome to the forums.
Something that seems so simple and easy
when I started with AWS, that IAM part was relatively difficult for me to understand :-)
so many articles that contradict each other.
just to confirm: Is that statement about the two documents (helpcenter and KB article), or more? The reason for the two is, that "experts" prefer the "pure information" and people like me (who deal with AWS only from time to time) prefer more "hand on" (the KB article).
So you create the bucket with:
- Bucket Versioning Enabled
- Block all public access enabled (on)
- Object lock enabled
- default retention disabled
correct

Best regards,
Hannes
sti
Novice
Posts: 8
Liked: never
Joined: Oct 05, 2022 7:37 am
Full Name: Stefan Timmermans
Contact:

Re: AWS S3 Immutability (Object Lock) error

Post by sti »

dalbertson wrote: Jun 29, 2020 6:42 pm https://helpcenter.veeam.com/docs/backu ... ml?ver=100

When enabling Object Lock on an S3 bucket, use the None option for the object lock configuration mode. Otherwise, you will not be able to register the bucket with Veeam Backup & Replication. Note that Veeam Backup & Replication will automatically use Compliance object lock mode for each uploaded object.
I did that, and I was able to register the S3 bucket with Veeam Backup & Replication. I created a scale-out repository and added this bucket in the capacity tier of it , I configured the bucket with 'object locking enabled' , retention mode 'none' , instead of the non operable compliance/governance retention mode where I failed to register with veeam B&R
Note : Underneath was carried out in a LAB on Scality RING 8.5, (don't do what I did this in your production infra)

I tried to delete the Backupjob from within Veeam B&R and due to the job setting that I have retention in place, I could proceed with the job deletion, but I got orphaned backups.
Lateron I connected an S3 browser to the bucket and with the Bucket Owner account (Access Token/Password S3 AWS IAM identity) I was able to access my bucket and wipe out my backup data despite Veeams retention policy in place.

Why is that you might think. When you open the bucket contents (with e.g. an S3 browser) and login with the owner identity, you will spot that the owner of the bucket has full control ! If you login with these credentials, you can fully delete 'immutable objects' . So there is definately a whole lot more to it then what I have been reading over here.
Mildur
Product Manager
Posts: 8549
Liked: 2223 times
Joined: May 13, 2017 4:51 pm
Full Name: Fabian K.
Location: Switzerland
Contact:

Re: AWS S3 Immutability (Object Lock) error

Post by Mildur »

Hi Stefan

The objects are not gone. They should have a deletion marker on it which can be removed. Then the objects will be visible again.
You should ask your object storage vendor for such a script to remove the deletion marker on the objects.

Thanks
Fabian
Product Management Analyst @ Veeam Software
sti
Novice
Posts: 8
Liked: never
Joined: Oct 05, 2022 7:37 am
Full Name: Stefan Timmermans
Contact:

Re: AWS S3 Immutability (Object Lock) error

Post by sti »

You definately need a script there to retrieve every single Object IDs (from files to folders, the whole tree in fact ) Veeam keeps in your bucket.
After reading what deletion markers are, we are only then recovering the last version of those Object IDs you reference. I suppose the script comes down to retrieve a full list of object IDs and then to remove the deletion marker on each of them.
veremin
Product Manager
Posts: 20270
Liked: 2252 times
Joined: Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin
Contact:

Re: AWS S3 Immutability (Object Lock) error

Post by veremin »

You can find more information regarding the delete marker removal script here. And the good news is in version 12 you will not need to run the script - the backup server will be able to restore data even from objects with delete markers. Thanks!
sti
Novice
Posts: 8
Liked: never
Joined: Oct 05, 2022 7:37 am
Full Name: Stefan Timmermans
Contact:

Re: AWS S3 Immutability (Object Lock) error

Post by sti »

Just want to add , that once you remove the bucket 'root' folder , any subsequent backups remain 'invisible' too.
So for as long as long as your deletion marker is in place on your bucket root , any new subsequent backups beyond the deletion , even though appearing as restore point
cannot be restored , Veeam Backup and replication gives you the following error message :

REST API error : S3 error 'The specified key does not exist'
Code : No suchkey , error code 404

So once your bucket 'root' has been deleted, you are forced to figure out how to script around that problem and regain proper access to your data.
We downloaded Veeam v12 beta. Can a Veeam Moderator figure out if the Veeam 12 beta is already operational at this level ?
veremin
Product Manager
Posts: 20270
Liked: 2252 times
Joined: Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin
Contact:

Re: AWS S3 Immutability (Object Lock) error

Post by veremin »

What exactly do you mean by operational? Can it be used reliably in production environments? No, as the product is still in the beta stage, during which we're trying to spot and fix technical and logical issues based on the received feedback. Thanks!
sti
Novice
Posts: 8
Liked: never
Joined: Oct 05, 2022 7:37 am
Full Name: Stefan Timmermans
Contact:

Re: AWS S3 Immutability (Object Lock) error

Post by sti »

What i meant was, is Veeam B+R v12 able to restore objects that are hidden due to the deletion marker ?
Has one tested that in a lab with Veeam B+R v12 ?
I got a script from the vendor, ran it but kept on getting
REST API error : S3 error 'The specified key does not exist'
Code : No suchkey , error code 404
On both v11 & v12beta at that time.
veremin
Product Manager
Posts: 20270
Liked: 2252 times
Joined: Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin
Contact:

Re: AWS S3 Immutability (Object Lock) error

Post by veremin »

Confirmed, the v12 should be able to initiate restores from objects having delete markers assigned (without the need to run the script first). If you are already on the RTM build and having issues with the said scenario, kindly reach our support team directly. Thanks!
Post Reply

Who is online

Users browsing this forum: walo and 8 guests