S3 Immutable Question

[bryanmeche]

I created an S3 bucket, setup a IAM API for Veeam to use with access to that bucket and the outlined permissions needed as discussed object-storage-f52/iam-json-for-aws-s3- ... 65327.html and set my S3 repository to immutable.

When I was exploring the bucket (not as root though my IAM account is a full admin) I saw on the properties of various objects they were set to immutable with an expiration of about ten days out. I didn't understand how it got to ten days since I'm just doing a POC I set the immutable timeframe to be three days, but I can experiment a bit with that later.

Either way, I was able to delete the immutable objects and the entire bucket full of immutable objects. I guess my question is since I was able to delete some immutable objects, I missed a step somewhere along the way. It was my understanding the only way to remove the objects I removed was to terminate the entire AWS account. Does anyone have a good step-by-step guide of setting up S3 with immutability or know of one? My Googling of that has failed me a bit.

[dalbertson]

It’s block generation. Explained here

https://helpcenter.veeam.com/docs/backu ... ml?ver=100

[dalbertson]

The other part about deleting the bucket won’t work. We use compliance mode and even though you are an admin or root it still won’t delete the data. It’s still there until it expires.

[bryanmeche]

Thanks Dustin! So I'm sorry for asking "basic" questions here as I'm very new to AWS and S3. The bucket has object lock showing as permanently enabled, with no automatic settings (none) for objects that are uploaded without an object lock config. I believe I read somewhere in a forum or in the documentation that part is correct. In the management interface I was able to delete objects that were set to immutable until nearly the end of this month.

I guess this is a stupid question, but if I was able to delete an immutable object from the AWS console did they just go somewhere else? Are they just not visible in the console anymore but still there and accessible via API?

I did rescan my storage in Veeam and it lost all the files so even Veeam can't see the immutable objects I was able to delete.

[dalbertson]

No worries about asking questions as that’s what we are here for. To help share knowledge and learn from each other.

It’s just hidden. If you go into the bucket and select show versions button they should show back up. I would suggest opening a ticket and support can help you get them to show back up in veeam.

[bryanmeche]

Ha! Thanks.

Since this is just a POC I don't need to use support resources right now. As long as Veeam would be available to help in the very unlikely event of an attack that made its way to AWS object storage I'm fine with calling the POC a success. It looks like there are various scripts around that can be used to "undelete" the deleted items in the bucket and I imagine once that is done it's just a matter of rescanning the storage in Veeam and going from there. Hopefully Veeam support has some tried and tested documentation on that scenario but I certainly see how it's possible to do.

[dalbertson]

This is indeed something that is known and planned for.

[barry11]

I also had this same experience and had these same questions. Did you ever figure out your statement about:

"When I was exploring the bucket (not as root though my IAM account is a full admin) I saw on the properties of various objects they were set to immutable with an expiration of about ten days out. I didn't understand how it got to ten days since I'm just doing a POC I set the immutable timeframe to be three days, but I can experiment a bit with that later."

I have mine set to 1 day in Veeam but the AWS S3 console states the Veeam backup files are object locked for 10 days.

[Gostev]

This is explained in the User's Guide. Thanks!

[frankive]

We have deleted an bucket with immutable object lock settings (as a test).
shouldn't that be impossible? if we really managed to deleted it, shouldn't it be restorable somewhere?
We are logged in a root in the aws console as part of this deletion for testing purpose.

[sfirmes]

Were there any locked objects in the bucket? If not, then you can delete the bucket.

[dalbertson]

@frankive While you "deleted" it and you cant see it in the console, the data is still there. When you delete versioned data from S3 it doesnt actually delete it. It puts a delete marker on the data and hides it. You can get the data back in the console by deleting the delete marker from each object.

Here is an explanation of delete markers. https://docs.aws.amazon.com/AmazonS3/la ... sions.html

The object lock flag will keep these from being deleted. Give me a few minutes and I will reply back with a script that could be used to remove the delete markers from all the objects in a bucket.

[frankive]

Update: the bucket was empty, thats why we could delete it.

I tried to delete another bucket which were full of demo backup data. We then get access denied when we try to delete the backup and its files. That's nice!
But it says i managed to delete 3 files, I am not sure which files this could be.

Any ideas?

[dalbertson]

Again, there are times where you will be able to "delete" a bucket but it does not delete it or the files. It puts a marker on it and hides it from the AWS console. Since object lock uses versioning there is a way to pull this via the aws cli.

https://aws.amazon.com/premiumsupport/k ... iguration/

In your case you were able to delete an empty bucket because there were no "locked" objects inside. But I have seen examples where people were able to delete a object locked files in a bucket and were upset. But as i mentioned this data wasn't gone it just was removed from the console view.

[frankive]

This is great information!
I will need to find a command to locate all files with deleted marked so we can see which 3 files which was deleted

[dalbertson]

For anyone interested there is a way to remove delete markers from the aws cli

Code: Select all

aws s3api delete-objects --bucket bucket-name --delete "$(aws s3api list-object-versions --bucket "bucket_name" --output=json --query='{Objects: DeleteMarkers[].{Key:Key,VersionId:VersionId}}')"

Change the bucketname. and as always make sure that you know what you are doing when you use scripts. Use this as your own risk.

[dalbertson]

@frankive https://aws.amazon.com/premiumsupport/k ... iguration/

[frankive]

Tried this command to list the 3 deleted objects, but the result as "null"

aws s3api list-object-versions --bucket NAME-OF-MY-BUCKET-WITH-DELETED-FILES --query 'DeleteMarkers[?IsLatest==`true`]'

Could this mean the 3 deleted files was just some files not even related to the bucket itself? some tempfiles?

[dalbertson]

It could. If you use the link above and you are not able to see the deleted objects in the console then its most likely not an issue. But if you could look at the logs and see what was deleted it may help to know.