Discussions related to using object storage as a backup target.
Post Reply
bryanmeche
Influencer
Posts: 12
Liked: never
Joined: Feb 23, 2015 2:02 pm
Full Name: Bryan
Contact:

S3 Immutable Question

Post by bryanmeche »

I created an S3 bucket, setup a IAM API for Veeam to use with access to that bucket and the outlined permissions needed as discussed object-storage-f52/iam-json-for-aws-s3- ... 65327.html and set my S3 repository to immutable.

When I was exploring the bucket (not as root though my IAM account is a full admin) I saw on the properties of various objects they were set to immutable with an expiration of about ten days out. I didn't understand how it got to ten days since I'm just doing a POC I set the immutable timeframe to be three days, but I can experiment a bit with that later.

Either way, I was able to delete the immutable objects and the entire bucket full of immutable objects. I guess my question is since I was able to delete some immutable objects, I missed a step somewhere along the way. It was my understanding the only way to remove the objects I removed was to terminate the entire AWS account. Does anyone have a good step-by-step guide of setting up S3 with immutability or know of one? My Googling of that has failed me a bit.
dalbertson
Veeam Software
Posts: 492
Liked: 175 times
Joined: Jul 21, 2015 12:38 pm
Full Name: Dustin Albertson
Contact:

Re: S3 Immutable Question

Post by dalbertson »

It’s block generation. Explained here

https://helpcenter.veeam.com/docs/backu ... ml?ver=100
Dustin Albertson | Director of Product Management - Cloud & Applications | Veeam Product Management, Alliances
dalbertson
Veeam Software
Posts: 492
Liked: 175 times
Joined: Jul 21, 2015 12:38 pm
Full Name: Dustin Albertson
Contact:

Re: S3 Immutable Question

Post by dalbertson »

The other part about deleting the bucket won’t work. We use compliance mode and even though you are an admin or root it still won’t delete the data. It’s still there until it expires.
Dustin Albertson | Director of Product Management - Cloud & Applications | Veeam Product Management, Alliances
bryanmeche
Influencer
Posts: 12
Liked: never
Joined: Feb 23, 2015 2:02 pm
Full Name: Bryan
Contact:

Re: S3 Immutable Question

Post by bryanmeche »

Thanks Dustin! So I'm sorry for asking "basic" questions here as I'm very new to AWS and S3. The bucket has object lock showing as permanently enabled, with no automatic settings (none) for objects that are uploaded without an object lock config. I believe I read somewhere in a forum or in the documentation that part is correct. In the management interface I was able to delete objects that were set to immutable until nearly the end of this month.

I guess this is a stupid question, but if I was able to delete an immutable object from the AWS console did they just go somewhere else? Are they just not visible in the console anymore but still there and accessible via API?

I did rescan my storage in Veeam and it lost all the files so even Veeam can't see the immutable objects I was able to delete.
dalbertson
Veeam Software
Posts: 492
Liked: 175 times
Joined: Jul 21, 2015 12:38 pm
Full Name: Dustin Albertson
Contact:

Re: S3 Immutable Question

Post by dalbertson »

No worries about asking questions as that’s what we are here for. To help share knowledge and learn from each other.

It’s just hidden. If you go into the bucket and select show versions button they should show back up. I would suggest opening a ticket and support can help you get them to show back up in veeam.
Dustin Albertson | Director of Product Management - Cloud & Applications | Veeam Product Management, Alliances
bryanmeche
Influencer
Posts: 12
Liked: never
Joined: Feb 23, 2015 2:02 pm
Full Name: Bryan
Contact:

Re: S3 Immutable Question

Post by bryanmeche »

Ha! Thanks.

Since this is just a POC I don't need to use support resources right now. As long as Veeam would be available to help in the very unlikely event of an attack that made its way to AWS object storage I'm fine with calling the POC a success. It looks like there are various scripts around that can be used to "undelete" the deleted items in the bucket and I imagine once that is done it's just a matter of rescanning the storage in Veeam and going from there. Hopefully Veeam support has some tried and tested documentation on that scenario but I certainly see how it's possible to do.
dalbertson
Veeam Software
Posts: 492
Liked: 175 times
Joined: Jul 21, 2015 12:38 pm
Full Name: Dustin Albertson
Contact:

Re: S3 Immutable Question

Post by dalbertson »

This is indeed something that is known and planned for.
Dustin Albertson | Director of Product Management - Cloud & Applications | Veeam Product Management, Alliances
barry11
Lurker
Posts: 1
Liked: never
Joined: Sep 16, 2020 8:54 pm
Full Name: Barry Strittmatter
Contact:

Re: S3 Immutable Question

Post by barry11 »

I also had this same experience and had these same questions. Did you ever figure out your statement about:

"When I was exploring the bucket (not as root though my IAM account is a full admin) I saw on the properties of various objects they were set to immutable with an expiration of about ten days out. I didn't understand how it got to ten days since I'm just doing a POC I set the immutable timeframe to be three days, but I can experiment a bit with that later."

I have mine set to 1 day in Veeam but the AWS S3 console states the Veeam backup files are object locked for 10 days.
Gostev
Chief Product Officer
Posts: 31455
Liked: 6646 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: S3 Immutable Question

Post by Gostev » 2 people like this post

This is explained in the User's Guide. Thanks!
frankive
Service Provider
Posts: 1092
Liked: 134 times
Joined: May 14, 2013 8:35 pm
Full Name: Frank Iversen
Location: Norway
Contact:

Re: S3 Immutable Question

Post by frankive »

We have deleted an bucket with immutable object lock settings (as a test).
shouldn't that be impossible? if we really managed to deleted it, shouldn't it be restorable somewhere?
We are logged in a root in the aws console as part of this deletion for testing purpose.
sfirmes
Veeam Software
Posts: 223
Liked: 117 times
Joined: Jul 24, 2018 8:38 pm
Full Name: Stephen Firmes
Contact:

Re: S3 Immutable Question

Post by sfirmes » 1 person likes this post

Were there any locked objects in the bucket? If not, then you can delete the bucket.
Senior Solutions Architect, Product Management - Alliances @ Veeam Software
dalbertson
Veeam Software
Posts: 492
Liked: 175 times
Joined: Jul 21, 2015 12:38 pm
Full Name: Dustin Albertson
Contact:

Re: S3 Immutable Question

Post by dalbertson » 2 people like this post

@frankive While you "deleted" it and you cant see it in the console, the data is still there. When you delete versioned data from S3 it doesnt actually delete it. It puts a delete marker on the data and hides it. You can get the data back in the console by deleting the delete marker from each object.

Here is an explanation of delete markers. https://docs.aws.amazon.com/AmazonS3/la ... sions.html

The object lock flag will keep these from being deleted. Give me a few minutes and I will reply back with a script that could be used to remove the delete markers from all the objects in a bucket.
Dustin Albertson | Director of Product Management - Cloud & Applications | Veeam Product Management, Alliances
frankive
Service Provider
Posts: 1092
Liked: 134 times
Joined: May 14, 2013 8:35 pm
Full Name: Frank Iversen
Location: Norway
Contact:

Re: S3 Immutable Question

Post by frankive »

Update: the bucket was empty, thats why we could delete it.

I tried to delete another bucket which were full of demo backup data. We then get access denied when we try to delete the backup and its files. That's nice!
But it says i managed to delete 3 files, I am not sure which files this could be.

Any ideas?
dalbertson
Veeam Software
Posts: 492
Liked: 175 times
Joined: Jul 21, 2015 12:38 pm
Full Name: Dustin Albertson
Contact:

Re: S3 Immutable Question

Post by dalbertson »

Again, there are times where you will be able to "delete" a bucket but it does not delete it or the files. It puts a marker on it and hides it from the AWS console. Since object lock uses versioning there is a way to pull this via the aws cli.

https://aws.amazon.com/premiumsupport/k ... iguration/

In your case you were able to delete an empty bucket because there were no "locked" objects inside. But I have seen examples where people were able to delete a object locked files in a bucket and were upset. But as i mentioned this data wasn't gone it just was removed from the console view.
Dustin Albertson | Director of Product Management - Cloud & Applications | Veeam Product Management, Alliances
frankive
Service Provider
Posts: 1092
Liked: 134 times
Joined: May 14, 2013 8:35 pm
Full Name: Frank Iversen
Location: Norway
Contact:

Re: S3 Immutable Question

Post by frankive »

This is great information!
I will need to find a command to locate all files with deleted marked so we can see which 3 files which was deleted
dalbertson
Veeam Software
Posts: 492
Liked: 175 times
Joined: Jul 21, 2015 12:38 pm
Full Name: Dustin Albertson
Contact:

Re: S3 Immutable Question

Post by dalbertson » 1 person likes this post

For anyone interested there is a way to remove delete markers from the aws cli

Code: Select all

aws s3api delete-objects --bucket bucket-name --delete "$(aws s3api list-object-versions --bucket "bucket_name" --output=json --query='{Objects: DeleteMarkers[].{Key:Key,VersionId:VersionId}}')"
Change the bucketname. and as always make sure that you know what you are doing when you use scripts. Use this as your own risk.
Dustin Albertson | Director of Product Management - Cloud & Applications | Veeam Product Management, Alliances
dalbertson
Veeam Software
Posts: 492
Liked: 175 times
Joined: Jul 21, 2015 12:38 pm
Full Name: Dustin Albertson
Contact:

Re: S3 Immutable Question

Post by dalbertson » 1 person likes this post

Dustin Albertson | Director of Product Management - Cloud & Applications | Veeam Product Management, Alliances
frankive
Service Provider
Posts: 1092
Liked: 134 times
Joined: May 14, 2013 8:35 pm
Full Name: Frank Iversen
Location: Norway
Contact:

Re: S3 Immutable Question

Post by frankive »

Tried this command to list the 3 deleted objects, but the result as "null"

aws s3api list-object-versions --bucket NAME-OF-MY-BUCKET-WITH-DELETED-FILES --query 'DeleteMarkers[?IsLatest==`true`]'

Could this mean the 3 deleted files was just some files not even related to the bucket itself? some tempfiles?
dalbertson
Veeam Software
Posts: 492
Liked: 175 times
Joined: Jul 21, 2015 12:38 pm
Full Name: Dustin Albertson
Contact:

Re: S3 Immutable Question

Post by dalbertson »

It could. If you use the link above and you are not able to see the deleted objects in the console then its most likely not an issue. But if you could look at the logs and see what was deleted it may help to know.
Dustin Albertson | Director of Product Management - Cloud & Applications | Veeam Product Management, Alliances
Post Reply

Who is online

Users browsing this forum: jigar.thakker and 9 guests