-
- Influencer
- Posts: 12
- Liked: never
- Joined: Feb 23, 2015 2:02 pm
- Full Name: Bryan
- Contact:
S3 Immutable Question
I created an S3 bucket, setup a IAM API for Veeam to use with access to that bucket and the outlined permissions needed as discussed object-storage-f52/iam-json-for-aws-s3- ... 65327.html and set my S3 repository to immutable.
When I was exploring the bucket (not as root though my IAM account is a full admin) I saw on the properties of various objects they were set to immutable with an expiration of about ten days out. I didn't understand how it got to ten days since I'm just doing a POC I set the immutable timeframe to be three days, but I can experiment a bit with that later.
Either way, I was able to delete the immutable objects and the entire bucket full of immutable objects. I guess my question is since I was able to delete some immutable objects, I missed a step somewhere along the way. It was my understanding the only way to remove the objects I removed was to terminate the entire AWS account. Does anyone have a good step-by-step guide of setting up S3 with immutability or know of one? My Googling of that has failed me a bit.
When I was exploring the bucket (not as root though my IAM account is a full admin) I saw on the properties of various objects they were set to immutable with an expiration of about ten days out. I didn't understand how it got to ten days since I'm just doing a POC I set the immutable timeframe to be three days, but I can experiment a bit with that later.
Either way, I was able to delete the immutable objects and the entire bucket full of immutable objects. I guess my question is since I was able to delete some immutable objects, I missed a step somewhere along the way. It was my understanding the only way to remove the objects I removed was to terminate the entire AWS account. Does anyone have a good step-by-step guide of setting up S3 with immutability or know of one? My Googling of that has failed me a bit.
-
- Veeam Software
- Posts: 492
- Liked: 175 times
- Joined: Jul 21, 2015 12:38 pm
- Full Name: Dustin Albertson
- Contact:
Re: S3 Immutable Question
Dustin Albertson | Director of Product Management - Cloud & Applications | Veeam Product Management, Alliances
-
- Veeam Software
- Posts: 492
- Liked: 175 times
- Joined: Jul 21, 2015 12:38 pm
- Full Name: Dustin Albertson
- Contact:
Re: S3 Immutable Question
The other part about deleting the bucket won’t work. We use compliance mode and even though you are an admin or root it still won’t delete the data. It’s still there until it expires.
Dustin Albertson | Director of Product Management - Cloud & Applications | Veeam Product Management, Alliances
-
- Influencer
- Posts: 12
- Liked: never
- Joined: Feb 23, 2015 2:02 pm
- Full Name: Bryan
- Contact:
Re: S3 Immutable Question
Thanks Dustin! So I'm sorry for asking "basic" questions here as I'm very new to AWS and S3. The bucket has object lock showing as permanently enabled, with no automatic settings (none) for objects that are uploaded without an object lock config. I believe I read somewhere in a forum or in the documentation that part is correct. In the management interface I was able to delete objects that were set to immutable until nearly the end of this month.
I guess this is a stupid question, but if I was able to delete an immutable object from the AWS console did they just go somewhere else? Are they just not visible in the console anymore but still there and accessible via API?
I did rescan my storage in Veeam and it lost all the files so even Veeam can't see the immutable objects I was able to delete.
I guess this is a stupid question, but if I was able to delete an immutable object from the AWS console did they just go somewhere else? Are they just not visible in the console anymore but still there and accessible via API?
I did rescan my storage in Veeam and it lost all the files so even Veeam can't see the immutable objects I was able to delete.
-
- Veeam Software
- Posts: 492
- Liked: 175 times
- Joined: Jul 21, 2015 12:38 pm
- Full Name: Dustin Albertson
- Contact:
Re: S3 Immutable Question
No worries about asking questions as that’s what we are here for. To help share knowledge and learn from each other.
It’s just hidden. If you go into the bucket and select show versions button they should show back up. I would suggest opening a ticket and support can help you get them to show back up in veeam.
It’s just hidden. If you go into the bucket and select show versions button they should show back up. I would suggest opening a ticket and support can help you get them to show back up in veeam.
Dustin Albertson | Director of Product Management - Cloud & Applications | Veeam Product Management, Alliances
-
- Influencer
- Posts: 12
- Liked: never
- Joined: Feb 23, 2015 2:02 pm
- Full Name: Bryan
- Contact:
Re: S3 Immutable Question
Ha! Thanks.
Since this is just a POC I don't need to use support resources right now. As long as Veeam would be available to help in the very unlikely event of an attack that made its way to AWS object storage I'm fine with calling the POC a success. It looks like there are various scripts around that can be used to "undelete" the deleted items in the bucket and I imagine once that is done it's just a matter of rescanning the storage in Veeam and going from there. Hopefully Veeam support has some tried and tested documentation on that scenario but I certainly see how it's possible to do.
Since this is just a POC I don't need to use support resources right now. As long as Veeam would be available to help in the very unlikely event of an attack that made its way to AWS object storage I'm fine with calling the POC a success. It looks like there are various scripts around that can be used to "undelete" the deleted items in the bucket and I imagine once that is done it's just a matter of rescanning the storage in Veeam and going from there. Hopefully Veeam support has some tried and tested documentation on that scenario but I certainly see how it's possible to do.
-
- Veeam Software
- Posts: 492
- Liked: 175 times
- Joined: Jul 21, 2015 12:38 pm
- Full Name: Dustin Albertson
- Contact:
Re: S3 Immutable Question
This is indeed something that is known and planned for.
Dustin Albertson | Director of Product Management - Cloud & Applications | Veeam Product Management, Alliances
-
- Lurker
- Posts: 1
- Liked: never
- Joined: Sep 16, 2020 8:54 pm
- Full Name: Barry Strittmatter
- Contact:
Re: S3 Immutable Question
I also had this same experience and had these same questions. Did you ever figure out your statement about:
"When I was exploring the bucket (not as root though my IAM account is a full admin) I saw on the properties of various objects they were set to immutable with an expiration of about ten days out. I didn't understand how it got to ten days since I'm just doing a POC I set the immutable timeframe to be three days, but I can experiment a bit with that later."
I have mine set to 1 day in Veeam but the AWS S3 console states the Veeam backup files are object locked for 10 days.
"When I was exploring the bucket (not as root though my IAM account is a full admin) I saw on the properties of various objects they were set to immutable with an expiration of about ten days out. I didn't understand how it got to ten days since I'm just doing a POC I set the immutable timeframe to be three days, but I can experiment a bit with that later."
I have mine set to 1 day in Veeam but the AWS S3 console states the Veeam backup files are object locked for 10 days.
-
- Chief Product Officer
- Posts: 31780
- Liked: 7280 times
- Joined: Jan 01, 2006 1:01 am
- Location: Baar, Switzerland
- Contact:
Re: S3 Immutable Question
This is explained in the User's Guide. Thanks!
-
- Service Provider
- Posts: 1092
- Liked: 134 times
- Joined: May 14, 2013 8:35 pm
- Full Name: Frank Iversen
- Location: Norway
- Contact:
Re: S3 Immutable Question
We have deleted an bucket with immutable object lock settings (as a test).
shouldn't that be impossible? if we really managed to deleted it, shouldn't it be restorable somewhere?
We are logged in a root in the aws console as part of this deletion for testing purpose.
shouldn't that be impossible? if we really managed to deleted it, shouldn't it be restorable somewhere?
We are logged in a root in the aws console as part of this deletion for testing purpose.
-
- Veeam Software
- Posts: 291
- Liked: 139 times
- Joined: Jul 24, 2018 8:38 pm
- Full Name: Stephen Firmes
- Contact:
Re: S3 Immutable Question
Were there any locked objects in the bucket? If not, then you can delete the bucket.
Steve Firmes | Senior Solutions Architect, Product Management - Alliances @ Veeam Software
-
- Veeam Software
- Posts: 492
- Liked: 175 times
- Joined: Jul 21, 2015 12:38 pm
- Full Name: Dustin Albertson
- Contact:
Re: S3 Immutable Question
@frankive While you "deleted" it and you cant see it in the console, the data is still there. When you delete versioned data from S3 it doesnt actually delete it. It puts a delete marker on the data and hides it. You can get the data back in the console by deleting the delete marker from each object.
Here is an explanation of delete markers. https://docs.aws.amazon.com/AmazonS3/la ... sions.html
The object lock flag will keep these from being deleted. Give me a few minutes and I will reply back with a script that could be used to remove the delete markers from all the objects in a bucket.
Here is an explanation of delete markers. https://docs.aws.amazon.com/AmazonS3/la ... sions.html
The object lock flag will keep these from being deleted. Give me a few minutes and I will reply back with a script that could be used to remove the delete markers from all the objects in a bucket.
Dustin Albertson | Director of Product Management - Cloud & Applications | Veeam Product Management, Alliances
-
- Service Provider
- Posts: 1092
- Liked: 134 times
- Joined: May 14, 2013 8:35 pm
- Full Name: Frank Iversen
- Location: Norway
- Contact:
Re: S3 Immutable Question
Update: the bucket was empty, thats why we could delete it.
I tried to delete another bucket which were full of demo backup data. We then get access denied when we try to delete the backup and its files. That's nice!
But it says i managed to delete 3 files, I am not sure which files this could be.
Any ideas?
I tried to delete another bucket which were full of demo backup data. We then get access denied when we try to delete the backup and its files. That's nice!
But it says i managed to delete 3 files, I am not sure which files this could be.
Any ideas?
-
- Veeam Software
- Posts: 492
- Liked: 175 times
- Joined: Jul 21, 2015 12:38 pm
- Full Name: Dustin Albertson
- Contact:
Re: S3 Immutable Question
Again, there are times where you will be able to "delete" a bucket but it does not delete it or the files. It puts a marker on it and hides it from the AWS console. Since object lock uses versioning there is a way to pull this via the aws cli.
https://aws.amazon.com/premiumsupport/k ... iguration/
In your case you were able to delete an empty bucket because there were no "locked" objects inside. But I have seen examples where people were able to delete a object locked files in a bucket and were upset. But as i mentioned this data wasn't gone it just was removed from the console view.
https://aws.amazon.com/premiumsupport/k ... iguration/
In your case you were able to delete an empty bucket because there were no "locked" objects inside. But I have seen examples where people were able to delete a object locked files in a bucket and were upset. But as i mentioned this data wasn't gone it just was removed from the console view.
Dustin Albertson | Director of Product Management - Cloud & Applications | Veeam Product Management, Alliances
-
- Service Provider
- Posts: 1092
- Liked: 134 times
- Joined: May 14, 2013 8:35 pm
- Full Name: Frank Iversen
- Location: Norway
- Contact:
Re: S3 Immutable Question
This is great information!
I will need to find a command to locate all files with deleted marked so we can see which 3 files which was deleted
I will need to find a command to locate all files with deleted marked so we can see which 3 files which was deleted
-
- Veeam Software
- Posts: 492
- Liked: 175 times
- Joined: Jul 21, 2015 12:38 pm
- Full Name: Dustin Albertson
- Contact:
Re: S3 Immutable Question
For anyone interested there is a way to remove delete markers from the aws cli
Change the bucketname. and as always make sure that you know what you are doing when you use scripts. Use this as your own risk.
Code: Select all
aws s3api delete-objects --bucket bucket-name --delete "$(aws s3api list-object-versions --bucket "bucket_name" --output=json --query='{Objects: DeleteMarkers[].{Key:Key,VersionId:VersionId}}')"
Dustin Albertson | Director of Product Management - Cloud & Applications | Veeam Product Management, Alliances
-
- Veeam Software
- Posts: 492
- Liked: 175 times
- Joined: Jul 21, 2015 12:38 pm
- Full Name: Dustin Albertson
- Contact:
Re: S3 Immutable Question
Dustin Albertson | Director of Product Management - Cloud & Applications | Veeam Product Management, Alliances
-
- Service Provider
- Posts: 1092
- Liked: 134 times
- Joined: May 14, 2013 8:35 pm
- Full Name: Frank Iversen
- Location: Norway
- Contact:
Re: S3 Immutable Question
Tried this command to list the 3 deleted objects, but the result as "null"
aws s3api list-object-versions --bucket NAME-OF-MY-BUCKET-WITH-DELETED-FILES --query 'DeleteMarkers[?IsLatest==`true`]'
Could this mean the 3 deleted files was just some files not even related to the bucket itself? some tempfiles?
aws s3api list-object-versions --bucket NAME-OF-MY-BUCKET-WITH-DELETED-FILES --query 'DeleteMarkers[?IsLatest==`true`]'
Could this mean the 3 deleted files was just some files not even related to the bucket itself? some tempfiles?
-
- Veeam Software
- Posts: 492
- Liked: 175 times
- Joined: Jul 21, 2015 12:38 pm
- Full Name: Dustin Albertson
- Contact:
Re: S3 Immutable Question
It could. If you use the link above and you are not able to see the deleted objects in the console then its most likely not an issue. But if you could look at the logs and see what was deleted it may help to know.
Dustin Albertson | Director of Product Management - Cloud & Applications | Veeam Product Management, Alliances
Who is online
Users browsing this forum: No registered users and 4 guests