-
- Novice
- Posts: 4
- Liked: 1 time
- Joined: Aug 13, 2021 6:24 am
- Full Name: Philipp Bundschuh
- Contact:
Preventing of destroying S3 bucket by an attacker
Currently we do some offloading to Wasabi S3 with object lock. Setup was done as described in Veeam manual. Deleting a file/block directly via the Wasabi GUI does not work as immutability is enabled. Now we want to "simulate" a cyber attack:
What if the attacker get S3 bucket authentication data from Veeam DB and connects with a S3-Browser. He would be able to delete (=mark the object as deleted) some or all blocks. Of course, the data is not really deleted, but in this case, Veeam is not able anymore to handle this bucket.
What to do in that case? As we don't know, which objects were deleted, we would not be able to identify the objects changed by the manual deletion process.
regards
What if the attacker get S3 bucket authentication data from Veeam DB and connects with a S3-Browser. He would be able to delete (=mark the object as deleted) some or all blocks. Of course, the data is not really deleted, but in this case, Veeam is not able anymore to handle this bucket.
What to do in that case? As we don't know, which objects were deleted, we would not be able to identify the objects changed by the manual deletion process.
regards
-
- Veeam Software
- Posts: 492
- Liked: 175 times
- Joined: Jul 21, 2015 12:38 pm
- Full Name: Dustin Albertson
- Contact:
Re: Preventing of destroying S3 bucket by an attacker
Hi @PhilippKE
Like you mentioned the data is not really gone, it has had a delete marker placed on the object itself. Thus, you should be able to remove the delete markers using the S3 CLI. All providers like AWS, wasabi etc usually have a way to script it out. This is no different. Check out this page for how to
https://wasabi-support.zendesk.com/hc/e ... -a-bucket-
Also to add, Veeam tracks the version IDs of the objects we place and when working with those objects we will call those objects by version id so the markers dont really matter, but using the method above it will remove the markers and put it back to "normal"
Like you mentioned the data is not really gone, it has had a delete marker placed on the object itself. Thus, you should be able to remove the delete markers using the S3 CLI. All providers like AWS, wasabi etc usually have a way to script it out. This is no different. Check out this page for how to
https://wasabi-support.zendesk.com/hc/e ... -a-bucket-
Also to add, Veeam tracks the version IDs of the objects we place and when working with those objects we will call those objects by version id so the markers dont really matter, but using the method above it will remove the markers and put it back to "normal"
Dustin Albertson | Director of Product Management - Cloud & Applications | Veeam Product Management, Alliances
-
- Novice
- Posts: 4
- Liked: 1 time
- Joined: Aug 13, 2021 6:24 am
- Full Name: Philipp Bundschuh
- Contact:
Re: Preventing of destroying S3 bucket by an attacker
Hi Dustin,
great, "delete marker" was the essential keyword I have looked for. Thank you!
I will try that in our lab.
Regards
Philipp
great, "delete marker" was the essential keyword I have looked for. Thank you!
I will try that in our lab.
Regards
Philipp
-
- Veeam Software
- Posts: 492
- Liked: 175 times
- Joined: Jul 21, 2015 12:38 pm
- Full Name: Dustin Albertson
- Contact:
Re: Preventing of destroying S3 bucket by an attacker
Dustin Albertson | Director of Product Management - Cloud & Applications | Veeam Product Management, Alliances
-
- Novice
- Posts: 7
- Liked: 1 time
- Joined: Sep 29, 2014 2:12 pm
- Full Name: Robert Horowski
- Contact:
Re: Preventing of destroying S3 bucket by an attacker
Hi @PhilippKE and @dalbertson
I'm currently trying to prepare a procedure for my customer for that exact scenario, when somebody gets credential to object storage and deletes (or put a delete marker) objects with API or S3-browser. So I've run backup and offloaded it to S3 (Wasabi) storage with immutability. Next I connected with S3-browser and deleted one of the objects (8.part), so now this object has current version marked as deleted. When trying to restore I get error in Veeam console
ERROR:
REST API error: 'S3 error: The specified key does not exist
Code: NoSuchKey', error code: 404
I am working with support on this (Case #05545115), but for now it does not work for me.
@PhilippKE
I am wondering if you were able to test this in your lab. Please let me know.
I'm currently trying to prepare a procedure for my customer for that exact scenario, when somebody gets credential to object storage and deletes (or put a delete marker) objects with API or S3-browser. So I've run backup and offloaded it to S3 (Wasabi) storage with immutability. Next I connected with S3-browser and deleted one of the objects (8.part), so now this object has current version marked as deleted. When trying to restore I get error in Veeam console
ERROR:
REST API error: 'S3 error: The specified key does not exist
Code: NoSuchKey', error code: 404
I am working with support on this (Case #05545115), but for now it does not work for me.
@PhilippKE
I am wondering if you were able to test this in your lab. Please let me know.
-
- Novice
- Posts: 4
- Liked: 1 time
- Joined: Aug 13, 2021 6:24 am
- Full Name: Philipp Bundschuh
- Contact:
Re: Preventing of destroying S3 bucket by an attacker
Hi Robert,
Veeam will not be able to see the "deleted" files, even they are just marked as deleted.
You will have to remove the deletion markers and recover the files to previous version. My example here works with Wasabi, but should also work with AWS:
After removing the deletion markers, I was able to import / rescan the S3-Bucket.
BTW: I also created a Veeam Support Case, but got the same answer like you; the engineer told me, that veeam is able to recover the files even they have a deletion marker; but does not work.
Regards
Philipp
Veeam will not be able to see the "deleted" files, even they are just marked as deleted.
You will have to remove the deletion markers and recover the files to previous version. My example here works with Wasabi, but should also work with AWS:
Code: Select all
aws s3api list-object-versions --bucket BUCKETNAME --endpoint-url=https://s3.REGION.wasabisys.com --output json --query 'DeleteMarkers[].[Key, VersionId]' | jq -r '.[] | "--key '\''" + .[0] + "'\'' --version-id " + .[1]' | xargs -L1 aws s3api delete-object --bucket BUCKETNAME --endpoint-url=https://s3.REGION.wasabisys.com
BTW: I also created a Veeam Support Case, but got the same answer like you; the engineer told me, that veeam is able to recover the files even they have a deletion marker; but does not work.
Regards
Philipp
-
- Novice
- Posts: 7
- Liked: 1 time
- Joined: Sep 29, 2014 2:12 pm
- Full Name: Robert Horowski
- Contact:
Re: Preventing of destroying S3 bucket by an attacker
Hi Philipp,
Thanks for sharing your code with me. I will check if I am able to get my backups back using your method and post back here about results.
Regards,
Robert
Thanks for sharing your code with me. I will check if I am able to get my backups back using your method and post back here about results.
Regards,
Robert
-
- Veteran
- Posts: 563
- Liked: 173 times
- Joined: Nov 15, 2019 4:09 pm
- Full Name: Alex Heylin
- Contact:
Re: Preventing of destroying S3 bucket by an attacker
If Veeam support say it works and two people have proved it doesn't, that seems like either a bug, a non-conformance to design, or a subject for an enhancement request.
Perhaps a PM can advise on this?
Perhaps a PM can advise on this?
-
- Product Manager
- Posts: 20382
- Liked: 2294 times
- Joined: Oct 26, 2012 3:28 pm
- Full Name: Vladimir Eremin
- Contact:
Re: Preventing of destroying S3 bucket by an attacker
We have asked the QA team to re-check the given scenario. I will let you know the results of our findings.
-
- Novice
- Posts: 4
- Liked: 1 time
- Joined: Aug 13, 2021 6:24 am
- Full Name: Philipp Bundschuh
- Contact:
Re: Preventing of destroying S3 bucket by an attacker
My Case: # 05416556
-
- Novice
- Posts: 7
- Liked: 1 time
- Joined: Sep 29, 2014 2:12 pm
- Full Name: Robert Horowski
- Contact:
Re: Preventing of destroying S3 bucket by an attacker
Hi PhilippKE,
I've tested your code with a very simple scenario where only 1 file was deleted and it worked. I was able to successfully restore my VM from capacity tier. I will run some more tests and try to do some more damage before restoring, but for now it looks good.
@Veremin @AlexHeylin
I also wanted to mention that my case has been escalated. I will let you know if anything new pops up.
Regards,
Robert
I've tested your code with a very simple scenario where only 1 file was deleted and it worked. I was able to successfully restore my VM from capacity tier. I will run some more tests and try to do some more damage before restoring, but for now it looks good.
@Veremin @AlexHeylin
I also wanted to mention that my case has been escalated. I will let you know if anything new pops up.
Regards,
Robert
-
- Product Manager
- Posts: 20382
- Liked: 2294 times
- Joined: Oct 26, 2012 3:28 pm
- Full Name: Vladimir Eremin
- Contact:
Re: Preventing of destroying S3 bucket by an attacker
The QA team has verified the said scenario and found an issue that prevents restore from objects with deleted markers assigned. To restore from such objects, you should remove deleted markers (manually or automatically with a script). The issue is addressed in v12. Thanks again for raising this!
-
- Enthusiast
- Posts: 56
- Liked: 6 times
- Joined: Jun 18, 2009 2:27 pm
- Full Name: Yves Smolders
- Contact:
Re: Preventing of destroying S3 bucket by an attacker
What is the attack is "uglier" and said hacker overwrites one file 5 times, the other 8, and doesn't delete anything (or he does...)
How to figure out which version of the file is the correct one?
Is there a defence against this?
Thanks
How to figure out which version of the file is the correct one?
Is there a defence against this?
Thanks
-
- Product Manager
- Posts: 20382
- Liked: 2294 times
- Joined: Oct 26, 2012 3:28 pm
- Full Name: Vladimir Eremin
- Contact:
Re: Preventing of destroying S3 bucket by an attacker
Using the PowerShell cmdlet, you can always revert the Scale-Out Backup Repository to one of its known previous states. However, I'm not sure how you will overwrite a locked object. Thanks!
-
- Service Provider
- Posts: 63
- Liked: 7 times
- Joined: Apr 04, 2011 8:56 am
- Full Name: Joern Westermann
- Contact:
Re: Preventing of destroying S3 bucket by an attacker
Are you sure this issue is fixed in V12?veremin wrote: ↑Aug 09, 2022 2:14 pm The QA team has verified the said scenario and found an issue that prevents restore from objects with deleted markers assigned. To restore from such objects, you should remove deleted markers (manually or automatically with a script). The issue is addressed in v12. Thanks again for raising this!
I just did a fresh install of V12 in a small testing environment and did a backup of 3 VMs on S3 compatible (Ceph). Then:
* I deleted the repo and imported it again. works fine as I have not deleted any data.
* I deleted the repo and all data inside the bucket (directly with a S3 browser). No backup jobs are found during import.
With the aws cli I can see that there are > 55,000 deleted objects. Recovery with a script would take hours.
Am I doing something wrong?
Thanks!
Joern
-
- Service Provider
- Posts: 63
- Liked: 7 times
- Joined: Apr 04, 2011 8:56 am
- Full Name: Joern Westermann
- Contact:
Re: Preventing of destroying S3 bucket by an attacker
Ah, found the solution: When adding a new S3 repo you have to select a bucket and create a subfolder. This subfolder name has to be the same when importing the "empty" (=with deleted objects) bucket.
Who is online
Users browsing this forum: No registered users and 18 guests