Discussions related to using object storage as a backup target.
Post Reply
PhilippKE
Novice
Posts: 4
Liked: 1 time
Joined: Aug 13, 2021 6:24 am
Full Name: Philipp Bundschuh
Contact:

Preventing of destroying S3 bucket by an attacker

Post by PhilippKE » 1 person likes this post

Currently we do some offloading to Wasabi S3 with object lock. Setup was done as described in Veeam manual. Deleting a file/block directly via the Wasabi GUI does not work as immutability is enabled. Now we want to "simulate" a cyber attack:

What if the attacker get S3 bucket authentication data from Veeam DB and connects with a S3-Browser. He would be able to delete (=mark the object as deleted) some or all blocks. Of course, the data is not really deleted, but in this case, Veeam is not able anymore to handle this bucket.

What to do in that case? As we don't know, which objects were deleted, we would not be able to identify the objects changed by the manual deletion process.

regards
dalbertson
Veeam Software
Posts: 492
Liked: 175 times
Joined: Jul 21, 2015 12:38 pm
Full Name: Dustin Albertson
Contact:

Re: Preventing of destroying S3 bucket by an attacker

Post by dalbertson » 1 person likes this post

Hi @PhilippKE

Like you mentioned the data is not really gone, it has had a delete marker placed on the object itself. Thus, you should be able to remove the delete markers using the S3 CLI. All providers like AWS, wasabi etc usually have a way to script it out. This is no different. Check out this page for how to :)

https://wasabi-support.zendesk.com/hc/e ... -a-bucket-

Also to add, Veeam tracks the version IDs of the objects we place and when working with those objects we will call those objects by version id so the markers dont really matter, but using the method above it will remove the markers and put it back to "normal"
Dustin Albertson | Director of Product Management - Cloud & Applications | Veeam Product Management, Alliances
PhilippKE
Novice
Posts: 4
Liked: 1 time
Joined: Aug 13, 2021 6:24 am
Full Name: Philipp Bundschuh
Contact:

Re: Preventing of destroying S3 bucket by an attacker

Post by PhilippKE »

Hi Dustin,
great, "delete marker" was the essential keyword I have looked for. Thank you!
I will try that in our lab.
Regards
Philipp
dalbertson
Veeam Software
Posts: 492
Liked: 175 times
Joined: Jul 21, 2015 12:38 pm
Full Name: Dustin Albertson
Contact:

Re: Preventing of destroying S3 bucket by an attacker

Post by dalbertson » 1 person likes this post

FYI here is the same method for AWS

AWS - https://aws.amazon.com/premiumsupport/k ... iguration/
Dustin Albertson | Director of Product Management - Cloud & Applications | Veeam Product Management, Alliances
r.horowski
Novice
Posts: 7
Liked: 1 time
Joined: Sep 29, 2014 2:12 pm
Full Name: Robert Horowski
Contact:

Re: Preventing of destroying S3 bucket by an attacker

Post by r.horowski »

Hi @PhilippKE and @dalbertson

I'm currently trying to prepare a procedure for my customer for that exact scenario, when somebody gets credential to object storage and deletes (or put a delete marker) objects with API or S3-browser. So I've run backup and offloaded it to S3 (Wasabi) storage with immutability. Next I connected with S3-browser and deleted one of the objects (8.part), so now this object has current version marked as deleted. When trying to restore I get error in Veeam console
ERROR:
REST API error: 'S3 error: The specified key does not exist
Code: NoSuchKey', error code: 404

I am working with support on this (Case #05545115), but for now it does not work for me.

@PhilippKE
I am wondering if you were able to test this in your lab. Please let me know.
PhilippKE
Novice
Posts: 4
Liked: 1 time
Joined: Aug 13, 2021 6:24 am
Full Name: Philipp Bundschuh
Contact:

Re: Preventing of destroying S3 bucket by an attacker

Post by PhilippKE »

Hi Robert,

Veeam will not be able to see the "deleted" files, even they are just marked as deleted.
You will have to remove the deletion markers and recover the files to previous version. My example here works with Wasabi, but should also work with AWS:

Code: Select all

aws s3api list-object-versions --bucket BUCKETNAME --endpoint-url=https://s3.REGION.wasabisys.com --output json --query 'DeleteMarkers[].[Key, VersionId]' | jq -r '.[] | "--key '\''" + .[0] + "'\'' --version-id " + .[1]' |  xargs -L1 aws s3api delete-object --bucket BUCKETNAME --endpoint-url=https://s3.REGION.wasabisys.com
After removing the deletion markers, I was able to import / rescan the S3-Bucket.

BTW: I also created a Veeam Support Case, but got the same answer like you; the engineer told me, that veeam is able to recover the files even they have a deletion marker; but does not work.

Regards
Philipp
r.horowski
Novice
Posts: 7
Liked: 1 time
Joined: Sep 29, 2014 2:12 pm
Full Name: Robert Horowski
Contact:

Re: Preventing of destroying S3 bucket by an attacker

Post by r.horowski »

Hi Philipp,

Thanks for sharing your code with me. I will check if I am able to get my backups back using your method and post back here about results.

Regards,
Robert
AlexHeylin
Veteran
Posts: 563
Liked: 173 times
Joined: Nov 15, 2019 4:09 pm
Full Name: Alex Heylin
Contact:

Re: Preventing of destroying S3 bucket by an attacker

Post by AlexHeylin »

If Veeam support say it works and two people have proved it doesn't, that seems like either a bug, a non-conformance to design, or a subject for an enhancement request.

Perhaps a PM can advise on this?
veremin
Product Manager
Posts: 20382
Liked: 2294 times
Joined: Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin
Contact:

Re: Preventing of destroying S3 bucket by an attacker

Post by veremin »

We have asked the QA team to re-check the given scenario. I will let you know the results of our findings.
PhilippKE
Novice
Posts: 4
Liked: 1 time
Joined: Aug 13, 2021 6:24 am
Full Name: Philipp Bundschuh
Contact:

Re: Preventing of destroying S3 bucket by an attacker

Post by PhilippKE »

My Case: # 05416556
r.horowski
Novice
Posts: 7
Liked: 1 time
Joined: Sep 29, 2014 2:12 pm
Full Name: Robert Horowski
Contact:

Re: Preventing of destroying S3 bucket by an attacker

Post by r.horowski »

Hi PhilippKE,

I've tested your code with a very simple scenario where only 1 file was deleted and it worked. I was able to successfully restore my VM from capacity tier. I will run some more tests and try to do some more damage before restoring, but for now it looks good.

@Veremin @AlexHeylin
I also wanted to mention that my case has been escalated. I will let you know if anything new pops up.

Regards,
Robert
veremin
Product Manager
Posts: 20382
Liked: 2294 times
Joined: Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin
Contact:

Re: Preventing of destroying S3 bucket by an attacker

Post by veremin »

The QA team has verified the said scenario and found an issue that prevents restore from objects with deleted markers assigned. To restore from such objects, you should remove deleted markers (manually or automatically with a script). The issue is addressed in v12. Thanks again for raising this!
TonioRoffo
Enthusiast
Posts: 56
Liked: 6 times
Joined: Jun 18, 2009 2:27 pm
Full Name: Yves Smolders
Contact:

Re: Preventing of destroying S3 bucket by an attacker

Post by TonioRoffo »

What is the attack is "uglier" and said hacker overwrites one file 5 times, the other 8, and doesn't delete anything (or he does...)

How to figure out which version of the file is the correct one?

Is there a defence against this?

Thanks
veremin
Product Manager
Posts: 20382
Liked: 2294 times
Joined: Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin
Contact:

Re: Preventing of destroying S3 bucket by an attacker

Post by veremin »

Using the PowerShell cmdlet, you can always revert the Scale-Out Backup Repository to one of its known previous states. However, I'm not sure how you will overwrite a locked object. Thanks!
JWester
Service Provider
Posts: 63
Liked: 7 times
Joined: Apr 04, 2011 8:56 am
Full Name: Joern Westermann
Contact:

Re: Preventing of destroying S3 bucket by an attacker

Post by JWester »

veremin wrote: Aug 09, 2022 2:14 pm The QA team has verified the said scenario and found an issue that prevents restore from objects with deleted markers assigned. To restore from such objects, you should remove deleted markers (manually or automatically with a script). The issue is addressed in v12. Thanks again for raising this!
Are you sure this issue is fixed in V12?
I just did a fresh install of V12 in a small testing environment and did a backup of 3 VMs on S3 compatible (Ceph). Then:
* I deleted the repo and imported it again. works fine as I have not deleted any data.
* I deleted the repo and all data inside the bucket (directly with a S3 browser). No backup jobs are found during import.
With the aws cli I can see that there are > 55,000 deleted objects. Recovery with a script would take hours.
Am I doing something wrong?
Thanks!
Joern
JWester
Service Provider
Posts: 63
Liked: 7 times
Joined: Apr 04, 2011 8:56 am
Full Name: Joern Westermann
Contact:

Re: Preventing of destroying S3 bucket by an attacker

Post by JWester »

Ah, found the solution: When adding a new S3 repo you have to select a bucket and create a subfolder. This subfolder name has to be the same when importing the "empty" (=with deleted objects) bucket.
Post Reply

Who is online

Users browsing this forum: No registered users and 18 guests