Preventing of destroying S3 bucket by an attacker

[PhilippKE]

Currently we do some offloading to Wasabi S3 with object lock. Setup was done as described in Veeam manual. Deleting a file/block directly via the Wasabi GUI does not work as immutability is enabled. Now we want to "simulate" a cyber attack:

What if the attacker get S3 bucket authentication data from Veeam DB and connects with a S3-Browser. He would be able to delete (=mark the object as deleted) some or all blocks. Of course, the data is not really deleted, but in this case, Veeam is not able anymore to handle this bucket.

What to do in that case? As we don't know, which objects were deleted, we would not be able to identify the objects changed by the manual deletion process.

regards

[dalbertson]

Hi @PhilippKE

Like you mentioned the data is not really gone, it has had a delete marker placed on the object itself. Thus, you should be able to remove the delete markers using the S3 CLI. All providers like AWS, wasabi etc usually have a way to script it out. This is no different. Check out this page for how to

https://wasabi-support.zendesk.com/hc/e ... -a-bucket-

Also to add, Veeam tracks the version IDs of the objects we place and when working with those objects we will call those objects by version id so the markers dont really matter, but using the method above it will remove the markers and put it back to "normal"

[PhilippKE]

Hi Dustin,
great, "delete marker" was the essential keyword I have looked for. Thank you!
I will try that in our lab.
Regards
Philipp

[dalbertson]

FYI here is the same method for AWS

AWS - https://aws.amazon.com/premiumsupport/k ... iguration/

[r.horowski]

Hi @PhilippKE and @dalbertson

I'm currently trying to prepare a procedure for my customer for that exact scenario, when somebody gets credential to object storage and deletes (or put a delete marker) objects with API or S3-browser. So I've run backup and offloaded it to S3 (Wasabi) storage with immutability. Next I connected with S3-browser and deleted one of the objects (8.part), so now this object has current version marked as deleted. When trying to restore I get error in Veeam console
ERROR:
REST API error: 'S3 error: The specified key does not exist
Code: NoSuchKey', error code: 404

I am working with support on this (Case #05545115), but for now it does not work for me.

@PhilippKE
I am wondering if you were able to test this in your lab. Please let me know.

[PhilippKE]

Hi Robert,

Veeam will not be able to see the "deleted" files, even they are just marked as deleted.
You will have to remove the deletion markers and recover the files to previous version. My example here works with Wasabi, but should also work with AWS:

Code: Select all

aws s3api list-object-versions --bucket BUCKETNAME --endpoint-url=https://s3.REGION.wasabisys.com --output json --query 'DeleteMarkers[].[Key, VersionId]' | jq -r '.[] | "--key '\''" + .[0] + "'\'' --version-id " + .[1]' |  xargs -L1 aws s3api delete-object --bucket BUCKETNAME --endpoint-url=https://s3.REGION.wasabisys.com

After removing the deletion markers, I was able to import / rescan the S3-Bucket.

BTW: I also created a Veeam Support Case, but got the same answer like you; the engineer told me, that veeam is able to recover the files even they have a deletion marker; but does not work.

Regards
Philipp

[r.horowski]

Hi Philipp,

Thanks for sharing your code with me. I will check if I am able to get my backups back using your method and post back here about results.

Regards,
Robert

[AlexHeylin]

If Veeam support say it works and two people have proved it doesn't, that seems like either a bug, a non-conformance to design, or a subject for an enhancement request.

Perhaps a PM can advise on this?

[veremin]

We have asked the QA team to re-check the given scenario. I will let you know the results of our findings.

[PhilippKE]

My Case: # 05416556

[r.horowski]

Hi PhilippKE,

I've tested your code with a very simple scenario where only 1 file was deleted and it worked. I was able to successfully restore my VM from capacity tier. I will run some more tests and try to do some more damage before restoring, but for now it looks good.

@Veremin @AlexHeylin
I also wanted to mention that my case has been escalated. I will let you know if anything new pops up.

Regards,
Robert

[veremin]

The QA team has verified the said scenario and found an issue that prevents restore from objects with deleted markers assigned. To restore from such objects, you should remove deleted markers (manually or automatically with a script). The issue is addressed in v12. Thanks again for raising this!

[TonioRoffo]

What is the attack is "uglier" and said hacker overwrites one file 5 times, the other 8, and doesn't delete anything (or he does...)

How to figure out which version of the file is the correct one?

Is there a defence against this?

Thanks

[veremin]

Using the PowerShell cmdlet, you can always revert the Scale-Out Backup Repository to one of its known previous states. However, I'm not sure how you will overwrite a locked object. Thanks!

[JWester]

The QA team has verified the said scenario and found an issue that prevents restore from objects with deleted markers assigned. To restore from such objects, you should remove deleted markers (manually or automatically with a script). The issue is addressed in v12. Thanks again for raising this!

Are you sure this issue is fixed in V12?
I just did a fresh install of V12 in a small testing environment and did a backup of 3 VMs on S3 compatible (Ceph). Then:
* I deleted the repo and imported it again. works fine as I have not deleted any data.
* I deleted the repo and all data inside the bucket (directly with a S3 browser). No backup jobs are found during import.
With the aws cli I can see that there are > 55,000 deleted objects. Recovery with a script would take hours.
Am I doing something wrong?
Thanks!
Joern

[JWester]

Ah, found the solution: When adding a new S3 repo you have to select a bucket and create a subfolder. This subfolder name has to be the same when importing the "empty" (=with deleted objects) bucket.