Feature Idea: S3 Versioning support

[tgregorius]

Hi,

If S3 compatible object storage with the immutabillity function activated, "only" the one version is marked as "undeletable". This has a decisive factor from Veeam's point of view: if you access the bucket via an S3 browser, all objects can be marked as "deleted". This action prevents Veeam from accessing the backup data. To make these objects readable for Veeam again, a manual restore of the "immutable" objects to the status "current" is necessary. From my point of view, it would make sense if Veeam could directly access this versioning and thus a change of the objects with other tools has no effect on the Veeam backups.

cheers
Tobias

[HannesK]

Hello Tobias,
just to clarify the goal and the situation.

The goal is to simplify restore after someone deleted data in an immutable bucket, right? You want to get rid of error messages like this one?



What I did is deleting the whole backup "folder" in S3 of my capacity tier.

Best regards,
Hannes

[tgregorius]

Hi Hannes,

correct.

reegards
Tobias

[HannesK]

Hello Tobias,
conversations are still going on, I will come back once I have a final answer.

Best regards,
Hannes

[Gostev]

But honestly, how often does one delete a bucket content in an S3 browser to justify implementing and forever-maintaining the perfect experience around this use case? I would agree if this was something backup admins had to face or do at least yearly as a part of some standard process. But this cannot possibly be the case here? We're talking some truly exceptional situation here.

[HannesK]

I would say, that happens as often as the backup server gets hacked and the attacker is motivated to also destroy the S3 storage

QA said that the software should be able to handle it automatically. So we are just trying to find out why it's not working.

[Gostev]

It's not automatic for sure, there's the dedicated PowerShell cmdlet for this. There's an existing topic in this subforum where I explained all details about 1 year ago.

[veremin]

The said cmdlet helps, if an attacker "nullifies" the restore points by shortening job retention. In this case an user can revert the Scale-Out Backup Repository to its previous state (still preserved by immutability).

However, if the attacker removes S3 bucket using S3 browser, the cmdlet will be of no help.

In this case the user will need to locate objects in removed folder using S3 browser (those objects will have "deleted" marker assigned) and remove this marker. After the user can add the object storage repository and import backups from it.

We have decided created a KB article which will describe the different types of potential attacks on immutable object storage repository and means to recover a data from it.

I will update the topic, once the KB article is ready.

Thanks!

[Gostev]

Removing markers from billions of objects manually is undoable though? Need some script I guess, or should be a part of the existing cmdlet functionality.

[veremin]

Should be doable via scripts, correct, we will provide some examples within the KB article.

[TonioRoffo]

Removing markers from billions of objects manually is undoable though? Need some script I guess, or should be a part of the existing cmdlet functionality.

Also, how long would this take - usually when you go here, you are in disaster recovery mode and every second counts.

[veremin]

We have re-tested this scenario and found that you cannot remove an S3 bucket if it hosts at least one immutable object. So an attacker can only try to remove the objects inside the bucket. Such objects will not be deleted but marked with a "delete" marker.

Currently, a product issue prevents restoration from objects with deleted markers. To restore from such objects, you should remove deleted markers (manually or automatically with a script). However, the issue will be addressed in v12 - v12 will be able to restore from the objects even if they have delete markers assigned.

Thanks!

[ksl28]

Should be doable via scripts, correct, we will provide some examples within the KB article.

Hi,
Could you provide a link to these KB articles?
And will these scripts work with any type of S3 vendor that supports immutable objects (Wasabi, Amazon, etc)?

We just did a PoC on this, and was shocked that we could delete the objects, untill i found this forum post

[veremin]

You can find the scripts below:
Removing delete markers from Wasabi Hot Cloud Storage
Removing delete markers from AWS S3

Thanks!