-
- Novice
- Posts: 5
- Liked: 2 times
- Joined: Sep 25, 2014 10:26 am
- Full Name: Tobias Gregorius
- Contact:
Feature Idea: S3 Versioning support
Hi,
If S3 compatible object storage with the immutabillity function activated, "only" the one version is marked as "undeletable". This has a decisive factor from Veeam's point of view: if you access the bucket via an S3 browser, all objects can be marked as "deleted". This action prevents Veeam from accessing the backup data. To make these objects readable for Veeam again, a manual restore of the "immutable" objects to the status "current" is necessary. From my point of view, it would make sense if Veeam could directly access this versioning and thus a change of the objects with other tools has no effect on the Veeam backups.
cheers
Tobias
If S3 compatible object storage with the immutabillity function activated, "only" the one version is marked as "undeletable". This has a decisive factor from Veeam's point of view: if you access the bucket via an S3 browser, all objects can be marked as "deleted". This action prevents Veeam from accessing the backup data. To make these objects readable for Veeam again, a manual restore of the "immutable" objects to the status "current" is necessary. From my point of view, it would make sense if Veeam could directly access this versioning and thus a change of the objects with other tools has no effect on the Veeam backups.
cheers
Tobias
-
- Product Manager
- Posts: 14585
- Liked: 2959 times
- Joined: Sep 01, 2014 11:46 am
- Full Name: Hannes Kasparick
- Location: Austria
- Contact:
Re: Feature Idea: S3 Versioning support
Hello Tobias,
just to clarify the goal and the situation.
The goal is to simplify restore after someone deleted data in an immutable bucket, right? You want to get rid of error messages like this one?
![Image](https://snipboard.io/kvVEth.jpg)
What I did is deleting the whole backup "folder" in S3 of my capacity tier.
Best regards,
Hannes
just to clarify the goal and the situation.
The goal is to simplify restore after someone deleted data in an immutable bucket, right? You want to get rid of error messages like this one?
![Image](https://snipboard.io/kvVEth.jpg)
What I did is deleting the whole backup "folder" in S3 of my capacity tier.
Best regards,
Hannes
-
- Novice
- Posts: 5
- Liked: 2 times
- Joined: Sep 25, 2014 10:26 am
- Full Name: Tobias Gregorius
- Contact:
Re: Feature Idea: S3 Versioning support
Hi Hannes,
correct.
reegards
Tobias
correct.
reegards
Tobias
-
- Product Manager
- Posts: 14585
- Liked: 2959 times
- Joined: Sep 01, 2014 11:46 am
- Full Name: Hannes Kasparick
- Location: Austria
- Contact:
Re: Feature Idea: S3 Versioning support
Hello Tobias,
conversations are still going on, I will come back once I have a final answer.
Best regards,
Hannes
conversations are still going on, I will come back once I have a final answer.
Best regards,
Hannes
-
- Chief Product Officer
- Posts: 31419
- Liked: 6992 times
- Joined: Jan 01, 2006 1:01 am
- Location: Baar, Switzerland
- Contact:
Re: Feature Idea: S3 Versioning support
But honestly, how often does one delete a bucket content in an S3 browser to justify implementing and forever-maintaining the perfect experience around this use case? I would agree if this was something backup admins had to face or do at least yearly as a part of some standard process. But this cannot possibly be the case here? We're talking some truly exceptional situation here.
-
- Product Manager
- Posts: 14585
- Liked: 2959 times
- Joined: Sep 01, 2014 11:46 am
- Full Name: Hannes Kasparick
- Location: Austria
- Contact:
Re: Feature Idea: S3 Versioning support
I would say, that happens as often as the backup server gets hacked and the attacker is motivated to also destroy the S3 storage ![Smile :-)](https://forums.veeam.com/images/smilies/icon_smile.gif)
QA said that the software should be able to handle it automatically. So we are just trying to find out why it's not working.
![Smile :-)](https://forums.veeam.com/images/smilies/icon_smile.gif)
QA said that the software should be able to handle it automatically. So we are just trying to find out why it's not working.
-
- Chief Product Officer
- Posts: 31419
- Liked: 6992 times
- Joined: Jan 01, 2006 1:01 am
- Location: Baar, Switzerland
- Contact:
Re: Feature Idea: S3 Versioning support
It's not automatic for sure, there's the dedicated PowerShell cmdlet for this. There's an existing topic in this subforum where I explained all details about 1 year ago.
-
- Product Manager
- Posts: 20297
- Liked: 2265 times
- Joined: Oct 26, 2012 3:28 pm
- Full Name: Vladimir Eremin
- Contact:
Re: Feature Idea: S3 Versioning support
The said cmdlet helps, if an attacker "nullifies" the restore points by shortening job retention. In this case an user can revert the Scale-Out Backup Repository to its previous state (still preserved by immutability).
However, if the attacker removes S3 bucket using S3 browser, the cmdlet will be of no help.
In this case the user will need to locate objects in removed folder using S3 browser (those objects will have "deleted" marker assigned) and remove this marker. After the user can add the object storage repository and import backups from it.
We have decided created a KB article which will describe the different types of potential attacks on immutable object storage repository and means to recover a data from it.
I will update the topic, once the KB article is ready.
Thanks!
However, if the attacker removes S3 bucket using S3 browser, the cmdlet will be of no help.
In this case the user will need to locate objects in removed folder using S3 browser (those objects will have "deleted" marker assigned) and remove this marker. After the user can add the object storage repository and import backups from it.
We have decided created a KB article which will describe the different types of potential attacks on immutable object storage repository and means to recover a data from it.
I will update the topic, once the KB article is ready.
Thanks!
-
- Chief Product Officer
- Posts: 31419
- Liked: 6992 times
- Joined: Jan 01, 2006 1:01 am
- Location: Baar, Switzerland
- Contact:
Re: Feature Idea: S3 Versioning support
Removing markers from billions of objects manually is undoable though? Need some script I guess, or should be a part of the existing cmdlet functionality.
-
- Product Manager
- Posts: 20297
- Liked: 2265 times
- Joined: Oct 26, 2012 3:28 pm
- Full Name: Vladimir Eremin
- Contact:
Re: Feature Idea: S3 Versioning support
Should be doable via scripts, correct, we will provide some examples within the KB article.
-
- Enthusiast
- Posts: 56
- Liked: 6 times
- Joined: Jun 18, 2009 2:27 pm
- Full Name: Yves Smolders
- Contact:
Re: Feature Idea: S3 Versioning support
Also, how long would this take - usually when you go here, you are in disaster recovery mode and every second counts.
-
- Product Manager
- Posts: 20297
- Liked: 2265 times
- Joined: Oct 26, 2012 3:28 pm
- Full Name: Vladimir Eremin
- Contact:
Re: Feature Idea: S3 Versioning support
We have re-tested this scenario and found that you cannot remove an S3 bucket if it hosts at least one immutable object. So an attacker can only try to remove the objects inside the bucket. Such objects will not be deleted but marked with a "delete" marker.
Currently, a product issue prevents restoration from objects with deleted markers. To restore from such objects, you should remove deleted markers (manually or automatically with a script). However, the issue will be addressed in v12 - v12 will be able to restore from the objects even if they have delete markers assigned.
Thanks!
Currently, a product issue prevents restoration from objects with deleted markers. To restore from such objects, you should remove deleted markers (manually or automatically with a script). However, the issue will be addressed in v12 - v12 will be able to restore from the objects even if they have delete markers assigned.
Thanks!
-
- Enthusiast
- Posts: 48
- Liked: 8 times
- Joined: Sep 21, 2016 8:31 am
- Full Name: Kristian Leth
- Contact:
Re: Feature Idea: S3 Versioning support
Hi,
Could you provide a link to these KB articles?
And will these scripts work with any type of S3 vendor that supports immutable objects (Wasabi, Amazon, etc)?
We just did a PoC on this, and was shocked that we could delete the objects, untill i found this forum post
![Smile :)](https://forums.veeam.com/images/smilies/icon_smile.gif)
-
- Product Manager
- Posts: 20297
- Liked: 2265 times
- Joined: Oct 26, 2012 3:28 pm
- Full Name: Vladimir Eremin
- Contact:
Re: Feature Idea: S3 Versioning support
You can find the scripts below:
Removing delete markers from Wasabi Hot Cloud Storage
Removing delete markers from AWS S3
Thanks!
Removing delete markers from Wasabi Hot Cloud Storage
Removing delete markers from AWS S3
Thanks!
Who is online
Users browsing this forum: No registered users and 4 guests