Discussions related to using object storage as a backup target.
Post Reply
tgregorius
Veeam Vanguard
Posts: 5
Liked: 2 times
Joined: Sep 25, 2014 10:26 am
Full Name: Tobias Gregorius
Contact:

Feature Idea: S3 Versioning support

Post by tgregorius »

Hi,

If S3 compatible object storage with the immutabillity function activated, "only" the one version is marked as "undeletable". This has a decisive factor from Veeam's point of view: if you access the bucket via an S3 browser, all objects can be marked as "deleted". This action prevents Veeam from accessing the backup data. To make these objects readable for Veeam again, a manual restore of the "immutable" objects to the status "current" is necessary. From my point of view, it would make sense if Veeam could directly access this versioning and thus a change of the objects with other tools has no effect on the Veeam backups.

cheers
Tobias
HannesK
Product Manager
Posts: 14287
Liked: 2877 times
Joined: Sep 01, 2014 11:46 am
Full Name: Hannes Kasparick
Location: Austria
Contact:

Re: Feature Idea: S3 Versioning support

Post by HannesK »

Hello Tobias,
just to clarify the goal and the situation.

The goal is to simplify restore after someone deleted data in an immutable bucket, right? You want to get rid of error messages like this one?

Image

What I did is deleting the whole backup "folder" in S3 of my capacity tier.

Best regards,
Hannes
tgregorius
Veeam Vanguard
Posts: 5
Liked: 2 times
Joined: Sep 25, 2014 10:26 am
Full Name: Tobias Gregorius
Contact:

Re: Feature Idea: S3 Versioning support

Post by tgregorius »

Hi Hannes,

correct.

reegards
Tobias
HannesK
Product Manager
Posts: 14287
Liked: 2877 times
Joined: Sep 01, 2014 11:46 am
Full Name: Hannes Kasparick
Location: Austria
Contact:

Re: Feature Idea: S3 Versioning support

Post by HannesK »

Hello Tobias,
conversations are still going on, I will come back once I have a final answer.

Best regards,
Hannes
Gostev
Chief Product Officer
Posts: 31460
Liked: 6648 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Feature Idea: S3 Versioning support

Post by Gostev »

But honestly, how often does one delete a bucket content in an S3 browser to justify implementing and forever-maintaining the perfect experience around this use case? I would agree if this was something backup admins had to face or do at least yearly as a part of some standard process. But this cannot possibly be the case here? We're talking some truly exceptional situation here.
HannesK
Product Manager
Posts: 14287
Liked: 2877 times
Joined: Sep 01, 2014 11:46 am
Full Name: Hannes Kasparick
Location: Austria
Contact:

Re: Feature Idea: S3 Versioning support

Post by HannesK »

I would say, that happens as often as the backup server gets hacked and the attacker is motivated to also destroy the S3 storage :-)

QA said that the software should be able to handle it automatically. So we are just trying to find out why it's not working.
Gostev
Chief Product Officer
Posts: 31460
Liked: 6648 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Feature Idea: S3 Versioning support

Post by Gostev »

It's not automatic for sure, there's the dedicated PowerShell cmdlet for this. There's an existing topic in this subforum where I explained all details about 1 year ago.
veremin
Product Manager
Posts: 20270
Liked: 2252 times
Joined: Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin
Contact:

Re: Feature Idea: S3 Versioning support

Post by veremin » 2 people like this post

The said cmdlet helps, if an attacker "nullifies" the restore points by shortening job retention. In this case an user can revert the Scale-Out Backup Repository to its previous state (still preserved by immutability).

However, if the attacker removes S3 bucket using S3 browser, the cmdlet will be of no help.

In this case the user will need to locate objects in removed folder using S3 browser (those objects will have "deleted" marker assigned) and remove this marker. After the user can add the object storage repository and import backups from it.

We have decided created a KB article which will describe the different types of potential attacks on immutable object storage repository and means to recover a data from it.

I will update the topic, once the KB article is ready.

Thanks!
Gostev
Chief Product Officer
Posts: 31460
Liked: 6648 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Feature Idea: S3 Versioning support

Post by Gostev »

Removing markers from billions of objects manually is undoable though? Need some script I guess, or should be a part of the existing cmdlet functionality.
veremin
Product Manager
Posts: 20270
Liked: 2252 times
Joined: Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin
Contact:

Re: Feature Idea: S3 Versioning support

Post by veremin »

Should be doable via scripts, correct, we will provide some examples within the KB article.
TonioRoffo
Enthusiast
Posts: 52
Liked: 5 times
Joined: Jun 18, 2009 2:27 pm
Full Name: Yves Smolders
Contact:

Re: Feature Idea: S3 Versioning support

Post by TonioRoffo »

Gostev wrote: Mar 24, 2022 2:02 pm Removing markers from billions of objects manually is undoable though? Need some script I guess, or should be a part of the existing cmdlet functionality.
Also, how long would this take - usually when you go here, you are in disaster recovery mode and every second counts.
veremin
Product Manager
Posts: 20270
Liked: 2252 times
Joined: Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin
Contact:

Re: Feature Idea: S3 Versioning support

Post by veremin » 2 people like this post

We have re-tested this scenario and found that you cannot remove an S3 bucket if it hosts at least one immutable object. So an attacker can only try to remove the objects inside the bucket. Such objects will not be deleted but marked with a "delete" marker.

Currently, a product issue prevents restoration from objects with deleted markers. To restore from such objects, you should remove deleted markers (manually or automatically with a script). However, the issue will be addressed in v12 - v12 will be able to restore from the objects even if they have delete markers assigned.

Thanks!
ksl28
Enthusiast
Posts: 48
Liked: 8 times
Joined: Sep 21, 2016 8:31 am
Full Name: Kristian Leth
Contact:

Re: Feature Idea: S3 Versioning support

Post by ksl28 » 1 person likes this post

veremin wrote: Mar 24, 2022 3:00 pm Should be doable via scripts, correct, we will provide some examples within the KB article.
Hi,
Could you provide a link to these KB articles?
And will these scripts work with any type of S3 vendor that supports immutable objects (Wasabi, Amazon, etc)?

We just did a PoC on this, and was shocked that we could delete the objects, untill i found this forum post :)
veremin
Product Manager
Posts: 20270
Liked: 2252 times
Joined: Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin
Contact:

Re: Feature Idea: S3 Versioning support

Post by veremin »

Post Reply

Who is online

Users browsing this forum: No registered users and 11 guests