Discussions related to using object storage as a backup target.
Post Reply
klemens
Lurker
Posts: 1
Liked: never
Joined: Nov 16, 2016 4:54 pm
Full Name: Klemens URBAN
Contact:

S3 obect lock (not) bulletproof ?

Post by klemens »

A few weeks ago I did some initial tests of a SOBR with an object lock enabled WASABI test bucket (Make recent backups immutable for n days). After the backup and a few restore tests I overwrote some objects in the S3 bucket with a new object of the same name (object lock requires versioning). As expected, the object created by Veeam was no longer the primary version, but theoretically retrievable under the old version ID. From this point on, restores containing the "manipulated" block could no longer be performed. Apparently the version id of an object is not kept in the backup metadata. The consistency of the backup could be restored by making the old versioned object the current object again via Get(Version) and Put.
The test indicates that the information itself cannot be destroyed, but a targeted attack on such a bucket by overwriting a few thousand objects can damage the S3 tier to such an extent that recovery becomes extremely difficult, since finding the correct version ID and restoring the "correct" version on the part of the S3 API is not trivial and also requires sufficient bandwidth (Get/Put of all affected objects).

klemens
Andreas Neufert
VP, Product Management
Posts: 6707
Liked: 1401 times
Joined: May 04, 2011 8:36 am
Full Name: Andreas Neufert
Location: Germany
Contact:

Re: S3 obect lock (not) bulletproof ?

Post by Andreas Neufert »

Hello Klemens, thanks for brining this point up.
Your scenario should not be any problem to Veeam as we of cause store the version of an object within our metadata.
So object changes should not lead to any issue.

There is one exception and this is when our version is marked as deleted when someone manually attempted to delete the objects. The object is kept protected but kind of hidden then. In this case you need to delete the deletemarker currently first. Wasabi has documented this here:
https://wasabi-support.zendesk.com/hc/e ... -a-bucket-
We will apply some automation around this in v12 to smoothen this process for customers.

I checked your organizations call history and did not find any Veeam support case where your situation was analyzed. Can you please share the support ticket number here so that we can look into this together?
veremin
Product Manager
Posts: 20270
Liked: 2252 times
Joined: Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin
Contact:

Re: S3 obect lock (not) bulletproof ?

Post by veremin »

The test indicates that the information itself cannot be destroyed, but a targeted attack on such a bucket by overwriting a few thousand objects can damage the S3 tier to such an extent that recovery becomes extremely difficult, since finding the correct version ID and restoring the "correct" version on the part of the S3 API is not trivial and also requires sufficient bandwidth
No need to find the correct version manually, as you can revert a Scale-Out Backup Repository to one of its previous states using this cmdlet. Thanks!
veremin
Product Manager
Posts: 20270
Liked: 2252 times
Joined: Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin
Contact:

Re: S3 obect lock (not) bulletproof ?

Post by veremin »

Andreas is correct here, in the provided example the backup server should be aware of the specific object version (it worked with), so there should not be any issues during the restore process. If you see different behavior, kindly reach our support team for further investigation:
Andreas Neufert wrote:Your scenario should not be any problem to Veeam as we of cause store the version of an object within our metadata.
So object changes should not lead to any issue.
Thanks!
Andreas Neufert
VP, Product Management
Posts: 6707
Liked: 1401 times
Joined: May 04, 2011 8:36 am
Full Name: Andreas Neufert
Location: Germany
Contact:

Re: S3 obect lock (not) bulletproof ?

Post by Andreas Neufert »

What might have happened is that you tool you used to instead of creating another object version has overwritten (deleting) the old object. Then this object would have become a "deleted" flag and is not visible anymore (but still there and protected). Wasabi has the above linked method to bring back the data. Veeam would then work normally.
We will automate some things with v12 to make this process more convenient.
Post Reply

Who is online

Users browsing this forum: MrSpock and 26 guests