S3 obect lock (not) bulletproof ?

[klemens]

A few weeks ago I did some initial tests of a SOBR with an object lock enabled WASABI test bucket (Make recent backups immutable for n days). After the backup and a few restore tests I overwrote some objects in the S3 bucket with a new object of the same name (object lock requires versioning). As expected, the object created by Veeam was no longer the primary version, but theoretically retrievable under the old version ID. From this point on, restores containing the "manipulated" block could no longer be performed. Apparently the version id of an object is not kept in the backup metadata. The consistency of the backup could be restored by making the old versioned object the current object again via Get(Version) and Put.
The test indicates that the information itself cannot be destroyed, but a targeted attack on such a bucket by overwriting a few thousand objects can damage the S3 tier to such an extent that recovery becomes extremely difficult, since finding the correct version ID and restoring the "correct" version on the part of the S3 API is not trivial and also requires sufficient bandwidth (Get/Put of all affected objects).

klemens

[Andreas Neufert]

Hello Klemens, thanks for brining this point up.
Your scenario should not be any problem to Veeam as we of cause store the version of an object within our metadata.
So object changes should not lead to any issue.

There is one exception and this is when our version is marked as deleted when someone manually attempted to delete the objects. The object is kept protected but kind of hidden then. In this case you need to delete the deletemarker currently first. Wasabi has documented this here:
https://wasabi-support.zendesk.com/hc/e ... -a-bucket-
We will apply some automation around this in v12 to smoothen this process for customers.

I checked your organizations call history and did not find any Veeam support case where your situation was analyzed. Can you please share the support ticket number here so that we can look into this together?

[veremin]

The test indicates that the information itself cannot be destroyed, but a targeted attack on such a bucket by overwriting a few thousand objects can damage the S3 tier to such an extent that recovery becomes extremely difficult, since finding the correct version ID and restoring the "correct" version on the part of the S3 API is not trivial and also requires sufficient bandwidth

No need to find the correct version manually, as you can revert a Scale-Out Backup Repository to one of its previous states using this cmdlet. Thanks!

[veremin]

Andreas is correct here, in the provided example the backup server should be aware of the specific object version (it worked with), so there should not be any issues during the restore process. If you see different behavior, kindly reach our support team for further investigation:

Your scenario should not be any problem to Veeam as we of cause store the version of an object within our metadata.
So object changes should not lead to any issue.

Thanks!

[Andreas Neufert]

What might have happened is that you tool you used to instead of creating another object version has overwritten (deleting) the old object. Then this object would have become a "deleted" flag and is not visible anymore (but still there and protected). Wasabi has the above linked method to bring back the data. Veeam would then work normally.
We will automate some things with v12 to make this process more convenient.