orphaned s3 backups

[jmc]

Hello everyone,

i have a tricky problem. it's about the use of an s3 compatible object storage like wasabi or idrive. i have been saving backups to the repository for a while now. as planned, immutable backups were also deleted again. but now i wanted to remove backups from veeam. for reasons not relevant here, this only worked to the extent that the backups from veeam were also removed, but unfortunately not from the s3. reimporting the backup was not successful. the backups were not recognized again. i know that the backups on the s3 were not removed, because even after weeks the storage space has not decreased significantly. it should have been only half the size.

fortunately, i have assigned a separate bucket to each backup or backup copy job in b&r. this means that only those files should currently exist on the s3 that are also known in veeam.

question:
is there a way to compare the current data of the backups in the bucket with the real ones and to define all files that are not assigned as orphaned and delete them, for example?

unfortunately i had this problem with the orphaned files several times.

unfortunately i can't simply delete the bucket or move the backups to a new one because of the amount of data.

thanks
jmc

[Mildur]

Hello jmc

i know that the backups on the s3 were not removed, because even after weeks the storage space has not decreased significantly. it should have been only half the size.

I assume this objects are still locked (immutable)?

It's important to note that manipulating objects on object storage outside of our solutions isn't supported and may cause issues with restores and backups. I recommend reaching out to our customer support team for assistance. Please share your case number for our reference.

Best,
Fabian

[jmc]

hello mildur,

no, they are no longer immutable. they are also completely decoupled from veeam. veeam with all backups etc. runs great. they are only files from veeam that are now simply zombies in the bucket in the event of a failure.
i am simply looking for a function that scans all the data in the veeam bucket and shows me the files that do NOT belong to the required veeam files. since veeam has all the metadata, it would not be difficult to see which ones are no longer in use. i can then delete them.

jeff

[edh]

Orphans object are the most common issue in S3 storages. Happens with all appliances/all providers. Years ago we decommisioned a 1PB S3 Cluster and we found 200TB orphan objects.
In a deep investigation we found that our CEPH touch severaltimes a corner case where rocksdb simply delete the entry(metadata) in the database but not process the next process to physically remove the data.
Sometimes by our experience is not a problem from Veeam.
Block storage are much more clean at space cleanup than object storage.

[jmc]

hello,

please don't get me wrong. everything is ok with veeam. i only put this in this forum because veeam knows its metadata of the active data and can therefore match the content of a bucket against its required data. the bucket is only for veeam and therefore all orphaned files cannot belong to any other application.

jeff