Discussions related to using object storage as a backup target.
Post Reply
jmc
Service Provider
Posts: 103
Liked: 9 times
Joined: Sep 12, 2011 11:49 am
Full Name: jmc
Location: Duisburg - Germany
Contact:

orphaned s3 backups

Post by jmc »

Hello everyone,

i have a tricky problem. it's about the use of an s3 compatible object storage like wasabi or idrive. i have been saving backups to the repository for a while now. as planned, immutable backups were also deleted again. but now i wanted to remove backups from veeam. for reasons not relevant here, this only worked to the extent that the backups from veeam were also removed, but unfortunately not from the s3. reimporting the backup was not successful. the backups were not recognized again. i know that the backups on the s3 were not removed, because even after weeks the storage space has not decreased significantly. it should have been only half the size.

fortunately, i have assigned a separate bucket to each backup or backup copy job in b&r. this means that only those files should currently exist on the s3 that are also known in veeam.

question:
is there a way to compare the current data of the backups in the bucket with the real ones and to define all files that are not assigned as orphaned and delete them, for example?

unfortunately i had this problem with the orphaned files several times.

unfortunately i can't simply delete the bucket or move the backups to a new one because of the amount of data.

thanks
jmc
"Who brakes late stays fast longer." - "Wer später bremst ist länger schnell"
Mildur
Product Manager
Posts: 10631
Liked: 2864 times
Joined: May 13, 2017 4:51 pm
Full Name: Fabian K.
Location: Switzerland
Contact:

Re: orphaned s3 backups

Post by Mildur »

Hello jmc
i know that the backups on the s3 were not removed, because even after weeks the storage space has not decreased significantly. it should have been only half the size.
I assume this objects are still locked (immutable)?

It's important to note that manipulating objects on object storage outside of our solutions isn't supported and may cause issues with restores and backups. I recommend reaching out to our customer support team for assistance. Please share your case number for our reference.

Best,
Fabian
Product Management Analyst @ Veeam Software
jmc
Service Provider
Posts: 103
Liked: 9 times
Joined: Sep 12, 2011 11:49 am
Full Name: jmc
Location: Duisburg - Germany
Contact:

Re: orphaned s3 backups

Post by jmc »

hello mildur,

no, they are no longer immutable. they are also completely decoupled from veeam. veeam with all backups etc. runs great. they are only files from veeam that are now simply zombies in the bucket in the event of a failure.
i am simply looking for a function that scans all the data in the veeam bucket and shows me the files that do NOT belong to the required veeam files. since veeam has all the metadata, it would not be difficult to see which ones are no longer in use. i can then delete them.

jeff
"Who brakes late stays fast longer." - "Wer später bremst ist länger schnell"
edh
Veeam Legend
Posts: 380
Liked: 118 times
Joined: Nov 02, 2020 2:48 pm
Full Name: Manuel Rios
Location: Madrid, Spain
Contact:

Re: orphaned s3 backups

Post by edh »

Orphans object are the most common issue in S3 storages. Happens with all appliances/all providers. Years ago we decommisioned a 1PB S3 Cluster and we found 200TB orphan objects.
In a deep investigation we found that our CEPH touch severaltimes a corner case where rocksdb simply delete the entry(metadata) in the database but not process the next process to physically remove the data.
Sometimes by our experience is not a problem from Veeam.
Block storage are much more clean at space cleanup than object storage.
Service Provider | VMCE
jmc
Service Provider
Posts: 103
Liked: 9 times
Joined: Sep 12, 2011 11:49 am
Full Name: jmc
Location: Duisburg - Germany
Contact:

Re: orphaned s3 backups

Post by jmc »

hello,

please don't get me wrong. everything is ok with veeam. i only put this in this forum because veeam knows its metadata of the active data and can therefore match the content of a bucket against its required data. the bucket is only for veeam and therefore all orphaned files cannot belong to any other application.

jeff
"Who brakes late stays fast longer." - "Wer später bremst ist länger schnell"
Post Reply

Who is online

Users browsing this forum: No registered users and 2 guests