Wasabi/CapacityTier: permanently growing buckets

[pirx]

We are having - again - the issue that backup data on capacity tier seems not to get deleted. We had this a couple of years ago on ASW S3 and deleted in a very time consuming process together with Veeam support >100TB. Now we see something similar on Wasabi. And this is getting expensive.

We have a retention of 10 weeks + 2 weeks immutability + 10 days block generations (?). That are ~100 days (3 months) the backups should be kept and then deleted on capacity tier. We are also seeing a reduction in VMs that needs to be backed up. After 3-4 months we should see that the utilisation does not grow constantly. I also don't see see this growth on performance tier.

There are no orphaned backups on capacity tier. If I check the objetcs with aws cli I get many from before 2024-11. But as I am not an object storage expert, I can't trace the objects back to backups.

What happens in Veeam when there is still data on a bucket that is not associated with any backup in Veeam DB? What options do I have to find data that should not be there anymore?


#1 bucket was created February 2024



#2 bucket was recreated in September as we had inconsistencies because of Wasabi API handling and they could not be solved by Veeam and Wasabi support. We had to recreate bucket and upload everything again.

[david.domask]

Hi pirx,

Thank you for sharing the details and sorry to hear about the challenges.

Do you have a Support Case to review this behavior already? If not, please create a case and be sure to include logs for Support to review, as well as details on your analysis that determined there were orphaned blocks.

The block removal is done as part of a Checkpoint removal. Rather than removing individual blocks, blocks are grouped under a logical "checkpoint", and as the checkpoint expires, it and blocks underneath the checkpoint are removed (assuming immutability period has passed, if still immutable, it will be checked on subsequent runs to see if it can be removed)

From just the above, I'm not quite sure I can point to any "smoking gun" for orphaned blocks, but it's best to work with Support on this as they will be able to tell more from the debug logs.

Thanks!

[pirx]

Ho David, I've not yet created a case as this will be very time consuming. As customer I just want let R&D be aware that the handling of capacity tier data still has much potential. In VBR and in Veeam One. For me it's really hard to get good data to work with. I have several cases the last years where support found unexpected leftover backups on capacity tier. This just seems to happen over an over again - and gets real expensive for us.

I now used following script from oleg.feoktistov to get some information about immutable backup on one SOBR

post485887.html#p485887

Not sure how I should interprete the result. Overall the there are 15500 backups in this SOBR capacity tier. ~3800 have a negative remaining immutability time. But the oldest CreationTimeUtc seems to match ~100 days and a lot of them even CreationTimeUtc within the past weeks (no sure about the negative immutability then, maybe it gets prolongt again).

[david.domask]

Hi pirx,

Thanks for the extra details -- barring that retention is set very long for these backups, I would agree it looks unexpected, but unfortunately for more specific information a case will be required. The checkpoint removal and retention are both logged quite dutifully, and we should be able to get more information (especially with the script output) about what happened during the retention/checkpoint removal.

Just a question though, do you see these backups in the UI under Backups > Object Storage? (you can check the file name in the Properties) That would mean Veeam is still aware of them, so understanding why they were not removed by retention would be the main focus.

[pirx]

The ones from the script I can see.



I could manually delete the above backup.

13.02.2025 14:34:34 [xxxx - xxx4106] Backup has been removed successfully



There are 2 different things: backups that are outside the expected retention/immutability period and data on buckets that are not visible in Veeam. My cli dump of objects in the buckets showed even much older data.

I understand that in the end will have to open a support case. But I also think that Veeam should improve the transparency of data on capacity tier. I'm struggling with this for years now. Object Storage with its small objects (nearly 1.000.000 in out case for our largest buckets) is a complete black box. There is no filesystem. For performance tier, I just look in the filesystem, search for files older than xxx or size bigger than xxx and I immediately know what is going on. Here I'm 100% lost without support.

But maybe I'm just not aware of the right reports, scripts, tools. How can I debug this? I did not find much useful in VO reports.

[veremin]

We're really sorry to hear that your issues with undeletable backups are still persisting. I understand that you've gone through multiple support cases, and they might be quite frustrating by now. However, without debug logs and a thorough investigation, it will be difficult for us to pinpoint the root causes of your problems.

If you still decide to open a case, we can promise to escalate it almost immediately to the R&D team for further investigation, so it won't remain at Tier 1 or Tier 2 levels for long.

Thank you for your understanding.