Large number of delete markers

[flomp]

I am running Veeam v11 11.0.1.1261 P20230227 with a SOBR to Wasabi for a bit more than a year now.
Repository size is about 3.5TB

Out of curiosity, I wrote a small script which calls `ListObjectVersions` to retrieve all objects in the repo.
I had to abort it, because it would have taken a day or so. However, after about an hour, I had the following statistics:

333723 versions, 821277 delete markers

Is it normal, that the bucket contains that many delete markers?

[Mildur]

Hello Flomp

We had a known issue in V11 around those delete markers. This issue is fixed in V12.
Please open a case with our customer support to check if you are affected by the issue and to receive guidance to clean up those delete markers. Don't forget to provide us with the case number.

Best,
Fabian

[flomp]

Thank you Fabian!

I can open a ticket - however, I am thinking about doing a clean install of v12 in the near future anyway, because we will move to another Wasabi tenant and also because I want to move to Postgres.

Therefore, does it make sense to open a ticket in that case?

[Mildur]

You're welcome.
If you plan to move to another tenant with new buckets, I would say you don't need to open a case.

Best,
Fabian

[tyler.jurgens]

I wrote a blog post about this exact issue:
https://explosive.cloud/minio-and-veeam ... e-markers/

I have to tweak some spots on the post, but the meat of the issue is covered there.

[AM1]

Hi,

If we have upgraded to the latest V12 from V11, will this cleanup the delete markers automaically in object storage, or do we need to contact support for help and the script?

Thanks,
Andreas

[tyler.jurgens]

No, the delete markers will remain until you perform steps to clean it up. Eg: ILM Policies (if you are running your own S3, eg: Minio), a cleanup script (as mentioned in my blog post above or by contacting support), or creating a new bucket and starting new backups to that using Veeam v12, or a patched v11. Or you could write your own script I suppose. I've also debated using ILM policies and writing a script to loop through the folders of the bucket to list contents, which would kick off the ILM policy faster than just letting it run normally.

Short of deleting the buckets and starting fresh, the script has worked well for me so far. That said, I have some buckets with over 100,000,000 delete markers in them and I've only ran it on buckets with a smaller amount (~50,000,000 delete markers). I've also found that the script needs to be run multiple times and it takes a very long time to complete (think weeks/months). For a bucket with 50,000,000 delete markers it's taken over 3 weeks to run through the first time. I'm currently running through it a second time.

Keep in mind, I'm running it on my own Minio buckets, so I don't need to worry about S3 API costs. YMMV.