Object Size Usage

[tyler.jurgens]

I've been digging into object storage a lot more lately and came across something curious.

I have a Minio cluster setup and I am performing direct to object storage backups (no backup copies, no SOBR offloading).

My Minio bucket is setup without immutability or versioning. About as plain as it can get.


My backup job is setup with storage optimization set to 4 MB and retention policy set to 3 restore points - no GFS policy set. Basically default settings, with the restore points and storage optimization changed. I wanted to get a sense of storage usage over time and object sizes over time without complicating matters with immutability. After the initial backup it looks pretty standard - lots of 4 MB objects, few 1 MB objects and a smattering of under 1024 B objects. Pretty much what I would expect to see.

The curiosity came in after the retention policy started to prune out older restore points. The 4 MB and 1 MB objects remained relatively similar (the bumps in the graphs are when new backups are taken, before the old ones are aged out), but the under 1024 B objects started increasing significantly. As I understand it, that's all metadata, but what I'm confused about is why the significant growth of metadata blocks just for retention? If this trend continues, the small blocks will quickly outpace all other block counts. I would expect the metadata would grow a bit, but I would expect it to level off at some point.


Is this expected to have the under 1024 B objects grow continuously? Will this ever level off, or will this continue to grow significantly?

[tyler.jurgens]

So far no end in sight for small object growth. Is Veeam not pruning out old metadata? Could the metadata be stored in larger objects if this is required (More metadata per object, rather than tiny metadata files)?

Really curious for an answer here, its not adding up to me.

[HannesK]

Hello,
are you using V11 or V12?

Best regards,
Hannes

[tyler.jurgens]

Hi Hannes. I'm on v12 - doing direct to object storage.

One backup job containing 13 VMs targeting a Minio bucket.

[tyler.jurgens]

12.0.0.1420 P20230412

[HannesK]

Hello,
hmm okay... the question is, what these small objects are. If we know that, we might be able to give an answer (from the description itself, it's hard to guess). Veeam support should also be able to help to find out the details (please post the case number for reference if you open a case)

Best regards,
Hannes

[tyler.jurgens]

Looks like these are deleted file markers.


Still trying to get a ticket open, I'll post when I have the ticket number.

Minio support states they wouldn't clean these up unless explicitly requested by the client, in this case Veeam.

[tyler.jurgens]

Case #06154552

[tyler.jurgens]

I heard back from Veeam, this was their response:

From the min.io support documentation we can see the following about that status:

For objects created while versioning is disabled or suspended, MinIO uses a null version ID. You can access or remove these objects by specifying null as the version ID as part of S3 operations.
https://min.io/docs/minio/linux/adminis ... oning.html

When not using immutability through Veeam, we do not specify the version ID as we do not expect there to be one. When using minio, a version ID is set to null.

This is causing this build up of deleted file markers which appear as the objects smaller that 1024B, as you suspected.

So apparently Veeam does not properly support using Minio when the backup repository is created without any immutability settings as Veeam will orphan a bunch of small deleted file markers. In my mind, this is a Veeam bug. However, even if its working as intended, its still left us with a sizeable problem.

I checked some of our newer Veeam customers (hence newly created buckets, all with immutability) and we get very few under 1024 B objects (500-5000 objects - nothing at all really).
When I check some of our older Veeam customers, even ones with immutability enabled, I see an excessive amount of small objects. When I check the object browsers, I see a great many 'delete markers' in their bucket. What I can guess is that we had some customers using our Minio S3 before immutability was enabled on their repositories - the delete markers started growing and then at some point they got changed to immutable buckets and the delete markers stopped getting created.

I now understand we can't use Buckets without Object Lock/Versioning and Veeam due to this issue. Going forward we will only setup Veeam repositories using Minio buckets with Object Lock/Immutability. There are likely some good use cases for not enabling immutability, but if doing so will cause this problem we simply can't utilize Veeam with Minio buckets in this way. Lessons learned here!

Now I have a problem though. We have multiple buckets with these delete markers in them that I don't know how or if I can clean them up.

Do you have any suggestions here? I know we could create a new bucket, have customers back up to that instead with the proper immutability set on the repository, but that leaves us with these old buckets that we may not be able to delete for a long time (when the customer no longer needs the backups in that bucket). Not to mention how we sort out billing on this issue.

Can we clean up these delete markers? Given only Veeam understands the data in these buckets, pruning outside of Veeam is risky at best.

[tyler.jurgens]

Minio support recommends creating an ILM rule to remove dangling DEL markers (that is to say, deleted versions where there is no longer any object left).

Code: Select all

mc ilm rule add --expire-delete-marker <alias>/<bucket>

Is this a safe operation for Veeam data? It looks safe to me, but I need to be certain before I role this out to other buckets.

[HannesK]

Hello,
my understanding of the scenario is, that the configuration is unsupported. In one of the screenshots in the case there is an "suspended" versioning setting. My minio shows "unversioned"



How did it happen, that your versioning is "suspended" and do you see the same small objects on buckets where versioning is "unversioned"?

Best regards,
Hannes

[tyler.jurgens]

Not sure how that became suspended either, tbh. I created the bucket with default settings, that's what happened. Its possible I enabled then disabled it, which doesn't turn it to unversioned but rather turns it to suspended.

Not really the point though anymore. I know how to avoid this (only use immutability with Minio buckets from now on). Perhaps using an "unversioned" bucket would work better? I can always test. That said, not really the point, as I know what to avoid going forward. As mentioned, lesson learned here.

The real problem is we have buckets experiencing the same under 1024 B dangling DEL markers and I need to know how to clean them up. I suspect we had these buckets created either with suspended versioning, or with versioning enabled but immutability disabled at the customer end and now we have a these tiny objects that need to be cleaned up.

[HannesK]

I see no reason, why delete markers could be used by us for mutable storage. we only follow the markers for restore from immutable storage.

yeah, turning on / off immutability is something the software does not expect, because AWS does not allow that.

[tyler.jurgens]

To be honest, I'm really glad I made the mistake of enabling/suspending the versioning on my test bucket. Let me explain.

We have seen multiple Veeam buckets with an excessive amount of "Less than 1024 B" objects and up until I started digging, assumed this was just how Veeam worked. I understand Veeam can create these small objects through regular use, but what we were seeing didn't add up. During the course of troubleshooting with support, they advised *my* specific bucket setup was not unsupported. That's fine, making one mistake in my many years on earth is acceptable.

However, when I pushed for a way to clean these dangling DeleteMarkers up, Veeam support provided me with a script that came as part of a v11 patch. It looks like there was a bug in v11 that would leave these DeleteMarkers around, even with a supported bucket configuration. With that, everything made sense. We have multiple customers that have used us since long before v12 came along, and those buckets are filled with these DeleteMarkers. Once they upgraded to v12, the DeleteMarkers stopped being created.

I didn't know with certainty that those other buckets had a problem because I didn't know what to look for - I assumed those small objects were valid, because why wouldn't they be? Hence the only ticket I could action with Veeam was the creation of small objects every time Veeam deleted a legitimate object. This turned out to be the dangling DeleteMarkers. Now that I knew what to look for, I was able to loop back and find them on the other Veeam buckets as well.

Now we need to go through and clean up those other buckets.

[jakeellingham]

What is the best way to find if you are affected by this? I also see alot of small object sizes.

[tyler.jurgens]

If you have Veeam buckets, and you see any significant amount of "Less than 1024 B" objects on that bucket, you are likely affected. The next Minio release will have additional metrics in both mc admin and prometheus to display the count of DeletedMarkers.

https://github.com/minio/mc/pull/4642
https://github.com/minio/minio/pull/17689

The Veeam support agent assigned my ticket is being very cagy with telling me which Veeam build the fix he provided me was included in, he's only said it's a known bug in v11. I don't know if there is a v11 build that has this fixed, or if your only recourse is to upgrade to v12. Given it's a v11 issue, it had to be affecting Capacity/Archive tier implementations, so look for any of your customers consuming Minio that way.

I have a blog post waiting to be released outlining this whole scenario and how I'm fixing it. DM me if you want it.