Question regarding immutability with MinIO

[tomas881]

Hi everyone,

I'm currently testing MinIO to create an immutable repository for Veeam Backup and Replication backups.
The MinIO service is running as a jail on a TrueNAS Core server. After deploying MinIO, I have created a bucket, enabled object locking and left retention unconfigured as stated in Veeam documentation.
Later I created a new backup job, selected the S3 repository as the first stage for the backup files and enabled immutability on Veeam and run the job.
Backup was done correctly and when I tried to delete the backup files from within Veeam the process failed as expected. But when I browse the MinIO bucket using the web GUI I could delete the files and folders within the bucket.
Is this behaviour expected? Is there anyway I could prevent this from happening? My guess would be that this can be done only if you have the Access Key and Secret Key but it would be nice if I could secure it a little bit more.
Thanks in advance!

[sfirmes]

If you are using Compliance mode, this isn't the expected behavior. VBR should reject your deletion attempt as you experienced, but you shouldn't be able to delete the objects regardless of how you try. This includes Minio's browser, s3 browser, AWS cli, etc..... I haven't tested MinIO using Governance mode yet (VBR supports Governance mode in version 12.1), but will do so soon and will update this thread.

[sfirmes]

Tomas,

I just set up a backup to go directly an immutable bucket on MinIO. I then tried to delete the objects from their object browser and I was unable to due to the objects being locked:



Within their browser you can see the lock information for the objects. Could you please verify that your objects are using COMPLIANCE mode and that the deletion date hasn't been reached yet:


If the objects are locked via Compliance mode and the lock retention date hasn't been met yet, you shouldn't be able to delete the objects.

Hope this helps.

[tomas881]

Hi sfirmes! Thanks a lot for your response. Sorry I didn't replied sooner, I didn't receive a notification

I posted the same question on the MinIO Slack channel and the "issue" was that I wasn't toggling "Delete All Versions" when doing a deletion, so a "Deleted" tag (or version) of the object was created but the object wasn't completely deleted. When enabling the "Delete All Versions" options I got the same message as you did, so the retention is working properly.

If I may, I would like to ask you two more questions.
In your previous post you say that Governance mode is available on version 12.1. Would your recommend to use Governance mode over Compliance mode with Veeam?

On the other hand. I was reading this page of the documentation: https://helpcenter.veeam.com/docs/backu ... ml?ver=120, because I saw that regardless that I set the immutability for 1 day the objects could not be deleted for 11 days. This +10 days of immutability are only for the first runs of the backup job, correct? If I want to keep the last 7 days as immutable, only the first week will stay for 17 days. After the 17th day all following backup files will be immutable for 7 days?

Thanks in advance.

[sfirmes]

Tomas,

Glad you resolved your deletion issue.

With regards to:

Would your recommend to use Governance mode over Compliance mode with Veeam?

Compliance mode is the most secure mode for locking the objects and making them immutable. I would only recommend governance mode for service providers like yourself if you are having issues with customers terminating their contracts early and you're forced to manage their locked date until the object lock expires.

Block Generation question:

If I want to keep the last 7 days as immutable, only the first week will stay for 17 days. After the 17th day all following backup files will be immutable for 7 days?

Block Generation is a mechanism we use to reduce I/O operations and associated costs. It is applied to all backup chains and not only the initial backup. If you want to keep 7 days of immutable backups, set the immutability on the repository to 7 days.

Hope this helps.

Steve

[dozo]

Hi Steve,
Apologies for bringing this backup.

Testing with the latest minio (minio-20250218162555.0.0-1.x86_64.rpm) and can verify in the browser: x-amz-object-lock-mode: compliance. Deleting all versions will not let me delete, but I can delete with the toggle to no. I randomly deleted stuff and tried a test restore to an incremental point, which failed with could not find block error.
It was successfully restored with the first point in the full backup chain. However, when trying another restore, there are no longer any restore points.

[jjonesasl]

Hi,

I have only ever used the mc client to delete objects with governance mode

mc rm --force --recursive --bypass --versions bucket/folder

--bypass
Optional
Allows removing an object held under GOVERNANCE object locking.
https://min.io/docs/minio/linux/referen ... mc-rm.html

[sfirmes]

Testing with the latest minio (minio-20250218162555.0.0-1.x86_64.rpm) and can verify in the browser: x-amz-object-lock-mode: compliance. Deleting all versions will not let me delete, but I can delete with the toggle to no. I randomly deleted stuff and tried a test restore to an incremental point, which failed with could not find block error.
It was successfully restored with the first point in the full backup chain. However, when trying another restore, there are no longer any restore points.

@dozo This is the expected behavior. Object(s) that VBR used where deleted and the restore failed cause it couldn't find the required objects. One thing to note here is when you delete an object without specifying the version id, the object will be marked as deletion via a new version of the object called a "deletion marker". The original object still exists and by deleting the deletion marker you can "resurrect" it.

Sort of like restoring a file from a recycle bin.

That is why when you de-selected the "Delete All Versions" option, you were able to delete the objects cause the version id wasn't used. When you select the "Delete All Versions" option, the version id is used and the object lock prevents the deletion.

So no need to apologize. This was a great question and one that I supect many other forum participants will learn from.

[sfirmes]

I have only ever used the mc client to delete objects with governance mode

mc rm --force --recursive --bypass --versions bucket/folder

Jacob,

Thanks for the mc example. I use the aws cli's delete-object since I use so many object storage platforms and mc is specific to MinIO. Similar to mc the delete-object has a "bypass" option for governance mode locks. Here is an example: aws s3api delete-object --bucket bucket_name --key object_key --version-id object_version_id --bypass-governance-retention

Thanks again.

Steve