Discussions related to using object storage as a backup target.
Post Reply
tomas881
Service Provider
Posts: 14
Liked: never
Joined: Dec 21, 2018 4:50 pm
Full Name: Tomas La Grotteria
Contact:

Question regarding immutability with MinIO

Post by tomas881 »

Hi everyone,

I'm currently testing MinIO to create an immutable repository for Veeam Backup and Replication backups.
The MinIO service is running as a jail on a TrueNAS Core server. After deploying MinIO, I have created a bucket, enabled object locking and left retention unconfigured as stated in Veeam documentation.
Later I created a new backup job, selected the S3 repository as the first stage for the backup files and enabled immutability on Veeam and run the job.
Backup was done correctly and when I tried to delete the backup files from within Veeam the process failed as expected. But when I browse the MinIO bucket using the web GUI I could delete the files and folders within the bucket.
Is this behaviour expected? Is there anyway I could prevent this from happening? My guess would be that this can be done only if you have the Access Key and Secret Key but it would be nice if I could secure it a little bit more.
Thanks in advance!
sfirmes
Veeam Software
Posts: 320
Liked: 150 times
Joined: Jul 24, 2018 8:38 pm
Full Name: Stephen Firmes
Contact:

Re: Question regarding immutability with MinIO

Post by sfirmes »

If you are using Compliance mode, this isn't the expected behavior. VBR should reject your deletion attempt as you experienced, but you shouldn't be able to delete the objects regardless of how you try. This includes Minio's browser, s3 browser, AWS cli, etc..... I haven't tested MinIO using Governance mode yet (VBR supports Governance mode in version 12.1), but will do so soon and will update this thread.
Steve Firmes | Senior Solutions Architect, Product Management - Alliances @ Veeam Software
sfirmes
Veeam Software
Posts: 320
Liked: 150 times
Joined: Jul 24, 2018 8:38 pm
Full Name: Stephen Firmes
Contact:

Re: Question regarding immutability with MinIO

Post by sfirmes »

Tomas,

I just set up a backup to go directly an immutable bucket on MinIO. I then tried to delete the objects from their object browser and I was unable to due to the objects being locked:

Image

Within their browser you can see the lock information for the objects. Could you please verify that your objects are using COMPLIANCE mode and that the deletion date hasn't been reached yet:
Image

If the objects are locked via Compliance mode and the lock retention date hasn't been met yet, you shouldn't be able to delete the objects.

Hope this helps.
Steve Firmes | Senior Solutions Architect, Product Management - Alliances @ Veeam Software
tomas881
Service Provider
Posts: 14
Liked: never
Joined: Dec 21, 2018 4:50 pm
Full Name: Tomas La Grotteria
Contact:

Re: Question regarding immutability with MinIO

Post by tomas881 »

Hi sfirmes! Thanks a lot for your response. Sorry I didn't replied sooner, I didn't receive a notification

I posted the same question on the MinIO Slack channel and the "issue" was that I wasn't toggling "Delete All Versions" when doing a deletion, so a "Deleted" tag (or version) of the object was created but the object wasn't completely deleted. When enabling the "Delete All Versions" options I got the same message as you did, so the retention is working properly.

If I may, I would like to ask you two more questions.
In your previous post you say that Governance mode is available on version 12.1. Would your recommend to use Governance mode over Compliance mode with Veeam?

On the other hand. I was reading this page of the documentation: https://helpcenter.veeam.com/docs/backu ... ml?ver=120, because I saw that regardless that I set the immutability for 1 day the objects could not be deleted for 11 days. This +10 days of immutability are only for the first runs of the backup job, correct? If I want to keep the last 7 days as immutable, only the first week will stay for 17 days. After the 17th day all following backup files will be immutable for 7 days?

Thanks in advance.
sfirmes
Veeam Software
Posts: 320
Liked: 150 times
Joined: Jul 24, 2018 8:38 pm
Full Name: Stephen Firmes
Contact:

Re: Question regarding immutability with MinIO

Post by sfirmes »

Tomas,

Glad you resolved your deletion issue.

With regards to:
Would your recommend to use Governance mode over Compliance mode with Veeam?
Compliance mode is the most secure mode for locking the objects and making them immutable. I would only recommend governance mode for service providers like yourself if you are having issues with customers terminating their contracts early and you're forced to manage their locked date until the object lock expires.

Block Generation question:
If I want to keep the last 7 days as immutable, only the first week will stay for 17 days. After the 17th day all following backup files will be immutable for 7 days?
Block Generation is a mechanism we use to reduce I/O operations and associated costs. It is applied to all backup chains and not only the initial backup. If you want to keep 7 days of immutable backups, set the immutability on the repository to 7 days.

Hope this helps.

Steve
Steve Firmes | Senior Solutions Architect, Product Management - Alliances @ Veeam Software
dozo
Lurker
Posts: 1
Liked: never
Joined: Feb 27, 2025 5:56 pm
Contact:

Re: Question regarding immutability with MinIO

Post by dozo »

Hi Steve,
Apologies for bringing this backup.

Testing with the latest minio (minio-20250218162555.0.0-1.x86_64.rpm) and can verify in the browser: x-amz-object-lock-mode: compliance. Deleting all versions will not let me delete, but I can delete with the toggle to no. I randomly deleted stuff and tried a test restore to an incremental point, which failed with could not find block error.
It was successfully restored with the first point in the full backup chain. However, when trying another restore, there are no longer any restore points.
jjonesasl
Service Provider
Posts: 4
Liked: never
Joined: Mar 02, 2025 6:49 pm
Full Name: Jacob Jones
Contact:

Re: Question regarding immutability with MinIO

Post by jjonesasl »

Hi,

I have only ever used the mc client to delete objects with governance mode

mc rm --force --recursive --bypass --versions bucket/folder

--bypass
Optional
Allows removing an object held under GOVERNANCE object locking.
https://min.io/docs/minio/linux/referen ... mc-rm.html
sfirmes
Veeam Software
Posts: 320
Liked: 150 times
Joined: Jul 24, 2018 8:38 pm
Full Name: Stephen Firmes
Contact:

Re: Question regarding immutability with MinIO

Post by sfirmes »

dozo wrote: Feb 27, 2025 8:09 pm Testing with the latest minio (minio-20250218162555.0.0-1.x86_64.rpm) and can verify in the browser: x-amz-object-lock-mode: compliance. Deleting all versions will not let me delete, but I can delete with the toggle to no. I randomly deleted stuff and tried a test restore to an incremental point, which failed with could not find block error.
It was successfully restored with the first point in the full backup chain. However, when trying another restore, there are no longer any restore points.
@dozo This is the expected behavior. Object(s) that VBR used where deleted and the restore failed cause it couldn't find the required objects. One thing to note here is when you delete an object without specifying the version id, the object will be marked as deletion via a new version of the object called a "deletion marker". The original object still exists and by deleting the deletion marker you can "resurrect" it.

Sort of like restoring a file from a recycle bin.

That is why when you de-selected the "Delete All Versions" option, you were able to delete the objects cause the version id wasn't used. When you select the "Delete All Versions" option, the version id is used and the object lock prevents the deletion.

So no need to apologize. This was a great question and one that I supect many other forum participants will learn from.
Steve Firmes | Senior Solutions Architect, Product Management - Alliances @ Veeam Software
sfirmes
Veeam Software
Posts: 320
Liked: 150 times
Joined: Jul 24, 2018 8:38 pm
Full Name: Stephen Firmes
Contact:

Re: Question regarding immutability with MinIO

Post by sfirmes »

jjonesasl wrote: Mar 03, 2025 2:03 pm I have only ever used the mc client to delete objects with governance mode

mc rm --force --recursive --bypass --versions bucket/folder
Jacob,

Thanks for the mc example. I use the aws cli's delete-object since I use so many object storage platforms and mc is specific to MinIO. Similar to mc the delete-object has a "bypass" option for governance mode locks. Here is an example: aws s3api delete-object --bucket bucket_name --key object_key --version-id object_version_id --bypass-governance-retention

Thanks again.

Steve
Steve Firmes | Senior Solutions Architect, Product Management - Alliances @ Veeam Software
Post Reply

Who is online

Users browsing this forum: No registered users and 10 guests