Remove existing S3 immutable backups before compliance period?

[CatMDV]

Our current backup system has S3 immutable buckets in capacity tier to offload data as soon as they appear on the performance tier.

We are currently undergoing a major overhaul of the backup system including completely wiping out the existing performance storage and starting off new (the way I set up the repository size and other settings are not scaling well, and the S3 bill is blowing up as a result).

I understand the concept of compliance mode S3 immutable storages and the inability to conventionally remove immutable buckets even using AWS console. But the migration period means I have to run parallel backups with the existing buckets and the new buckets for at least a month and a half. This would not be feasible since our office payment card has a monthly dollar limit (3rd world problems). Hence, I will need to remove the existing buckets somehow, and then create new buckets.

So my question is, how do I completely remove the immutable buckets before hitting the object lock date? Can i remove the associated user, thereby removing the S3 buckets associated with it? (It is under an IAM user created by a root user in our AWS organization, and there is another unrelated root user in the same organization as well). Or would I actually need to remove the whole organization from AWS to make this possible? Or is either of these options even possible?

Thanks,
Sam

[Gostev]

Hi, Sam. If this was possible from your end, then backups would not be considered immutable... however, AWS Support might be able to do this so I suggest you check with them. I assume the technical capability is there because at the very least they do need a process to get rid of immutable data belonging to non-paying customers. Thanks

[Royadiel]

Hi Sam.
I understand your concerns regarding the S3 immutable buckets and the need to remove them for your backup system overhaul. Here's some important information to consider:

First and foremost, you should check whether your S3 buckets are in compliance mode or governance mode. This distinction is crucial as it affects your ability to delete data:

1. Compliance Mode: If your buckets are in compliance mode, deletion is not possible until the retention period expires. There are no exceptions to this rule.

2. Governance Mode: If your buckets are in governance mode, you may be able to delete data with the right permissions. Users with specific IAM permissions can override or remove the retention settings.

If your buckets are in governance mode, you might be able to delete the data with the appropriate permissions. To do this:

1. Ensure you have the `s3:BypassGovernanceRetention` permission in your IAM policy.
2. Use the `--bypass-governance-retention` flag when using AWS CLI commands to delete objects.

It's important to note that simply removing the associated user or the entire organization will not automatically delete the S3 buckets or their contents. S3 buckets and their data persist independently of the IAM users or the organization structure.

If you find that your buckets are in compliance mode or you don't have the necessary permissions in governance mode, you may need to wait until the retention period expires before you can delete the data. In this case, you might want to consider:

1. Temporarily increasing your payment card limit for the migration period.
2. Exploring alternative funding options for the parallel backup period.
3. Discussing with AWS support about potential solutions for your specific situation.

As Anton said, object lock settings are designed to protect data from accidental or malicious deletion, so there are intentionally limited options for overriding these settings.

Roy

[CatMDV]

thanks for the help. The backups are indeed in compliance mode. Hence it appears that my choices are limited here. But this is still useful since I can take this information to the management to make future decisions on the situation.