Discussions related to using object storage as a backup target.
Post Reply
jazzoberoi
Enthusiast
Posts: 96
Liked: 24 times
Joined: Oct 08, 2014 9:07 am
Full Name: Jazz Oberoi
Contact:

How does deletion with governance work?

Post by jazzoberoi »

Hi Steve.
Appreciate the reply and great article :)

I’m starting from scratch with a new VBR server.
Could you please also explain how the delete process works in governance mode? Is this done from the S3 bucket end or from VBR server end?

Wondering if there is a happy compromise that can be made so we can set long GFS retention without fearing lock-in if things start getting out of hand with costs.

Unless, veeam introduce a more streamlined and granular immutability where we can say select daily/weekly/monthly as immutable but not yearly etc..

#MOD: split from Backup Copy to Object Storage vs. Scale-Out Backup Repository Copy Mode
jazzoberoi
Enthusiast
Posts: 96
Liked: 24 times
Joined: Oct 08, 2014 9:07 am
Full Name: Jazz Oberoi
Contact:

Re: Backup Copy to Object Storage vs. Scale-Out Backup Repository Copy Mode

Post by jazzoberoi »

Hi Steve,
Just following up to see if you can please explain how to delete a immutable backup in governance mode please..
Mildur
Product Manager
Posts: 9926
Liked: 2632 times
Joined: May 13, 2017 4:51 pm
Full Name: Fabian K.
Location: Switzerland
Contact:

Re: Backup Copy to Object Storage vs. Scale-Out Backup Repository Copy Mode

Post by Mildur »

Hi Jazz

Any user with the s3:BypassGovernanceRetention permission can delete such objects.
https://docs.aws.amazon.com/AmazonS3/la ... ing-bypass

It's not possible from within the Veeam console. You need to logon directly to the storage (object storage UI, object storage browsing tools, ...) and delete such objects from there.

Best,
Fabian
Product Management Analyst @ Veeam Software
jazzoberoi
Enthusiast
Posts: 96
Liked: 24 times
Joined: Oct 08, 2014 9:07 am
Full Name: Jazz Oberoi
Contact:

Re: Backup Copy to Object Storage vs. Scale-Out Backup Repository Copy Mode

Post by jazzoberoi »

Hi Mildur,
Thanks for your reply.

Wouldn’t that mess up the retention points in Veeam though? As these is no concept of VBK / VIB files in S3, how will we know which objects belong to which GFS point, and whether or not deleting them will cause issue with other restore points?

It would be great if we could get some Governance bypass within Veeam to manage this?
Mildur
Product Manager
Posts: 9926
Liked: 2632 times
Joined: May 13, 2017 4:51 pm
Full Name: Fabian K.
Location: Switzerland
Contact:

Re: Backup Copy to Object Storage vs. Scale-Out Backup Repository Copy Mode

Post by Mildur »

Hi Jazz
Wouldn’t that mess up the retention points in Veeam though? As these is no concept of VBK / VIB files in S3, how will we know which objects belong to which GFS point, and whether or not deleting them will cause issue with other restore points?
Of course. Deleting objects used by Veeam directly in object storage is not supported.
Governance mode is meant for service providers to delete "EVERY OBJECT" when a customer has cancelled his contract with the service provider.
It gives the service provider an option to completely clean out the bucket and then delete it. A service provider needs that functionality or he may need to keep ex-customers backups for years.
It would be great if we could get some Governance bypass within Veeam to manage this?
Every attacker or malicious insider would love that feature as well :)
The primary reason for Immutability is to protect your backups against deletion. Having a bypass option directly in the Veeam console is not a good idea. Any attacker will be able to make use of the bypass as well.
Also make sure that the S3 credentials you use to connect from the backup server to the bucket doesn't have this bypass permission. An attacker can export those credentials from the Veeam server and start deleting your backups.

s3:BypassGovernanceRetention must only be given to dedicated users which are only used for emergency access.

Best,
Fabian
Product Management Analyst @ Veeam Software
jazzoberoi
Enthusiast
Posts: 96
Liked: 24 times
Joined: Oct 08, 2014 9:07 am
Full Name: Jazz Oberoi
Contact:

Re: Backup Copy to Object Storage vs. Scale-Out Backup Repository Copy Mode

Post by jazzoberoi »

Mildur wrote: Mar 15, 2024 11:33 am Hi Jazz

Any user with the s3:BypassGovernanceRetention permission can delete such objects.
https://docs.aws.amazon.com/AmazonS3/la ... ing-bypass

It's not possible from within the Veeam console. You need to logon directly to the storage (object storage UI, object storage browsing tools, ...) and delete such objects from there.

Best,
Fabian
Hmm.. looks like Steve, the author of this article has a different view on where it should be deleted from! The only question is how is it done from Veeam?
https://community.veeam.com/blogs-and-p ... art-7-6757

Image
Mildur
Product Manager
Posts: 9926
Liked: 2632 times
Joined: May 13, 2017 4:51 pm
Full Name: Fabian K.
Location: Switzerland
Contact:

Re: How does deletion with governance work?

Post by Mildur »

Steves comment is correct. Deleting restore points of selected machines is only supported from the backup console. As you mentioned earlier, there are no backup files. A backup administrator will not be able to know which objects has to be deleted. Therefore the backup console has to be used.
Which I confirmed already in my last comment:
Of course. Deleting objects used by Veeam directly in object storage is not supported.
The backup console wouldn‘t work for backups which are still immutable. The console would throw an error that those restore points are still immutable. Doesn‘t matter if governance or compliance mode was used.

I also explained why we have the governance mode. Service Provider who need to delete all backups, because a customer has left. The entire repository. Deleting all objects directly on the object storage is ok and totally supported, because the backup server won‘t use this repository anymore. Just remove the repository from the backup server and delete everything on it.
Governance mode is meant for service providers to delete "EVERY OBJECT" when a customer has cancelled his contract with the service provider.
It gives the service provider an option to completely clean out the bucket and then delete it. A service provider needs that functionality or he may need to keep ex-customers backups for years.

To summarize this topic:
- you can use governance mode as an exit strategy to delete an entire bucket with immutable backups
- you cannot delete backups from single machines directly on object storage (outside of Veeam Backup & Replication)
- we most likely will never introduce an option to delete immutable backups within the backup console when governance mode was used. This opens a huge security hole

Best,
Fabian
Product Management Analyst @ Veeam Software
tomers@y-tech.net
Service Provider
Posts: 101
Liked: 10 times
Joined: Aug 09, 2017 7:31 am
Full Name: Tomer Schwaitzer
Contact:

Re: How does deletion with governance work?

Post by tomers@y-tech.net »

This has no logic.
Service providers do not use repository bucket per customer. We have lots of customers, all use the same repository and bucket.
We need a way to delete specific customer backups, not the entire bucket.
Mildur
Product Manager
Posts: 9926
Liked: 2632 times
Joined: May 13, 2017 4:51 pm
Full Name: Fabian K.
Location: Switzerland
Contact:

Re: How does deletion with governance work?

Post by Mildur »

Hi Tomer

Do you have a question about "governance mode" or is it a new request unrelated to governance mode?
If it's the latter, I will move your comment to a new topic.

Thank you,
Fabian
Product Management Analyst @ Veeam Software
tomers@y-tech.net
Service Provider
Posts: 101
Liked: 10 times
Joined: Aug 09, 2017 7:31 am
Full Name: Tomer Schwaitzer
Contact:

Re: How does deletion with governance work?

Post by tomers@y-tech.net »

Hi Fabian,
This is regarding governance mode of course.
tyler.jurgens
Veeam Legend
Posts: 412
Liked: 234 times
Joined: Apr 11, 2023 1:18 pm
Full Name: Tyler Jurgens
Contact:

Re: How does deletion with governance work?

Post by tyler.jurgens »

It is generally recommended to use one bucket per tenant when you are using S3 through Cloud Connect. I know you can get away without doing that, but it has challenges - like the one you described.
Tyler Jurgens
Veeam Legend x3 | vExpert ** | VMCE | VCP 2020 | Tanzu Vanguard | VUG Canada Leader | VMUG Calgary Leader
Blog: https://explosive.cloud
Twitter: @Tyler_Jurgens BlueSky: @explosive.cloud
tomers@y-tech.net
Service Provider
Posts: 101
Liked: 10 times
Joined: Aug 09, 2017 7:31 am
Full Name: Tomer Schwaitzer
Contact:

Re: How does deletion with governance work?

Post by tomers@y-tech.net »

Well.. Configuring new bucket for each tenant is much more challenging.. we have automation and self boarding, at the moment it does not include new bucket for each tenant.
If this is a working scenario with other service providers I will be glad to learn.
I assume most of the SP does not do that.
Post Reply

Who is online

Users browsing this forum: Google [Bot] and 8 guests