Using object storage as a backup target
Post Reply
YouGotServered
Service Provider
Posts: 104
Liked: 20 times
Joined: Mar 11, 2016 7:41 pm
Full Name: Cory Wallace
Contact:

Feature Request: Remove "s3:ListAllMyBuckets" requirement

Post by YouGotServered »

Hello,
We are a CSP and we are interested in offering S3 storage to our customers to use with the VBR implementations. In order to do this, we would ideally create individual buckets for our clients and delegate permissions with an IAM policy.

Unfortunately, to add an S3 repository to Veeam, you have to be able to list all buckets in the S3 account. We don't want to do this for a number of reasons, but the biggest is that it would essentially expose a list of all of our clients (in the bucket names) to our other clients. Sure, we could obfuscate the client names on the buckets with a numbering system, but that seems like a lame workaround, and it would require us keeping an updated legend on our end that we would have to cross reference constantly.

I know that with generic S3 storage browsers (or at least the two that I have tried), if you don't have permissions to list all buckets, they'll warn you with a message and ask if you would like to manually add one, in which case, manually typing the bucket name will populate it and everything will work as expected.

Is it reasonable to expect that such a feature would be available in Veeam? Can I already do this now and I'm just missing something? Right now, if I try to add an S3 account without that feature, Veeam fails to add it, citing "Access Denied" and tells me to check the logs. I'm aware that the page here cites that this permission is needed, so I'm not surprised that it isn't working, just bummed :)

Source: https://helpcenter.veeam.com/docs/backu ... ml?ver=100

HannesK
Veeam Software
Posts: 6119
Liked: 899 times
Joined: Sep 01, 2014 11:46 am
Location: Austria
Contact:

Re: Feature Request: Remove "s3:ListAllMyBuckets" requirement

Post by HannesK »

Hello,
they'll warn you with a message and ask if you would like to manually add one, in which case, manually typing the bucket name will populate it and everything will work as expected.
if only 1% of our customers would open a case because of that error message, that would be more than 3500 cases for support explaining that they have to type in the bucket name manually :-)

To answer your question: there are currently no plans to change this behavior

I ask myself, why not every of your clients gets it's own account.

Best regards,
Hannes

dalbertson
Veeam Software
Posts: 280
Liked: 70 times
Joined: Jul 21, 2015 12:38 pm
Full Name: Dustin Albertson
Contact:

Re: Feature Request: Remove "s3:ListAllMyBuckets" requirement

Post by dalbertson » 1 person likes this post

You can use an IAM role to not show the buckets. Why is this not acceptable? If you create an IAM role for each customer that has their permissions set to a specific bucket then that’s all they will see.

To add onto what Hannes said, I would recommend setting up accounts for each customer and then you managing the customers data inside their own account. This way you can hand over the account if a customer wants to leave. Also if I’m not mistaken I think it’s required per eula to do it this way. I know in Azure it is.

YouGotServered
Service Provider
Posts: 104
Liked: 20 times
Joined: Mar 11, 2016 7:41 pm
Full Name: Cory Wallace
Contact:

Re: Feature Request: Remove "s3:ListAllMyBuckets" requirement

Post by YouGotServered »

We are a wasabi partner and they have two official methods - use their partner API to provision accounts or use separate buckets with IAM policies under one master account. Unfortunately we don't have the resources in house to work with their API, so we opted for the second method (which is well within the terms of their EULA).

I played around with the IAM policies for a while and couldn't get the desired results, if I could get it to work, that would be a perfectly acceptable result :) would you mind giving me a suggestion on what that policy would look like? For example, if I wanted to give "user1" permission to view and use "user1bucket" only and nothing else, how would you construction that?

Thanks so much!

dalbertson
Veeam Software
Posts: 280
Liked: 70 times
Joined: Jul 21, 2015 12:38 pm
Full Name: Dustin Albertson
Contact:

Re: Feature Request: Remove "s3:ListAllMyBuckets" requirement

Post by dalbertson »

@YouGotServered This is really a limitation of the wasabi IAM policies at this moment. For example this policy below works for AWS and just shows the bucket that is listed in the policy

Code: Select all

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "SecureBucketPolicy0",
            "Effect": "Allow",
            "Action": [
                "s3:PutObject",
                "s3:GetObject",
                "s3:AbortMultipartUpload",
                "s3:ListBucket",
                "s3:DeleteObject",
                "s3:GetBucketLocation",
                "s3:ListMultipartUploadParts"
            ],
            "Resource": [
                "arn:aws:s3:::bucketname/*",
                "arn:aws:s3:::bucketname"
            ]
        },
        {
            "Sid": "SecureBucketPolicy1",
            "Effect": "Allow",
            "Action": [
                "s3:ListAllMyBuckets",
                "s3:HeadBucket"
            ],
            "Resource": "*"
        }
    ]
}
@wasabi_vilas is with wasabi. Maybe he can help push this from their side or knows of another way to accomplish this.

YouGotServered
Service Provider
Posts: 104
Liked: 20 times
Joined: Mar 11, 2016 7:41 pm
Full Name: Cory Wallace
Contact:

Re: Feature Request: Remove "s3:ListAllMyBuckets" requirement

Post by YouGotServered »

@Dalbertson I'll give that a shot and report back. I've been working with Vilas a lot recently on Office 365 backups with Wasabi, I'm sure he's tired of speaking to me :)

Thanks so much for the help.

dalbertson
Veeam Software
Posts: 280
Liked: 70 times
Joined: Jul 21, 2015 12:38 pm
Full Name: Dustin Albertson
Contact:

Re: Feature Request: Remove "s3:ListAllMyBuckets" requirement

Post by dalbertson »

@YouGotServered If you think he is tired of speaking with you...just imagine how he is with me...i have a direct connect to him. I think he deserves hazard pay but i wont tell him that.

YouGotServered
Service Provider
Posts: 104
Liked: 20 times
Joined: Mar 11, 2016 7:41 pm
Full Name: Cory Wallace
Contact:

Re: Feature Request: Remove "s3:ListAllMyBuckets" requirement

Post by YouGotServered »

@Dalbertson,
So I gave that a shot, and while I am able to add the bucket in Veeam, Veeam's bucket browser (along with the other S3 browsers I've tried) still show all available buckets (no access, however). Also worth noting, I couldn't save the policy until I deleted the "s3:HeadBucket" permission, it looks like Wasabi may not support that. Could that be the issue?

Thanks.

dalbertson
Veeam Software
Posts: 280
Liked: 70 times
Joined: Jul 21, 2015 12:38 pm
Full Name: Dustin Albertson
Contact:

Re: Feature Request: Remove "s3:ListAllMyBuckets" requirement

Post by dalbertson »

Correct. I forgot to mention that I did test ways around it but without support for head and list all buckets being separate from the permissions for the individual bucket, it will show all the buckets.

YouGotServered
Service Provider
Posts: 104
Liked: 20 times
Joined: Mar 11, 2016 7:41 pm
Full Name: Cory Wallace
Contact:

Re: Feature Request: Remove "s3:ListAllMyBuckets" requirement

Post by YouGotServered »

Ahh bummer! I know Wasabi is iterating pretty quick on their end. @wasabi_vilas, any chance of this being available in the near future? Seeing as this is one of the supported methods for delegating access from a partner account, I'm surprised this feature isn't available now.

YouGotServered
Service Provider
Posts: 104
Liked: 20 times
Joined: Mar 11, 2016 7:41 pm
Full Name: Cory Wallace
Contact:

Re: Feature Request: Remove "s3:ListAllMyBuckets" requirement

Post by YouGotServered »

HannesK wrote: Mar 11, 2020 6:42 am Hello,

if only 1% of our customers would open a case because of that error message, that would be more than 3500 cases for support explaining that they have to type in the bucket name manually :-)

To answer your question: there are currently no plans to change this behavior

I ask myself, why not every of your clients gets it's own account.

Best regards,
Hannes
Hannes,
I think there's confusion here. I'm not asking to change the behavior completely. I agree with you that there would be a flood of cases if people HAD to type the bucket name manually. That's not what I'm asking for. What I'm asking for is the option to do it manually for a reason like this where regardless of the scenario, we don't want the end user to see all buckets in the account. What if the sysadmin doesn't want the backup guys to see the buckets for other data, just the backup data? I think there are plenty of use cases for this and it seems like it could be a simple text field to enter the name manually if you would like.

YouGotServered
Service Provider
Posts: 104
Liked: 20 times
Joined: Mar 11, 2016 7:41 pm
Full Name: Cory Wallace
Contact:

Re: Feature Request: Remove "s3:ListAllMyBuckets" requirement

Post by YouGotServered »

I spoke with Vilas from Wasabi. This ability to use s3:HeadBucket is on the roadmap for now, but it's not coming anytime soon, Vilas said that 6 months would be a longshot, so 9 months to a year is more realistic. I can appreciate that Wasabi is prioritizing optimizing the stability of their platform over pushing new features right now - I have worked with vendors in the past where they pushed new features while their old ones were broken, and that is significantly worse to deal with than waiting longer for new features.

Despite my understanding, I'm still a little bummed. Due to this, if we want to use Wasabi for our clients, we'll have to create and manage 40 separate accounts for them instead of using our one partner account as intended.

If Veeam could find a way to implement direct bucket mapping, that would be super cool, but I'm sure it would take just as long as Wasabi's implementation of s3:HeadBucket anyways. I still don't think it would be a terrible feature to have.

HannesK
Veeam Software
Posts: 6119
Liked: 899 times
Joined: Sep 01, 2014 11:46 am
Location: Austria
Contact:

Re: Feature Request: Remove "s3:ListAllMyBuckets" requirement

Post by HannesK »

we don't want the end user to see all buckets in the account
how do you really prevent access in that situation? imagine situations where people don't use long UUID bucket names (example: customer0001, customer0002). People might think that security by obscurity really helps them. If I was your customer, I would write a powershell-script that brute-forces for bucket-names.

As far as I know, hiding a bucket is not like "access based enumeration". It's "just" hiding and people still have full access

sure, encryption prevents read access. but I can still delete other customers data.

YouGotServered
Service Provider
Posts: 104
Liked: 20 times
Joined: Mar 11, 2016 7:41 pm
Full Name: Cory Wallace
Contact:

Re: Feature Request: Remove "s3:ListAllMyBuckets" requirement

Post by YouGotServered »

@Hannes,
Again, I think there's confusion. I never said that hiding buckets is a good replacement for access based security. I still want access based security so that brute forcing isn't an option. I addition to access based security, I want to prevent people from listing all the buckets to see bucket names. There's often information in bucket names that we don't want public - we simply don't want to be forced into listing them.

Does that make more sense?

HannesK
Veeam Software
Posts: 6119
Liked: 899 times
Joined: Sep 01, 2014 11:46 am
Location: Austria
Contact:

Re: Feature Request: Remove "s3:ListAllMyBuckets" requirement

Post by HannesK »

yes, if bucket names include confidential information, then it makes sense.

Let's see how many other customers have that situation.

vpock
Influencer
Posts: 12
Liked: 1 time
Joined: Jan 29, 2014 9:42 pm
Full Name: Dominic Paquin
Contact:

Re: Feature Request: Remove "s3:ListAllMyBuckets" requirement

Post by vpock » 1 person likes this post

I'm 6 months late to the party, but I'll still give my grain of salt.

@YouGotServered I feel you. I'm in exactly the spot as you. I just want to be able to "hide" bucket browsing, not have to give s3:ListAllMyBuckets in the ACL. I spoke with Vilas, and the solution is still to use the partner API. But we are not certain the revenue will justify it, as a majority of our customers cannot store data in the US. We know Wasabi has a canadian datacenter on the roadmap, but until then, we have a limited number of customers we could onboard.

In any case, I feel like Veeam's justification of requiring s3:ListAllMyBuckets to limit support case is weak. There has to be a way to work around this?

bytewiseits
Service Provider
Posts: 24
Liked: 11 times
Joined: Nov 23, 2018 12:23 am
Full Name: Dion Norman
Contact:

Re: Feature Request: Remove "s3:ListAllMyBuckets" requirement

Post by bytewiseits » 2 people like this post

We have also hit this issue as well signing up to Wasabi recently. We are only a small MSP so going to their partner API to create truly separate accounts for each customer isn't an option for us either. The other option Wasabi gives with separating out bucket permissions per user does work but is incompatible with Veeam due to the bucket listing requirement.

Regarding the support issues around removing the s3:ListAllMyBuckets requirement, could Veeam maybe just try and list the buckets as it does now, but if it errors, report that bucket listing is not enabled and then give an option to manually specify the bucket name?
This way it does not change the original setup workflow, but should listing buckets be an issue, will still give us a way to manually specify and continue on. I would think that this would be very manageable approach.

veremin
Product Manager
Posts: 17834
Liked: 1662 times
Joined: Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin
Contact:

Re: Feature Request: Remove "s3:ListAllMyBuckets" requirement

Post by veremin »

Thanks, guys, for raising this again and sharing your ideas.

We will try to provide an ability to manually specify bucket name in one of the next product releases.

Thanks!

Post Reply

Who is online

Users browsing this forum: No registered users and 2 guests