How to configure object storage so that on-prem uses direct connect, but cloud use public endpoints

[appleoddity]

We have on-prem infrastructure, backing up to AWS S3 buckets, and a VBR server in AWS. We have a direct connect between one site and AWS and an S3 interface endpoint reachable over direct connect.

I want to force some on-prem sites to utilize the direct connect to transfer data to S3, while health checks / archive tiering and other sites use the internet or a S3 gateway endpoint in AWS.

To accomplish this:
- I will have different S3 bucket configurations for each site (it's a folder in the same bucket)
- I will have two different entries in AmazonS3Regions.xml (one for direct connect, and one for non-direct connect)

I have been able to make this work. But, my concern is that I do NOT want the helper appliances in AWS to use the S3 interface endpoint, and instead I want them to utilize public S3 IPs that will go over an S3 gateway endpoint.

To accomplish this:
- I added the following in AmazonS3Regions.xml for the S3 endpoint: veeam.s3.us-east-1.amazonaws.com
- Created an on-prem DNS zone for veeam.s3.us-east-1.amazonaws.com
- Added the IP addresses of the S3 interface endpoint as A records in this new zone, as well as a wildcard CNAME pointing to the S3 interface endpoint vpce custom DNS (i.e. *.vpce-xxxxxxxxxxxxxxxxx-yyyyyyyy.s3.us-east-1.vpce.amazonaws.com

With this setup, veeam.s3.us-east-1.amazonaws.com resolves to private IPs on-prem, but to public IPs in AWS. In addition, because the certificate returned from S3 has SANs that cover *.s3.us-east-1.amazonaws.com there are no certificate issues.

The problem is, when I try to then check the bucket configuration in VBR console, I get an error saying the bucket cannot be found. In the log, I can see it validating the certificate and connecting successfully to veeam.s3.us-east-1.amazonaws.com, but I cannot find any further logs that indicate why it can't find the bucket.

If I simply add bucket.vpce-xxxxxxxxxxxxxxxxx-yyyyyyyy.s3.us-east-1.vpce.amazonaws.com as the S3 endpoint in AmazonS3Regions.xml, everything works fine. But, doesn't this make the helper appliance also use the private IPs?

My questions are this:
- If I specify a custom endpoint in AmazonS3Regions.xml that resolves to private S3 interface endpoints - will the helper appliance in AWS also use this private endpoint?
- If the above is true, how do I get my on-prem proxies to use the private endpoints, while the helper appliances use the public endpoints?

[david.domask]

Hi appleoddity,

- I will have different S3 bucket configurations for each site (it's a folder in the same bucket)

While this is possible, please note it is not recommended. The linked page refers to Scale-out Backup Repositories (SOBR), but in effect you're doing the same thing hosting multiple Object Storage repositories in the same bucket.

I have been able to make this work. But, my concern is that I do NOT want the helper appliances in AWS to use the S3 interface endpoint, and instead I want them to utilize public S3 IPs that will go over an S3 gateway endpoint.

Helper Appliances have a separate control to enable use of private endpoints, so if this is not enabled, helper appliances will try to connect over normal AWS endpoints.