Using object storage as a backup target
vIdaho1
Influencer
Posts: 11
Liked: 2 times
Joined: May 10, 2021 5:17 pm
Full Name: Andrew Foster
Contact:

Utilize AWS Direct Connect/Private Link for Capacity Tier

Post by vIdaho1 »

Hello,

We have an on prem VBR implementation and want to utilize AWS S3 for capacity tier with our SOBR. We have a private 10Gbp link from our DC to Equinix DC and then 10Gbp direct connect into AWS. We have created an AWS private link and VPC endpoint to our S3 bucket. The problem is the capacity tier traffic still uses our internet connection and within Veeam B&R you cannot specify a service endpoint for AWS S3, only S3 compatible.

I opened a support case with Veeam but there response wasn't any help. It sounds like using private link with capacity tier can be done but I have not been able to find any specific instructions on what all we have to configure to get this to work. Does anyone know what specifically has to be configured to be able to get private links with AWS working with capacity tier so we can ensure our traffic for capacity tier is not routed over the internet?

Response from support on my case #04785260
"In order to use a private link you'll need to configure DNS to facilitate it. Unfortunately there is no way from within Veeam to accomplish this.

Unfortunately this process isn't really supported by Veeam, so there is no official documentation on this that I can share with you.

This thread on our forums may be of some use to you, however:
object-storage-f52/s3-offload-copy-and- ... 65300.html"

HannesK
Veeam Software
Posts: 9104
Liked: 1663 times
Joined: Sep 01, 2014 11:46 am
Full Name: Hannes Kasparick
Location: Austria
Contact:

Re: Utilize AWS Direct Connect/Private Link for Capacity Tier

Post by HannesK »

Hello,
and welcome to the forums.

Just to be sure: you are asking for this scenario, right? https://aws.amazon.com/blogs/aws/aws-pr ... available/

Best regards,
Hannes

vIdaho1
Influencer
Posts: 11
Liked: 2 times
Joined: May 10, 2021 5:17 pm
Full Name: Andrew Foster
Contact:

Re: Utilize AWS Direct Connect/Private Link for Capacity Tier

Post by vIdaho1 »

Hi HannesK,
Yes, that is the scenario I am asking about. Really hoping we can utilize that and if not that what is another way we can keep the traffic on our private network all the way to the S3 bucket?

HannesK
Veeam Software
Posts: 9104
Liked: 1663 times
Joined: Sep 01, 2014 11:46 am
Full Name: Hannes Kasparick
Location: Austria
Contact:

Re: Utilize AWS Direct Connect/Private Link for Capacity Tier

Post by HannesK » 1 person likes this post

ok - I just heard that you have a meeting with one of our architects who is an expert on that topic. Maybe you an him together could create a guide how to configure it :-)

Tos
Enthusiast
Posts: 88
Liked: never
Joined: Aug 21, 2014 7:26 am
Full Name: Toshihiro Kobayashi
Contact:

[MERGED] Offload to Amazon S3 using AWS Private Link

Post by Tos »

Hi
A new AWS service called "AWS PrivateLink for Amazon S3" has been released.
If we use this, it will be possible to transfer to S3 using a private IP.
https://docs.aws.amazon.com/AmazonS3/la ... oints.html

In this case, we have questions.

Q1
Is it possible to use this AWS PrivateLink function to archive backup data to S3 using a private IP even when using Veeam's archive to Amazon S3 ?
https://helpcenter.veeam.com/docs/backu ... ml?ver=110

Q2
If Q1 yes, is it possible to limit the bandwidth using Veeam's network traffic rule function ?
(Will Internet traffic rules work ?)
https://helpcenter.veeam.com/docs/backu ... ml?ver=110

Let me know.

HannesK
Veeam Software
Posts: 9104
Liked: 1663 times
Joined: Sep 01, 2014 11:46 am
Full Name: Hannes Kasparick
Location: Austria
Contact:

Re: Utilize AWS Direct Connect/Private Link for Capacity Tier

Post by HannesK »

Hello,
is it the same question like above? If yes, please stay tuned.

Q2: probably not, because it's not "internet"

Best regards,
Hannes

HannesK
Veeam Software
Posts: 9104
Liked: 1663 times
Joined: Sep 01, 2014 11:46 am
Full Name: Hannes Kasparick
Location: Austria
Contact:

Re: Utilize AWS Direct Connect/Private Link for Capacity Tier

Post by HannesK »

Hello,
I just heard that it works if you add the private IP address as S3 compatible storage.

Best regards,
Hannes

vIdaho1
Influencer
Posts: 11
Liked: 2 times
Joined: May 10, 2021 5:17 pm
Full Name: Andrew Foster
Contact:

Re: Utilize AWS Direct Connect/Private Link for Capacity Tier

Post by vIdaho1 »

It does work if you add your AWS storage as S3 compatible but you cannot utilize One Zone-Infrequent Access, only S3 Standard. At least at this point anyway.

NiX
Veeam Software
Posts: 46
Liked: 37 times
Joined: Oct 01, 2015 4:53 pm
Full Name: Nicholas Serrecchia
Location: Long Beach, Ca.
Contact:

Re: Utilize AWS Direct Connect/Private Link for Capacity Tier

Post by NiX »

Correct @vIdaho1 . Utilizing S3 compatible storage, you can use S3, even Object Lock, but Veeam will not flag the appropriate objects for IA or IA one zone.
For S3 compatible storage, all you need to do is input the VPC endpoint ID FQDN into the Service point box when setting up your S3 compatible storage. The VPC endpoint should look something like this vpce-1a2b3c4d-5e6f.s3.us-east-1.vpce.amazonaws.com.

@vIdaho1Do you have a need for IA with Private link?
Nicholas Serrecchia - Veeam
Senior Solutions Architect - Public Cloud - NA

vIdaho1
Influencer
Posts: 11
Liked: 2 times
Joined: May 10, 2021 5:17 pm
Full Name: Andrew Foster
Contact:

Re: Utilize AWS Direct Connect/Private Link for Capacity Tier

Post by vIdaho1 »

@Nix yes, we would still like to have it as an option. We're working with AWS to compare costs between the two tiers now before we renegotiate our discount, but just from using their online calculator standard still looks to be more expensive then IA even with the difference in API costs.

HannesK
Veeam Software
Posts: 9104
Liked: 1663 times
Joined: Sep 01, 2014 11:46 am
Full Name: Hannes Kasparick
Location: Austria
Contact:

Re: Utilize AWS Direct Connect/Private Link for Capacity Tier

Post by HannesK » 1 person likes this post

Good news: we plan to publish a KB article how to configure it by adding the connections directly to the XML file and disable automatic updates of the AWS regions.

I will post it once the KB article is done.

veremin
Product Manager
Posts: 18857
Liked: 1905 times
Joined: Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin
Contact:

Re: Utilize AWS Direct Connect/Private Link for Capacity Tier

Post by veremin »

Small clarification - the KB will be published at the time of 11a release which brings additional registry key enabling communication with AWS infrastructure via private IP addresses. Thanks!

vIdaho1
Influencer
Posts: 11
Liked: 2 times
Joined: May 10, 2021 5:17 pm
Full Name: Andrew Foster
Contact:

Re: Utilize AWS Direct Connect/Private Link for Capacity Tier

Post by vIdaho1 »

Any ETA on when 11a will be released?

HannesK
Veeam Software
Posts: 9104
Liked: 1663 times
Joined: Sep 01, 2014 11:46 am
Full Name: Hannes Kasparick
Location: Austria
Contact:

Re: Utilize AWS Direct Connect/Private Link for Capacity Tier

Post by HannesK »

We are targeting August. But as always, it will only be released "when it's ready" :-)

theviking84
Veeam Legend
Posts: 102
Liked: 10 times
Joined: Nov 16, 2020 2:58 pm
Full Name: David Dunworthy
Contact:

[MERGED] Archive tier aws

Post by theviking84 »

I believe I saw a post on here that dealt with veeam server trying to use only public ip or dns name of the appliance vms during archive tier processes.

I have same scenario where my ec2 repo does not have internet access and veeam talks to it by private ip. This works fine.

Offloads to s3 are fine also for sobr operations.

But when I want to set up archive tier, will veeam server be able to use internal ip address of the appliance vms that are stood up during archive tier moves?

If this isn't clear I will try to find the related post or explain better. But what i recall is you guys said it would be 11a patch before this would work. I'm hoping that is still true and wondering when 11a will be out if so. Otherwise there is no way for archive tier to work for me since the ec2 repo does not have internet, at least that is what I suspect is the case.

HannesK
Veeam Software
Posts: 9104
Liked: 1663 times
Joined: Sep 01, 2014 11:46 am
Full Name: Hannes Kasparick
Location: Austria
Contact:

Re: Archive tier aws

Post by HannesK »

Hello,
I assume you mean this topic post415704.html ?

August is the current goal for 11a

Best regards,
Hannes

theviking84
Veeam Legend
Posts: 102
Liked: 10 times
Joined: Nov 16, 2020 2:58 pm
Full Name: David Dunworthy
Contact:

Re: Utilize AWS Direct Connect/Private Link for Capacity Tier

Post by theviking84 »

That's it, thank you Hannes. Awaiting 11a.

inferno66
Influencer
Posts: 13
Liked: 1 time
Joined: Mar 17, 2021 8:54 am
Full Name: Julien
Contact:

Re: Utilize AWS Direct Connect/Private Link for Capacity Tier

Post by inferno66 »

Hello,

Also awaiting for this update (hopefully this month), as my proxy appliance will be on the same VPC / Same Subnet than my Veeam server (and with direct S3 access), so I want them to communicate on private IP.

Also another from my understanding only gfs backup can be offloaded to Archive Tier (which make sense), so only Weekly / Montlhy / Yearly.
But is it possible to tag a manually started full backup (test job) on GFS in order to be able to "test" archive offloading?

Regards

HannesK
Veeam Software
Posts: 9104
Liked: 1663 times
Joined: Sep 01, 2014 11:46 am
Full Name: Hannes Kasparick
Location: Austria
Contact:

Re: Utilize AWS Direct Connect/Private Link for Capacity Tier

Post by HannesK » 1 person likes this post

Hello,
But is it possible to tag a manually started full backup (test job) on GFS in order to be able to "test" archive offloading?
what I always did during beta was the following

- schedule a job with synthetic full every day
- configure weekly GFS for today
- schedule the backup job to run in a few minutes (do not run the job manually)
- check whether the GFS flag was set in backups -> disk
- SOBR: copy mode and move to archive tier after 0 days. remove the "archive backups only if the..." checkbox to allow that the tiering really happens on the next day
- wait for the next day's synthetic full and archive tier tiering

Best regards,
Hannes

vIdaho1
Influencer
Posts: 11
Liked: 2 times
Joined: May 10, 2021 5:17 pm
Full Name: Andrew Foster
Contact:

Re: Utilize AWS Direct Connect/Private Link for Capacity Tier

Post by vIdaho1 » 2 people like this post

With 11a now released as RTM I just wanted to confirm the kb will be released to address this issue? I did not see anything on https://www.veeam.com/kb4215 regarding adding the functionality for utilizing AWS private link when adding AWS S3 as a capacity tier.

veremin
Product Manager
Posts: 18857
Liked: 1905 times
Joined: Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin
Contact:

Re: Utilize AWS Direct Connect/Private Link for Capacity Tier

Post by veremin » 1 person likes this post

The KB article regarding usage of AWS Privatelink should be published this week. Thanks!

markusc
Novice
Posts: 3
Liked: never
Joined: Oct 13, 2021 1:54 pm
Contact:

Re: Utilize AWS Direct Connect/Private Link for Capacity Tier

Post by markusc »

Hi @veremin, can you please update us about the KB article? We need this feature, to be able to deploy the S3-Proxy with a private IP-Adress.
I just opened a case (number ID: #02467892), without response...
Thank you.

veremin
Product Manager
Posts: 18857
Liked: 1905 times
Joined: Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin
Contact:

Re: Utilize AWS Direct Connect/Private Link for Capacity Tier

Post by veremin »

Sure, I will post back, when support team publish the KB article. Thanks!

HannesK
Veeam Software
Posts: 9104
Liked: 1663 times
Joined: Sep 01, 2014 11:46 am
Full Name: Hannes Kasparick
Location: Austria
Contact:

Re: Utilize AWS Direct Connect/Private Link for Capacity Tier

Post by HannesK »


vIdaho1
Influencer
Posts: 11
Liked: 2 times
Joined: May 10, 2021 5:17 pm
Full Name: Andrew Foster
Contact:

Re: Utilize AWS Direct Connect/Private Link for Capacity Tier

Post by vIdaho1 »

Thank you for posting that. I really hope a VPN connection isn't a requirement of using a PrivateLink. That doesn't make sense to me since we have a private connection for our DC all the way to AWS and the DNS name resolves locally with a local IP. When adding capacity tier as just S3 I am able to put in our PrivateLink endpoint and tier just fine, I just couldn't use a PrivateLink when adding as AWS, so the VPN part has me a little confused.

markusc
Novice
Posts: 3
Liked: never
Joined: Oct 13, 2021 1:54 pm
Contact:

Re: Utilize AWS Direct Connect/Private Link for Capacity Tier

Post by markusc »

@vIdaho1, we have the same issue.
We have a direct private connection to AWS and hope, the VPN connection will not be required to enable a private connection to the S3 buckets

veremin
Product Manager
Posts: 18857
Liked: 1905 times
Joined: Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin
Contact:

Re: Utilize AWS Direct Connect/Private Link for Capacity Tier

Post by veremin »

Actually, it's not a requirement, but rather an additional layer of security. If you are not concerned about this matter, just follow the KB article from the step 2 dismissing VPN configuration part. Thanks!

markusc
Novice
Posts: 3
Liked: never
Joined: Oct 13, 2021 1:54 pm
Contact:

Re: Utilize AWS Direct Connect/Private Link for Capacity Tier

Post by markusc »

Hi @veremin
We just opened a case (Number #05070873) about that. Can you please update us, because we won't test the VPN connection.
Thank you.

veremin
Product Manager
Posts: 18857
Liked: 1905 times
Joined: Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin
Contact:

Re: Utilize AWS Direct Connect/Private Link for Capacity Tier

Post by veremin »

support ticket wrote:I don’t have done this and went directly to step 6, as we want only to create a S3-Glacier bucket to
archive our VMs with VeeamZIP
This is not possible, you cannot use Archive storage outside of Scale-Out Backup Repository. Thanks!

vIdaho1
Influencer
Posts: 11
Liked: 2 times
Joined: May 10, 2021 5:17 pm
Full Name: Andrew Foster
Contact:

Re: Utilize AWS Direct Connect/Private Link for Capacity Tier

Post by vIdaho1 »

@veremin
Can you provide clarification around exactly how we should be editing the XML file as the KB isn't totally clear on whether we should be removing all other S3 endpoint entries and only having a single one for bucket or just adding an additional one for bucket type leaving the several other S3 endpoint entries for our region? Since the KB isn't very clear I tried just adding an additional one to the appropriate region with our vpc endpoint DNS, but it still did not go over our direct connect and private link.

Looking at the KB's example it has this, which only has one S3 endpoint, the bucket one you're adding.

Code: Select all

<Region Id="eu-central-1" Name="EU (Frankfurt)" Type="Global">
<Endpoint Type="S3">bucket.<DNS name></Endpoint>
<Endpoint Type="EC2">ec2.eu-central-1.amazonaws.com</Endpoint>
<Endpoint Type="IAM">iam.amazonaws.com</Endpoint>
<Protocol>HTTP</Protocol>
<Protocol>HTTPS</Protocol>
<LocationConstraint>eu-central-1</LocationConstraint>
<SignatureVersion>4</SignatureVersion>
</Region>
But if you look at that region in the XML file (pasted below) it has multiple S3 endpoints. So did you remove the others and then add the bucket one? Sorry, but this KB is really lacking detail to help us successfully implement this feature we have been waiting for quickly. Like, you should have the VPN is not a requirement because when it says step 1. VPN it makes the reader assume it is a requirement. Then it isn't clear if we should only be adding an additional S3 endpoint, modifying an existing one, or removing all but the bucket one, especially with region example in the KB.

From current XML file on management server.

Code: Select all

  <Region Id="eu-central-1" Name="EU (Frankfurt)" Type="Global">
    <Endpoint Type="S3">s3.eu-central-1.amazonaws.com</Endpoint>
    <Endpoint Type="S3">s3-eu-central-1.amazonaws.com</Endpoint>
    <Endpoint Type="S3">s3.dualstack.eu-central-1.amazonaws.com</Endpoint>
    <Endpoint Type="EC2">ec2.eu-central-1.amazonaws.com</Endpoint>
    <Endpoint Type="IAM">iam.amazonaws.com</Endpoint>
    <Protocol>HTTP</Protocol>
    <Protocol>HTTPS</Protocol>
    <LocationConstraint>eu-central-1</LocationConstraint>
    <SignatureVersion>4</SignatureVersion>
  </Region>

Post Reply

Who is online

Users browsing this forum: No registered users and 2 guests