-
- Enthusiast
- Posts: 57
- Liked: 4 times
- Joined: Jan 21, 2019 1:38 pm
- Full Name: Dariusz Tyka
- Contact:
S3 offload/copy and DirectConnect
Hi,
have a question on how to best configure backup infrastructure to utilize AWS DirectConnect connection from on-prem to cloud storage.
From our datacenter we have 200Mb line to Internet and 1Gb fiber connection to AWS. Also we have a VPC endpoint/gateway that allows us to access S3 directly from our AWS VPC. We would like to utilize this 1Gb connection to offload/copy backups to S3 storage. What would be the best way to do that? By default Veeam server will try to connect directly to Internet - would an option be to create a gateway server as an EC2 instance and select it within an Veeam S3 repository? Is anyone using such config and it is performing well?
And second question - is there an official document for v10 about minimal permission for AWS cloud account to offload data to S3?
Here https://helpcenter.veeam.com/docs/backu ... ml?ver=100 it is mentioned to grant full admin permission but from security perspective it is not the best idea.
Dariusz
have a question on how to best configure backup infrastructure to utilize AWS DirectConnect connection from on-prem to cloud storage.
From our datacenter we have 200Mb line to Internet and 1Gb fiber connection to AWS. Also we have a VPC endpoint/gateway that allows us to access S3 directly from our AWS VPC. We would like to utilize this 1Gb connection to offload/copy backups to S3 storage. What would be the best way to do that? By default Veeam server will try to connect directly to Internet - would an option be to create a gateway server as an EC2 instance and select it within an Veeam S3 repository? Is anyone using such config and it is performing well?
And second question - is there an official document for v10 about minimal permission for AWS cloud account to offload data to S3?
Here https://helpcenter.veeam.com/docs/backu ... ml?ver=100 it is mentioned to grant full admin permission but from security perspective it is not the best idea.
Dariusz
-
- Veeam Software
- Posts: 492
- Liked: 175 times
- Joined: Jul 21, 2015 12:38 pm
- Full Name: Dustin Albertson
- Contact:
Re: S3 offload/copy and DirectConnect
Hi,
as far as the minimal permissions for s3 they are listed here. https://helpcenter.veeam.com/docs/backu ... 100#rpasos
and here is the policy template i created
as far as the minimal permissions for s3 they are listed here. https://helpcenter.veeam.com/docs/backu ... 100#rpasos
and here is the policy template i created
Code: Select all
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "SecureBucketPolicy0",
"Effect": "Allow",
"Action": [
"s3:PutObject",
"s3:GetObject",
"s3:AbortMultipartUpload",
"s3:ListBucket",
"s3:DeleteObject",
"s3:GetBucketLocation",
"s3:ListMultipartUploadParts"
],
"Resource": [
"arn:aws:s3:::bucketname/*",
"arn:aws:s3:::bucketname"
]
},
{
"Sid": "SecureBucketPolicy1",
"Effect": "Allow",
"Action": [
"s3:ListAllMyBuckets",
"s3:HeadBucket"
],
"Resource": "*"
}
]
}
Dustin Albertson | Director of Product Management - Cloud & Applications | Veeam Product Management, Alliances
-
- Veeam Software
- Posts: 492
- Liked: 175 times
- Joined: Jul 21, 2015 12:38 pm
- Full Name: Dustin Albertson
- Contact:
Re: S3 offload/copy and DirectConnect
Also with the direct connect question you could leverage a public vif to accomplish this.
https://docs.aws.amazon.com/directconne ... linterface
https://docs.aws.amazon.com/directconne ... faces.html
https://docs.aws.amazon.com/directconne ... linterface
https://docs.aws.amazon.com/directconne ... faces.html
Dustin Albertson | Director of Product Management - Cloud & Applications | Veeam Product Management, Alliances
-
- Enthusiast
- Posts: 57
- Liked: 4 times
- Joined: Jan 21, 2019 1:38 pm
- Full Name: Dariusz Tyka
- Contact:
Re: S3 offload/copy and DirectConnect
Thanks for your reply dalbertson. I'll check the minimal S3 permissions. Regarding direct connect public vifs - yes we are aware of it but it will be additional cost to our private vifs we are using now. How about gateway server I've mentioned earlier? Will it work as we can expect?
Dariusz
Dariusz
-
- Veeam Software
- Posts: 492
- Liked: 175 times
- Joined: Jul 21, 2015 12:38 pm
- Full Name: Dustin Albertson
- Contact:
Re: S3 offload/copy and DirectConnect
Yes that would work as well
Dustin Albertson | Director of Product Management - Cloud & Applications | Veeam Product Management, Alliances
-
- Enthusiast
- Posts: 57
- Liked: 4 times
- Joined: Jan 21, 2019 1:38 pm
- Full Name: Dariusz Tyka
- Contact:
Re: S3 offload/copy and DirectConnect
Hi,
I've deployed a gateway server on EC2 instance and configured this gateway server within our object storage repository. I can see in logs that data is offloaded to S3 using gateway server. Gateway server has access to S3 via S3 endpoint and via proxy server. How I can be sure S3 offload is done using S3 endpoint not via proxy? Or proxy is not supported and gateway server needs to have direct Internet access to offload data to S3? Like it is mentioned here for backup repository server?
https://helpcenter.veeam.com/docs/backu ... ml?ver=100
Also screen from tracert command on gateway server:
https://ibb.co/JrXq4Lk
Dariusz
I've deployed a gateway server on EC2 instance and configured this gateway server within our object storage repository. I can see in logs that data is offloaded to S3 using gateway server. Gateway server has access to S3 via S3 endpoint and via proxy server. How I can be sure S3 offload is done using S3 endpoint not via proxy? Or proxy is not supported and gateway server needs to have direct Internet access to offload data to S3? Like it is mentioned here for backup repository server?
https://helpcenter.veeam.com/docs/backu ... ml?ver=100
Also screen from tracert command on gateway server:
https://ibb.co/JrXq4Lk
Dariusz
-
- Veeam Software
- Posts: 2010
- Liked: 669 times
- Joined: Sep 25, 2019 10:32 am
- Full Name: Oleg Feoktistov
- Contact:
Re: S3 offload/copy and DirectConnect
Hi Dariusz,
Best regards,
Oleg
Have you tried configuring flow logs for VPC subnet your proxy is placed in and tracing required traffic through either CloudWatch or archived logs in S3 bucket?How I can be sure S3 offload is done using S3 endpoint not via proxy?
Never tested double proxying, but as per my understanding, as long as routing is adjusted correctly between EC2 gateway and proxy, it should work.Or proxy is not supported and gateway server needs to have direct Internet access to offload data to S3?
Best regards,
Oleg
-
- Enthusiast
- Posts: 57
- Liked: 5 times
- Joined: Jun 25, 2018 3:41 am
- Contact:
Re: S3 offload/copy and DirectConnect
Im glad someone else has this issue as I do too. The AWS Restore uses the public vif and not the private vif (to the routed VPC, we have both public and private) when restoring and I'm hoping this can be changed down the track or a work around is possible. I'll look into proxying/gateway server too.
-
- Enthusiast
- Posts: 57
- Liked: 4 times
- Joined: Jan 21, 2019 1:38 pm
- Full Name: Dariusz Tyka
- Contact:
Re: S3 offload/copy and DirectConnect
Hi oleg.feoktistov,
thank you for your answer. But I still do not know if gateway server requires direct Internet access to store/receive backups to/from S3 bucket? Or it can work via the proxy? According to this info direct internet access is required:
https://helpcenter.veeam.com/docs/backu ... ml?ver=100
Dariusz
thank you for your answer. But I still do not know if gateway server requires direct Internet access to store/receive backups to/from S3 bucket? Or it can work via the proxy? According to this info direct internet access is required:
https://helpcenter.veeam.com/docs/backu ... ml?ver=100
Dariusz
-
- Veeam Software
- Posts: 492
- Liked: 175 times
- Joined: Jul 21, 2015 12:38 pm
- Full Name: Dustin Albertson
- Contact:
Re: S3 offload/copy and DirectConnect
It should be able to work via a proxy, as its really all DNS. If the public s3 endpoint replies back via the proxy then it will work.
To test if the gateway server is using the VPC endpoint here is a method.
1.) Log into the EC2 instance in the VPC (Gateway server)
2.) Install the AWS CLI client. (https://docs.aws.amazon.com/cli/latest/ ... cliv2.html)
3.) run the command "aws ec2 describe-prefix-lists" , or if using powershell "Get-EC2PrefixList"
The output of this command will contain the VPC endpoints prefix lists, you could then apply this to a policy on the S3 bucket to only allow this VPC prefix to access.
If the backup still offloads then you know it is going across the VPC...if it doesn't offload then its trying to go to the public endpoints.
To test if the gateway server is using the VPC endpoint here is a method.
1.) Log into the EC2 instance in the VPC (Gateway server)
2.) Install the AWS CLI client. (https://docs.aws.amazon.com/cli/latest/ ... cliv2.html)
3.) run the command "aws ec2 describe-prefix-lists" , or if using powershell "Get-EC2PrefixList"
The output of this command will contain the VPC endpoints prefix lists, you could then apply this to a policy on the S3 bucket to only allow this VPC prefix to access.
If the backup still offloads then you know it is going across the VPC...if it doesn't offload then its trying to go to the public endpoints.
Code: Select all
{
"Version": "2008-10-17",
"Statement": [
{
"Effect": "Deny",
"Principal": "*",
"Action": [
"s3:ListBucket"
],
"Resource": [
"arn:aws:s3:::mybucket"
],
"Condition": {
"StringNotEquals": {
"aws:sourceVpc": [
"VPCPrefixFromCommand"
]
}
}
}
]
}
Dustin Albertson | Director of Product Management - Cloud & Applications | Veeam Product Management, Alliances
-
- Veeam Software
- Posts: 492
- Liked: 175 times
- Joined: Jul 21, 2015 12:38 pm
- Full Name: Dustin Albertson
- Contact:
Re: S3 offload/copy and DirectConnect
To further add to what i mentioned above, here is a screenshot of the command process,
install aws cli and open powershell.
type aws --version to verify its installed, run aws configure to add aws account with permissions to check (this can be a temp iam user), as you can see i added Access key, secret, default region, and output as json.
then i ran the command Get-EC2PrefixList.
in the output you can see the PrefixListId and that is what you would copy into the policy i put above to limit access to the bucket from only this VPC and validate that traffic is coming from the right area.
install aws cli and open powershell.
type aws --version to verify its installed, run aws configure to add aws account with permissions to check (this can be a temp iam user), as you can see i added Access key, secret, default region, and output as json.
then i ran the command Get-EC2PrefixList.
in the output you can see the PrefixListId and that is what you would copy into the policy i put above to limit access to the bucket from only this VPC and validate that traffic is coming from the right area.
Dustin Albertson | Director of Product Management - Cloud & Applications | Veeam Product Management, Alliances
-
- Enthusiast
- Posts: 57
- Liked: 5 times
- Joined: Jun 25, 2018 3:41 am
- Contact:
Re: S3 offload/copy and DirectConnect
When I did a restore and ticked the "use a proxy" method, it came back and said there wasn't a security group associated with our external address to the connection point which was/is s3.amazon etc etc. So that method doesn't work but I could try spinning up a proxy inside in ec2 and try that offload method but that's a pain to some degree.
-
- Enthusiast
- Posts: 57
- Liked: 4 times
- Joined: Jan 21, 2019 1:38 pm
- Full Name: Dariusz Tyka
- Contact:
Re: S3 offload/copy and DirectConnect
Hi,
I've made some more tests and here are the findings:
1. installed AWS CLI - but didn't do initial configuration. EC2 instance uses asigned AWS role to access S3 buckets.
- output from aws ec2 describe-prefix-lists:
Could not connect to the endpoint URL: "https://ec2.eu-west-1.amazonaws.com/"
- ping ec2.eu-west-1.amazonaws.com:
Pinging eu-west-1.ec2.amazonaws.com [54.239.39.130] with 32 bytes of data:
Request timed out.
Request timed out.
Request timed out.
Request timed out.
- ping s3.eu-west-1.amazonaws.com:
Pinging s3.eu-west-1.amazonaws.com [52.218.89.123] with 32 bytes of data:
Reply from 52.218.89.123: bytes=32 time<1ms TTL=58
Reply from 52.218.89.123: bytes=32 time<1ms TTL=57
Reply from 52.218.89.123: bytes=32 time<1ms TTL=57
Reply from 52.218.89.123: bytes=32 time<1ms TTL=57
- aws s3 ls - I was able to list all S3 buckets
- aws s3 cp sth to S3://bucket_name:
Completed 397.2 MiB/3.2 GiB (97.0 MiB/s) with 6 file(s) remaining.
So it looks like it is copying to S3 just fine using S3 endpoint with almost 100MB/s. As our Internet connection in datacenter is only 300Mbps.
When I try to restore VM from S3 bucket (still I have more recent existing on-prem backups for this VM) it is restoring fine. But I see a lot of traffic between gateway server and backup server. I would expect only missing chunks are downloaded from S3 and all remaining are fetched from on-prem backup. But why I see almost same amount of data going out from backup to gateway and from gateway to backup server?
Are those chunks from local and S3 data merged on gateway or backup server? Looks for me on gateway one?
Dariusz
I've made some more tests and here are the findings:
1. installed AWS CLI - but didn't do initial configuration. EC2 instance uses asigned AWS role to access S3 buckets.
- output from aws ec2 describe-prefix-lists:
Could not connect to the endpoint URL: "https://ec2.eu-west-1.amazonaws.com/"
- ping ec2.eu-west-1.amazonaws.com:
Pinging eu-west-1.ec2.amazonaws.com [54.239.39.130] with 32 bytes of data:
Request timed out.
Request timed out.
Request timed out.
Request timed out.
- ping s3.eu-west-1.amazonaws.com:
Pinging s3.eu-west-1.amazonaws.com [52.218.89.123] with 32 bytes of data:
Reply from 52.218.89.123: bytes=32 time<1ms TTL=58
Reply from 52.218.89.123: bytes=32 time<1ms TTL=57
Reply from 52.218.89.123: bytes=32 time<1ms TTL=57
Reply from 52.218.89.123: bytes=32 time<1ms TTL=57
- aws s3 ls - I was able to list all S3 buckets
- aws s3 cp sth to S3://bucket_name:
Completed 397.2 MiB/3.2 GiB (97.0 MiB/s) with 6 file(s) remaining.
So it looks like it is copying to S3 just fine using S3 endpoint with almost 100MB/s. As our Internet connection in datacenter is only 300Mbps.
When I try to restore VM from S3 bucket (still I have more recent existing on-prem backups for this VM) it is restoring fine. But I see a lot of traffic between gateway server and backup server. I would expect only missing chunks are downloaded from S3 and all remaining are fetched from on-prem backup. But why I see almost same amount of data going out from backup to gateway and from gateway to backup server?
Are those chunks from local and S3 data merged on gateway or backup server? Looks for me on gateway one?
Dariusz
-
- Veeam Software
- Posts: 492
- Liked: 175 times
- Joined: Jul 21, 2015 12:38 pm
- Full Name: Dustin Albertson
- Contact:
Re: S3 offload/copy and DirectConnect
You are correct, when we restore from S3 we only pull the objects that we cant pull from the performance tier in order to save egress charges. The traffic you see if the flow of data and is dependant upon where you are restoring to. If on prem then the data you see from the gateway to VBR is the data from s3.
Dustin Albertson | Director of Product Management - Cloud & Applications | Veeam Product Management, Alliances
-
- Novice
- Posts: 3
- Liked: never
- Joined: Nov 22, 2017 1:56 am
- Full Name: Andy Yin
- Contact:
Re: S3 offload/copy and DirectConnect
if the selected gateway server is not available?
the offload fails over to the SOBR Extend server?
the offload fails over to the SOBR Extend server?
-
- Chief Product Officer
- Posts: 31754
- Liked: 7259 times
- Joined: Jan 01, 2006 1:01 am
- Location: Baar, Switzerland
- Contact:
Re: S3 offload/copy and DirectConnect
I don't think so. If you specify a gateway, we will stick to it. If you don't enable this option, then we will go to object storage directly from SOBR extents.
-
- Product Manager
- Posts: 20368
- Liked: 2288 times
- Joined: Oct 26, 2012 3:28 pm
- Full Name: Vladimir Eremin
- Contact:
Re: S3 offload/copy and DirectConnect
Anton's correct, backup server does not fail over to performance extents, if the specified gateway is unavailable. Thanks!
Who is online
Users browsing this forum: No registered users and 10 guests