Discussions related to using object storage as a backup target.
Post Reply
dariusz.tyka
Enthusiast
Posts: 54
Liked: 4 times
Joined: Jan 21, 2019 1:38 pm
Full Name: Dariusz Tyka
Contact:

S3 offload/copy and DirectConnect

Post by dariusz.tyka »

Hi,

have a question on how to best configure backup infrastructure to utilize AWS DirectConnect connection from on-prem to cloud storage.
From our datacenter we have 200Mb line to Internet and 1Gb fiber connection to AWS. Also we have a VPC endpoint/gateway that allows us to access S3 directly from our AWS VPC. We would like to utilize this 1Gb connection to offload/copy backups to S3 storage. What would be the best way to do that? By default Veeam server will try to connect directly to Internet - would an option be to create a gateway server as an EC2 instance and select it within an Veeam S3 repository? Is anyone using such config and it is performing well?

And second question - is there an official document for v10 about minimal permission for AWS cloud account to offload data to S3?
Here https://helpcenter.veeam.com/docs/backu ... ml?ver=100 it is mentioned to grant full admin permission but from security perspective it is not the best idea.

Dariusz
dalbertson
Veeam Software
Posts: 492
Liked: 175 times
Joined: Jul 21, 2015 12:38 pm
Full Name: Dustin Albertson
Contact:

Re: S3 offload/copy and DirectConnect

Post by dalbertson »

Hi,

as far as the minimal permissions for s3 they are listed here. https://helpcenter.veeam.com/docs/backu ... 100#rpasos

and here is the policy template i created

Code: Select all

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "SecureBucketPolicy0",
            "Effect": "Allow",
            "Action": [
                "s3:PutObject",
                "s3:GetObject",
                "s3:AbortMultipartUpload",
                "s3:ListBucket",
                "s3:DeleteObject",
                "s3:GetBucketLocation",
                "s3:ListMultipartUploadParts"
            ],
            "Resource": [
                "arn:aws:s3:::bucketname/*",
                "arn:aws:s3:::bucketname"
            ]
        },
        {
            "Sid": "SecureBucketPolicy1",
            "Effect": "Allow",
            "Action": [
                "s3:ListAllMyBuckets",
                "s3:HeadBucket"
            ],
            "Resource": "*"
        }
    ]
}
Dustin Albertson | Director of Product Management - Cloud & Applications | Veeam Product Management, Alliances
dalbertson
Veeam Software
Posts: 492
Liked: 175 times
Joined: Jul 21, 2015 12:38 pm
Full Name: Dustin Albertson
Contact:

Re: S3 offload/copy and DirectConnect

Post by dalbertson »

Also with the direct connect question you could leverage a public vif to accomplish this.

https://docs.aws.amazon.com/directconne ... linterface

https://docs.aws.amazon.com/directconne ... faces.html
Dustin Albertson | Director of Product Management - Cloud & Applications | Veeam Product Management, Alliances
dariusz.tyka
Enthusiast
Posts: 54
Liked: 4 times
Joined: Jan 21, 2019 1:38 pm
Full Name: Dariusz Tyka
Contact:

Re: S3 offload/copy and DirectConnect

Post by dariusz.tyka »

Thanks for your reply dalbertson. I'll check the minimal S3 permissions. Regarding direct connect public vifs - yes we are aware of it but it will be additional cost to our private vifs we are using now. How about gateway server I've mentioned earlier? Will it work as we can expect?

Dariusz
dalbertson
Veeam Software
Posts: 492
Liked: 175 times
Joined: Jul 21, 2015 12:38 pm
Full Name: Dustin Albertson
Contact:

Re: S3 offload/copy and DirectConnect

Post by dalbertson »

Yes that would work as well
Dustin Albertson | Director of Product Management - Cloud & Applications | Veeam Product Management, Alliances
dariusz.tyka
Enthusiast
Posts: 54
Liked: 4 times
Joined: Jan 21, 2019 1:38 pm
Full Name: Dariusz Tyka
Contact:

Re: S3 offload/copy and DirectConnect

Post by dariusz.tyka »

Hi,

I've deployed a gateway server on EC2 instance and configured this gateway server within our object storage repository. I can see in logs that data is offloaded to S3 using gateway server. Gateway server has access to S3 via S3 endpoint and via proxy server. How I can be sure S3 offload is done using S3 endpoint not via proxy? Or proxy is not supported and gateway server needs to have direct Internet access to offload data to S3? Like it is mentioned here for backup repository server?
https://helpcenter.veeam.com/docs/backu ... ml?ver=100

Also screen from tracert command on gateway server:
https://ibb.co/JrXq4Lk

Dariusz
oleg.feoktistov
Veeam Software
Posts: 1912
Liked: 635 times
Joined: Sep 25, 2019 10:32 am
Full Name: Oleg Feoktistov
Contact:

Re: S3 offload/copy and DirectConnect

Post by oleg.feoktistov » 1 person likes this post

Hi Dariusz,
How I can be sure S3 offload is done using S3 endpoint not via proxy?
Have you tried configuring flow logs for VPC subnet your proxy is placed in and tracing required traffic through either CloudWatch or archived logs in S3 bucket?
Or proxy is not supported and gateway server needs to have direct Internet access to offload data to S3?
Never tested double proxying, but as per my understanding, as long as routing is adjusted correctly between EC2 gateway and proxy, it should work.

Best regards,
Oleg
popjls
Enthusiast
Posts: 52
Liked: 5 times
Joined: Jun 25, 2018 3:41 am
Contact:

Re: S3 offload/copy and DirectConnect

Post by popjls »

Im glad someone else has this issue as I do too. The AWS Restore uses the public vif and not the private vif (to the routed VPC, we have both public and private) when restoring and I'm hoping this can be changed down the track or a work around is possible. I'll look into proxying/gateway server too.
dariusz.tyka
Enthusiast
Posts: 54
Liked: 4 times
Joined: Jan 21, 2019 1:38 pm
Full Name: Dariusz Tyka
Contact:

Re: S3 offload/copy and DirectConnect

Post by dariusz.tyka »

Hi oleg.feoktistov,

thank you for your answer. But I still do not know if gateway server requires direct Internet access to store/receive backups to/from S3 bucket? Or it can work via the proxy? According to this info direct internet access is required:
https://helpcenter.veeam.com/docs/backu ... ml?ver=100

Dariusz
dalbertson
Veeam Software
Posts: 492
Liked: 175 times
Joined: Jul 21, 2015 12:38 pm
Full Name: Dustin Albertson
Contact:

Re: S3 offload/copy and DirectConnect

Post by dalbertson » 1 person likes this post

It should be able to work via a proxy, as its really all DNS. If the public s3 endpoint replies back via the proxy then it will work.

To test if the gateway server is using the VPC endpoint here is a method.

1.) Log into the EC2 instance in the VPC (Gateway server)
2.) Install the AWS CLI client. (https://docs.aws.amazon.com/cli/latest/ ... cliv2.html)
3.) run the command "aws ec2 describe-prefix-lists" , or if using powershell "Get-EC2PrefixList"

The output of this command will contain the VPC endpoints prefix lists, you could then apply this to a policy on the S3 bucket to only allow this VPC prefix to access.
If the backup still offloads then you know it is going across the VPC...if it doesn't offload then its trying to go to the public endpoints.

Code: Select all

{
  "Version": "2008-10-17",
  "Statement": [
    {
      "Effect": "Deny",
      "Principal": "*",
      "Action": [
        "s3:ListBucket"
      ],
      "Resource": [
        "arn:aws:s3:::mybucket"
      ],
      "Condition": {
        "StringNotEquals": {
          "aws:sourceVpc": [
            "VPCPrefixFromCommand"
          ]
        }
      }
    }
  ]
}
Dustin Albertson | Director of Product Management - Cloud & Applications | Veeam Product Management, Alliances
dalbertson
Veeam Software
Posts: 492
Liked: 175 times
Joined: Jul 21, 2015 12:38 pm
Full Name: Dustin Albertson
Contact:

Re: S3 offload/copy and DirectConnect

Post by dalbertson »

To further add to what i mentioned above, here is a screenshot of the command process,

install aws cli and open powershell.
type aws --version to verify its installed, run aws configure to add aws account with permissions to check (this can be a temp iam user), as you can see i added Access key, secret, default region, and output as json.
then i ran the command Get-EC2PrefixList.

in the output you can see the PrefixListId and that is what you would copy into the policy i put above to limit access to the bucket from only this VPC and validate that traffic is coming from the right area.

Image
Dustin Albertson | Director of Product Management - Cloud & Applications | Veeam Product Management, Alliances
popjls
Enthusiast
Posts: 52
Liked: 5 times
Joined: Jun 25, 2018 3:41 am
Contact:

Re: S3 offload/copy and DirectConnect

Post by popjls »

When I did a restore and ticked the "use a proxy" method, it came back and said there wasn't a security group associated with our external address to the connection point which was/is s3.amazon etc etc. So that method doesn't work but I could try spinning up a proxy inside in ec2 and try that offload method but that's a pain to some degree.
dariusz.tyka
Enthusiast
Posts: 54
Liked: 4 times
Joined: Jan 21, 2019 1:38 pm
Full Name: Dariusz Tyka
Contact:

Re: S3 offload/copy and DirectConnect

Post by dariusz.tyka »

Hi,

I've made some more tests and here are the findings:
1. installed AWS CLI - but didn't do initial configuration. EC2 instance uses asigned AWS role to access S3 buckets.
- output from aws ec2 describe-prefix-lists:
Could not connect to the endpoint URL: "https://ec2.eu-west-1.amazonaws.com/"

- ping ec2.eu-west-1.amazonaws.com:
Pinging eu-west-1.ec2.amazonaws.com [54.239.39.130] with 32 bytes of data:
Request timed out.
Request timed out.
Request timed out.
Request timed out.

- ping s3.eu-west-1.amazonaws.com:
Pinging s3.eu-west-1.amazonaws.com [52.218.89.123] with 32 bytes of data:
Reply from 52.218.89.123: bytes=32 time<1ms TTL=58
Reply from 52.218.89.123: bytes=32 time<1ms TTL=57
Reply from 52.218.89.123: bytes=32 time<1ms TTL=57
Reply from 52.218.89.123: bytes=32 time<1ms TTL=57

- aws s3 ls - I was able to list all S3 buckets
- aws s3 cp sth to S3://bucket_name:
Completed 397.2 MiB/3.2 GiB (97.0 MiB/s) with 6 file(s) remaining.

So it looks like it is copying to S3 just fine using S3 endpoint with almost 100MB/s. As our Internet connection in datacenter is only 300Mbps.

When I try to restore VM from S3 bucket (still I have more recent existing on-prem backups for this VM) it is restoring fine. But I see a lot of traffic between gateway server and backup server. I would expect only missing chunks are downloaded from S3 and all remaining are fetched from on-prem backup. But why I see almost same amount of data going out from backup to gateway and from gateway to backup server?
Are those chunks from local and S3 data merged on gateway or backup server? Looks for me on gateway one?

Dariusz
dalbertson
Veeam Software
Posts: 492
Liked: 175 times
Joined: Jul 21, 2015 12:38 pm
Full Name: Dustin Albertson
Contact:

Re: S3 offload/copy and DirectConnect

Post by dalbertson »

You are correct, when we restore from S3 we only pull the objects that we cant pull from the performance tier in order to save egress charges. The traffic you see if the flow of data and is dependant upon where you are restoring to. If on prem then the data you see from the gateway to VBR is the data from s3.
Dustin Albertson | Director of Product Management - Cloud & Applications | Veeam Product Management, Alliances
andy.yin
Novice
Posts: 3
Liked: never
Joined: Nov 22, 2017 1:56 am
Full Name: Andy Yin
Contact:

Re: S3 offload/copy and DirectConnect

Post by andy.yin »

if the selected gateway server is not available?
the offload fails over to the SOBR Extend server?
Gostev
Chief Product Officer
Posts: 31458
Liked: 6648 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: S3 offload/copy and DirectConnect

Post by Gostev »

I don't think so. If you specify a gateway, we will stick to it. If you don't enable this option, then we will go to object storage directly from SOBR extents.
veremin
Product Manager
Posts: 20270
Liked: 2252 times
Joined: Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin
Contact:

Re: S3 offload/copy and DirectConnect

Post by veremin »

Anton's correct, backup server does not fail over to performance extents, if the specified gateway is unavailable. Thanks!
Post Reply

Who is online

Users browsing this forum: Bing [Bot], Ivan239 and 6 guests