S3 offload/copy and DirectConnect

[dariusz.tyka]

Hi,

have a question on how to best configure backup infrastructure to utilize AWS DirectConnect connection from on-prem to cloud storage.
From our datacenter we have 200Mb line to Internet and 1Gb fiber connection to AWS. Also we have a VPC endpoint/gateway that allows us to access S3 directly from our AWS VPC. We would like to utilize this 1Gb connection to offload/copy backups to S3 storage. What would be the best way to do that? By default Veeam server will try to connect directly to Internet - would an option be to create a gateway server as an EC2 instance and select it within an Veeam S3 repository? Is anyone using such config and it is performing well?

And second question - is there an official document for v10 about minimal permission for AWS cloud account to offload data to S3?
Here https://helpcenter.veeam.com/docs/backu ... ml?ver=100 it is mentioned to grant full admin permission but from security perspective it is not the best idea.

Dariusz

[dalbertson]

Hi,

as far as the minimal permissions for s3 they are listed here. https://helpcenter.veeam.com/docs/backu ... 100#rpasos

and here is the policy template i created

Code: Select all

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "SecureBucketPolicy0",
            "Effect": "Allow",
            "Action": [
                "s3:PutObject",
                "s3:GetObject",
                "s3:AbortMultipartUpload",
                "s3:ListBucket",
                "s3:DeleteObject",
                "s3:GetBucketLocation",
                "s3:ListMultipartUploadParts"
            ],
            "Resource": [
                "arn:aws:s3:::bucketname/*",
                "arn:aws:s3:::bucketname"
            ]
        },
        {
            "Sid": "SecureBucketPolicy1",
            "Effect": "Allow",
            "Action": [
                "s3:ListAllMyBuckets",
                "s3:HeadBucket"
            ],
            "Resource": "*"
        }
    ]
}

[dalbertson]

Also with the direct connect question you could leverage a public vif to accomplish this.

https://docs.aws.amazon.com/directconne ... linterface

https://docs.aws.amazon.com/directconne ... faces.html

[dariusz.tyka]

Thanks for your reply dalbertson. I'll check the minimal S3 permissions. Regarding direct connect public vifs - yes we are aware of it but it will be additional cost to our private vifs we are using now. How about gateway server I've mentioned earlier? Will it work as we can expect?

Dariusz

[dalbertson]

Yes that would work as well

[dariusz.tyka]

Hi,

I've deployed a gateway server on EC2 instance and configured this gateway server within our object storage repository. I can see in logs that data is offloaded to S3 using gateway server. Gateway server has access to S3 via S3 endpoint and via proxy server. How I can be sure S3 offload is done using S3 endpoint not via proxy? Or proxy is not supported and gateway server needs to have direct Internet access to offload data to S3? Like it is mentioned here for backup repository server?
https://helpcenter.veeam.com/docs/backu ... ml?ver=100

Also screen from tracert command on gateway server:
https://ibb.co/JrXq4Lk

Dariusz

[oleg.feoktistov]

Hi Dariusz,

How I can be sure S3 offload is done using S3 endpoint not via proxy?

Have you tried configuring flow logs for VPC subnet your proxy is placed in and tracing required traffic through either CloudWatch or archived logs in S3 bucket?

Or proxy is not supported and gateway server needs to have direct Internet access to offload data to S3?

Never tested double proxying, but as per my understanding, as long as routing is adjusted correctly between EC2 gateway and proxy, it should work.

Best regards,
Oleg

[popjls]

Im glad someone else has this issue as I do too. The AWS Restore uses the public vif and not the private vif (to the routed VPC, we have both public and private) when restoring and I'm hoping this can be changed down the track or a work around is possible. I'll look into proxying/gateway server too.

[dariusz.tyka]

Hi oleg.feoktistov,

thank you for your answer. But I still do not know if gateway server requires direct Internet access to store/receive backups to/from S3 bucket? Or it can work via the proxy? According to this info direct internet access is required:
https://helpcenter.veeam.com/docs/backu ... ml?ver=100

Dariusz

[dalbertson]

It should be able to work via a proxy, as its really all DNS. If the public s3 endpoint replies back via the proxy then it will work.

To test if the gateway server is using the VPC endpoint here is a method.

1.) Log into the EC2 instance in the VPC (Gateway server)
2.) Install the AWS CLI client. (https://docs.aws.amazon.com/cli/latest/ ... cliv2.html)
3.) run the command "aws ec2 describe-prefix-lists" , or if using powershell "Get-EC2PrefixList"

The output of this command will contain the VPC endpoints prefix lists, you could then apply this to a policy on the S3 bucket to only allow this VPC prefix to access.
If the backup still offloads then you know it is going across the VPC...if it doesn't offload then its trying to go to the public endpoints.

Code: Select all

{
  "Version": "2008-10-17",
  "Statement": [
    {
      "Effect": "Deny",
      "Principal": "*",
      "Action": [
        "s3:ListBucket"
      ],
      "Resource": [
        "arn:aws:s3:::mybucket"
      ],
      "Condition": {
        "StringNotEquals": {
          "aws:sourceVpc": [
            "VPCPrefixFromCommand"
          ]
        }
      }
    }
  ]
}

[dalbertson]

To further add to what i mentioned above, here is a screenshot of the command process,

install aws cli and open powershell.
type aws --version to verify its installed, run aws configure to add aws account with permissions to check (this can be a temp iam user), as you can see i added Access key, secret, default region, and output as json.
then i ran the command Get-EC2PrefixList.

in the output you can see the PrefixListId and that is what you would copy into the policy i put above to limit access to the bucket from only this VPC and validate that traffic is coming from the right area.

[popjls]

When I did a restore and ticked the "use a proxy" method, it came back and said there wasn't a security group associated with our external address to the connection point which was/is s3.amazon etc etc. So that method doesn't work but I could try spinning up a proxy inside in ec2 and try that offload method but that's a pain to some degree.

[dariusz.tyka]

Hi,

I've made some more tests and here are the findings:
1. installed AWS CLI - but didn't do initial configuration. EC2 instance uses asigned AWS role to access S3 buckets.
- output from aws ec2 describe-prefix-lists:
Could not connect to the endpoint URL: "https://ec2.eu-west-1.amazonaws.com/"

- ping ec2.eu-west-1.amazonaws.com:
Pinging eu-west-1.ec2.amazonaws.com [54.239.39.130] with 32 bytes of data:
Request timed out.
Request timed out.
Request timed out.
Request timed out.

- ping s3.eu-west-1.amazonaws.com:
Pinging s3.eu-west-1.amazonaws.com [52.218.89.123] with 32 bytes of data:
Reply from 52.218.89.123: bytes=32 time<1ms TTL=58
Reply from 52.218.89.123: bytes=32 time<1ms TTL=57
Reply from 52.218.89.123: bytes=32 time<1ms TTL=57
Reply from 52.218.89.123: bytes=32 time<1ms TTL=57

- aws s3 ls - I was able to list all S3 buckets
- aws s3 cp sth to S3://bucket_name:
Completed 397.2 MiB/3.2 GiB (97.0 MiB/s) with 6 file(s) remaining.

So it looks like it is copying to S3 just fine using S3 endpoint with almost 100MB/s. As our Internet connection in datacenter is only 300Mbps.

When I try to restore VM from S3 bucket (still I have more recent existing on-prem backups for this VM) it is restoring fine. But I see a lot of traffic between gateway server and backup server. I would expect only missing chunks are downloaded from S3 and all remaining are fetched from on-prem backup. But why I see almost same amount of data going out from backup to gateway and from gateway to backup server?
Are those chunks from local and S3 data merged on gateway or backup server? Looks for me on gateway one?

Dariusz

[dalbertson]

You are correct, when we restore from S3 we only pull the objects that we cant pull from the performance tier in order to save egress charges. The traffic you see if the flow of data and is dependant upon where you are restoring to. If on prem then the data you see from the gateway to VBR is the data from s3.

[andy.yin]

if the selected gateway server is not available?
the offload fails over to the SOBR Extend server?

[Gostev]

I don't think so. If you specify a gateway, we will stick to it. If you don't enable this option, then we will go to object storage directly from SOBR extents.

[veremin]

Anton's correct, backup server does not fail over to performance extents, if the specified gateway is unavailable. Thanks!