Agentless, cloud-native backup for Amazon AWS
Post Reply
ScottChapman
Novice
Posts: 4
Liked: never
Joined: Oct 15, 2021 9:18 pm
Full Name: Scott Chapman
Contact:

AWS Instance Restore Features required

Post by ScottChapman »

Today, restoring instances (specifically to a new account/region/AZ) requires more manual effort than it should, and requires (at least in our use case) manual effort after the instances are up to get their networking working.
  1. There's no way to specify the IP address. Ideally this should be prepopulated with the IP address of the original server.
  2. The user data from the original instance is not used on the startup of the new instance. In our case (on Ubuntu 20.04) that means cloud-init notices that it's on a new instance and the hostname gets re-written to a default using the IP address (which due to #1 is some new address). If the original user data was passed through, then at least cloud-init would assign the correct hostname.
  3. Ideally, once you've selected the account and region, you'd go out and find the VPCs and subnets and automatically pre-select the same-named VPCs/subnets, with of course an option to override them as need be.
  4. When doing tests, it's tedious to go through the process for every server. Ideally we'd select a number of instances (presumably from the same subnet, or even better let us select all the instances in a subnet with a single click) and then select parameters for all of those servers: i.e. we'd like them to go back to the same-named VPC/subnet/security group/etc. Allowing overrides of course, but if we need to move everything from one AZ to another, we generally would want it to all come back up the same way it was.
  5. Ideally we'd be able to save our recovery plan for later execution. Ideally that plan would be built generically not instance by instance. I.E. restore all instances from this subnet or with a particular tag, so that when we go to run that recovery plan, we don't have to update it for changes in the instance inventory. I would envision this ideally being a mapping from original to new VPC/subnet/etc. names that is then applied to all instances selected for restoration.
#1 and #2 are already done by at least one of your major competitors and seems to be a very basic requirement that is hard to understand why they're not already there. How do people work around their instances coming up with different IP addresses and different host names?

#3, #4, and #5 are at least partially done by competitors as well. Those also would be easier with VPC recovery which still isn't working for us on v4. (Yes we have a problem open for that.) But at least we can manually create the VPC structures needed.

The goal would be able to quickly and easily restore from one account, region, and/or az to another, which at the moment it's certainly not quick and easy.

We liked the design and usability of Veeam for AWS better than the competitors, but the recovery issues almost pushed us to a competing product. The closest competing product had it's own (less significant) recovery issues, but if it would have been a little bit better we probably would have chosen it. The saving grace at the moment is that we hope to not have to restore on a regular basis, although we do have a goal of performing regular DR tests.

nielsengelen
Veeam Software
Posts: 4657
Liked: 966 times
Joined: Jul 15, 2013 11:09 am
Full Name: Niels Engelen
Contact:

Re: AWS Instance Restore Features required

Post by nielsengelen »

1. We preserve the IP when u restore to the original location. To another location, this should also work if the same range is available there so it looks to me this is a networking issue within the VPCs but that is hard to tell without insight.
2. We don't adjust anything in-guest so this should work if everything is set up correct regarding networking / IP passing out etc.
3. If u restore to the original location, we take the original available one. If you want to change them, then select restore to another location but use the same region. This is by design possible already. We don't want customers to waste time in case of a quick restore to the original location with all these settings which we already know.
4. This is possible by multi selecting the instances in the restore wizard (same applies here as my answer to question 3).
5. Am I correct that you are looking to have a sort of "restore templates" in which you add a list of instances and you could "failover" with the click of 1 button (sort of speak)?

Feel free to send me a direct message to dig deeper in these points as I may have misunderstood things.
Personal blog: https://foonet.be
GitHub: https://github.com/nielsengelen

ScottChapman
Novice
Posts: 4
Liked: never
Joined: Oct 15, 2021 9:18 pm
Full Name: Scott Chapman
Contact:

Re: AWS Instance Restore Features required

Post by ScottChapman »

  1. I have not seen this work yet.
  2. This is specifically related to how cloud-init works and the lack of the "User Data" as was specified for the original launch of the AMI.
  3. You present the VPCs/subnets for selectin but the user has to select them. If a VPC/subnet exists with the same name, it seems like you could select that by default already.
  4. Somehow I missed you could select multiple at the same time. Excellent!
  5. Yes, the idea is to get to easy button to do a restore. Presumably with some display of "this is what we're about to do".
I'll DM you about points 1 and 2 in particular.

nielsengelen
Veeam Software
Posts: 4657
Liked: 966 times
Joined: Jul 15, 2013 11:09 am
Full Name: Niels Engelen
Contact:

Re: AWS Instance Restore Features required

Post by nielsengelen »

3. I understand your point of view there and while it may make sense to present the one with the same name, it may not always be relevant and have the same network/settings. I will however discuss this and we may add it as an enhancement in the future.

5. Understood. Indeed, it may be something additional to add as we currently have similar logic in Veeam Backup & Replication with failover plans. Thanks for the request!
Personal blog: https://foonet.be
GitHub: https://github.com/nielsengelen

vincenteberwein
Novice
Posts: 6
Liked: never
Joined: Nov 10, 2021 10:03 am
Full Name: Vincent Eberwein
Contact:

Re: AWS Instance Restore Features required

Post by vincenteberwein »

Hello,

we have also experienced that on an instance restore to the original location, the newly restored EC2 instance is not assigned the original IP address of the old EC2 instance. Instead, AWS assigns a new IP address within the subnet. We have also opened a support case (#05149445) for this.

How could you resolve the issue so that the original IP address is set for the restored EC2 instance when performing an instance restore?

Kind regards,
Vincent

nielsengelen
Veeam Software
Posts: 4657
Liked: 966 times
Joined: Jul 15, 2013 11:09 am
Full Name: Niels Engelen
Contact:

Re: AWS Instance Restore Features required

Post by nielsengelen »

Hi Vincent,

Please keep working with support as I believe this is a bug currently and it’s being looked into.
Personal blog: https://foonet.be
GitHub: https://github.com/nielsengelen

ScottChapman
Novice
Posts: 4
Liked: never
Joined: Oct 15, 2021 9:18 pm
Full Name: Scott Chapman
Contact:

Re: AWS Instance Restore Features required

Post by ScottChapman »

For the IP address not being restored, this is an issue for us primarily because we use /etc/hosts for name resolution. As the number of servers have grown that has started to get a little cumbersome, so we're in the process of moving to using Route 53 private zones for name resolution. This is not entirely trivial either because you still have to register your servers in those private zones. (Zones plural because for DR testing we restore to a separate VPC that we obviously want to not interfere with production.) And who wants to (re)do that for every DR test?

So I created a script to run on startup that will update the appropriate Route53 zone with the current host and IP address. In hope that it might help someone else, the pertinent parts of the script look something like this:

Code: Select all

TOKEN=`curl -s -X PUT "http://169.254.169.254/latest/api/token" -H "X-aws-ec2-metadata-token-ttl-seconds: 21600"`
thishost=`hostname`
ip=`curl -H "X-aws-ec2-metadata-token: $TOKEN" --no-progress-meter http://169.254.169.254/latest/meta-data/local-ipv4`
mac=`curl -H "X-aws-ec2-metadata-token: $TOKEN" --no-progress-meter http://169.254.169.254/latest/meta-data/network/interfaces/macs`
vpc=`curl -H "X-aws-ec2-metadata-token: $TOKEN" --no-progress-meter http://169.254.169.254/latest/meta-data/network/interfaces/macs/${mac}vpc-id`
credentials=`curl -H "X-aws-ec2-metadata-token: $TOKEN" --no-progress-meter http://169.254.169.254/latest/meta-data/iam/security-credentials/<yourInstancesIAMrole>`

export AWS_ACCESS_KEY_ID=`echo ${credentials} | jq -r .AccessKeyId`
export AWS_SECRET_ACCESS_KEY=`echo ${credentials} | jq -r .SecretAccessKey`
export AWS_SESSION_TOKEN=`echo ${credentials} | jq -r .Token`

# If we're in one of the production VPCs, we're in the "Prod" VPC, otherwise we're in "DR"
if [[ "${vpc}" == "vpc-xxxxxx" || "${vpc}" == "vpc-xxxxxx" ]]; then
    vpcmode="PROD"
zone="xxxxxx"
else
    vpcmode="DR"
zone="xxxxxx"
fi

echo '{"Comment":"Startup registration","Changes":[{"Action":"UPSERT","ResourceRecordSet":{"Name":"'${thishost}'.internal.eps","Type":"A","TTL":300,"ResourceRecords":[{"Value":"'${ip}'"}]}}]}' >/${HOME}/route53.json
aws route53 change-resource-record-sets --hosted-zone-id ${zone} --change-batch file:///${HOME}/route53.json --region us-east-1 --output json >/${HOME}/route53_out.json
On Ubuntu, /etc/systemd/resolved.conf is updated to include Domains=internal.eps so existing references to hostname will resolve to hostname.internal.eps which is the domain name for the Route53 private zones. (Note you don't need valid domain names for private zones.)

We don't have this quite rolled out yet, although hopefully it will be over the next few days, initial testing seems fine. But no warranty expressed or implied. Use at own risk, etc. etc.

Note that there's IAM permission work you'll need to do too to allow the instances to update the Route 53 private zones. Alas, while working on this, I discovered that Veeam is also not assigning the original IAM role to the instance when it's restored. I've been told that's now been added as a feature request as well. In the meantime, to deal with that we'll not automatically start the instances after restoration and manually add the IAM role to each instance before manually starting it. Which just slows down the restoration process and adds to the manual procedures required.

So after much work, procedural changes, configuration changes, and a new startup script, we may be ready to actually run a DR test in a week or so. Pretty far from the advertised "Achieve near-zero RTOs with fast wizard-driven recoveries." We knew there were deficiencies from our POC, but we missed at least the IAM issue during the POC and failed to think through the details of how problematic the issue with different IPs and lack of user data could be. At least at this point I believe / hope we've found all the gaps and Veeam is aware of them and has them at least documented as feature requests. But it does feel more like a version 0.x product than a version 4.0 product at this point.

nielsengelen
Veeam Software
Posts: 4657
Liked: 966 times
Joined: Jul 15, 2013 11:09 am
Full Name: Niels Engelen
Contact:

Re: AWS Instance Restore Features required

Post by nielsengelen »

Hi Scott,

We currently don't have integration with Route53 which may explain what you are seeing and need to adjust.

On the original AMI role not being assigned to the instance, we will look into this as it could be a bug as well.

Thanks for sharing the detailed explanation.
Personal blog: https://foonet.be
GitHub: https://github.com/nielsengelen

ScottChapman
Novice
Posts: 4
Liked: never
Joined: Oct 15, 2021 9:18 pm
Full Name: Scott Chapman
Contact:

Re: AWS Instance Restore Features required

Post by ScottChapman »

I wasn't suggesting that you should integrate with Route53. As I mentioned in the first line, we use /etc/hosts for name resolution of our servers, but Veeam restores servers to different IP addresses, which means that they can't talk to each other anymore until all the /etc/hosts on all the servers are updated with all the new IPs.

To work around that issue and because /etc/hosts is somewhat cumbersome for increasing numbers of servers, we're moving to using Route53 private zones for name resolution within the VPC. Which does not solve the issue with Veeam restoring to different IP addresses, but it would be a single place to update instead of updating multiple /etc/hosts files. But to better deal with that situation, I came up with the script that would update Route53 with the server's current IP address at startup. So that when Veeam restores a server to a different IP address, at least the server will update Route 53 with the new address and the servers will be able to talk to each other again.

nielsengelen
Veeam Software
Posts: 4657
Liked: 966 times
Joined: Jul 15, 2013 11:09 am
Full Name: Niels Engelen
Contact:

Re: AWS Instance Restore Features required

Post by nielsengelen »

The original IP should be preserved. I will definitely check up on this with our teams to assure this gets looked into.
Personal blog: https://foonet.be
GitHub: https://github.com/nielsengelen

nielsengelen
Veeam Software
Posts: 4657
Liked: 966 times
Joined: Jul 15, 2013 11:09 am
Full Name: Niels Engelen
Contact:

Re: AWS Instance Restore Features required

Post by nielsengelen »

@ScottChapman to clarify on the original IP, are you talking about the public or private IP? As we do have a hotfix currently related due to an issue with not preserving the private IP which can be obtained via support.
Personal blog: https://foonet.be
GitHub: https://github.com/nielsengelen

Post Reply

Who is online

Users browsing this forum: No registered users and 1 guest