AWS Instance Restore Features required

[ScottChapman]

Today, restoring instances (specifically to a new account/region/AZ) requires more manual effort than it should, and requires (at least in our use case) manual effort after the instances are up to get their networking working.

1. There's no way to specify the IP address. Ideally this should be prepopulated with the IP address of the original server.
2. The user data from the original instance is not used on the startup of the new instance. In our case (on Ubuntu 20.04) that means cloud-init notices that it's on a new instance and the hostname gets re-written to a default using the IP address (which due to #1 is some new address). If the original user data was passed through, then at least cloud-init would assign the correct hostname.
3. Ideally, once you've selected the account and region, you'd go out and find the VPCs and subnets and automatically pre-select the same-named VPCs/subnets, with of course an option to override them as need be.
4. When doing tests, it's tedious to go through the process for every server. Ideally we'd select a number of instances (presumably from the same subnet, or even better let us select all the instances in a subnet with a single click) and then select parameters for all of those servers: i.e. we'd like them to go back to the same-named VPC/subnet/security group/etc. Allowing overrides of course, but if we need to move everything from one AZ to another, we generally would want it to all come back up the same way it was.
5. Ideally we'd be able to save our recovery plan for later execution. Ideally that plan would be built generically not instance by instance. I.E. restore all instances from this subnet or with a particular tag, so that when we go to run that recovery plan, we don't have to update it for changes in the instance inventory. I would envision this ideally being a mapping from original to new VPC/subnet/etc. names that is then applied to all instances selected for restoration.

#1 and #2 are already done by at least one of your major competitors and seems to be a very basic requirement that is hard to understand why they're not already there. How do people work around their instances coming up with different IP addresses and different host names?

#3, #4, and #5 are at least partially done by competitors as well. Those also would be easier with VPC recovery which still isn't working for us on v4. (Yes we have a problem open for that.) But at least we can manually create the VPC structures needed.

The goal would be able to quickly and easily restore from one account, region, and/or az to another, which at the moment it's certainly not quick and easy.

We liked the design and usability of Veeam for AWS better than the competitors, but the recovery issues almost pushed us to a competing product. The closest competing product had it's own (less significant) recovery issues, but if it would have been a little bit better we probably would have chosen it. The saving grace at the moment is that we hope to not have to restore on a regular basis, although we do have a goal of performing regular DR tests.

[nielsengelen]

1. We preserve the IP when u restore to the original location. To another location, this should also work if the same range is available there so it looks to me this is a networking issue within the VPCs but that is hard to tell without insight.
2. We don't adjust anything in-guest so this should work if everything is set up correct regarding networking / IP passing out etc.
3. If u restore to the original location, we take the original available one. If you want to change them, then select restore to another location but use the same region. This is by design possible already. We don't want customers to waste time in case of a quick restore to the original location with all these settings which we already know.
4. This is possible by multi selecting the instances in the restore wizard (same applies here as my answer to question 3).
5. Am I correct that you are looking to have a sort of "restore templates" in which you add a list of instances and you could "failover" with the click of 1 button (sort of speak)?

Feel free to send me a direct message to dig deeper in these points as I may have misunderstood things.

[ScottChapman]

1. I have not seen this work yet.
2. This is specifically related to how cloud-init works and the lack of the "User Data" as was specified for the original launch of the AMI.
3. You present the VPCs/subnets for selectin but the user has to select them. If a VPC/subnet exists with the same name, it seems like you could select that by default already.
4. Somehow I missed you could select multiple at the same time. Excellent!
5. Yes, the idea is to get to easy button to do a restore. Presumably with some display of "this is what we're about to do".

I'll DM you about points 1 and 2 in particular.

[nielsengelen]

3. I understand your point of view there and while it may make sense to present the one with the same name, it may not always be relevant and have the same network/settings. I will however discuss this and we may add it as an enhancement in the future.

5. Understood. Indeed, it may be something additional to add as we currently have similar logic in Veeam Backup & Replication with failover plans. Thanks for the request!

[vincenteberwein]

Hello,

we have also experienced that on an instance restore to the original location, the newly restored EC2 instance is not assigned the original IP address of the old EC2 instance. Instead, AWS assigns a new IP address within the subnet. We have also opened a support case (#05149445) for this.

How could you resolve the issue so that the original IP address is set for the restored EC2 instance when performing an instance restore?

Kind regards,
Vincent

[nielsengelen]

Hi Vincent,

Please keep working with support as I believe this is a bug currently and it’s being looked into.

[ScottChapman]

For the IP address not being restored, this is an issue for us primarily because we use /etc/hosts for name resolution. As the number of servers have grown that has started to get a little cumbersome, so we're in the process of moving to using Route 53 private zones for name resolution. This is not entirely trivial either because you still have to register your servers in those private zones. (Zones plural because for DR testing we restore to a separate VPC that we obviously want to not interfere with production.) And who wants to (re)do that for every DR test?

So I created a script to run on startup that will update the appropriate Route53 zone with the current host and IP address. In hope that it might help someone else, the pertinent parts of the script look something like this:

Code: Select all

TOKEN=`curl -s -X PUT "http://169.254.169.254/latest/api/token" -H "X-aws-ec2-metadata-token-ttl-seconds: 21600"`
thishost=`hostname`
ip=`curl -H "X-aws-ec2-metadata-token: $TOKEN" --no-progress-meter http://169.254.169.254/latest/meta-data/local-ipv4`
mac=`curl -H "X-aws-ec2-metadata-token: $TOKEN" --no-progress-meter http://169.254.169.254/latest/meta-data/network/interfaces/macs`
vpc=`curl -H "X-aws-ec2-metadata-token: $TOKEN" --no-progress-meter http://169.254.169.254/latest/meta-data/network/interfaces/macs/${mac}vpc-id`
credentials=`curl -H "X-aws-ec2-metadata-token: $TOKEN" --no-progress-meter http://169.254.169.254/latest/meta-data/iam/security-credentials/<yourInstancesIAMrole>`

export AWS_ACCESS_KEY_ID=`echo ${credentials} | jq -r .AccessKeyId`
export AWS_SECRET_ACCESS_KEY=`echo ${credentials} | jq -r .SecretAccessKey`
export AWS_SESSION_TOKEN=`echo ${credentials} | jq -r .Token`

# If we're in one of the production VPCs, we're in the "Prod" VPC, otherwise we're in "DR"
if [[ "${vpc}" == "vpc-xxxxxx" || "${vpc}" == "vpc-xxxxxx" ]]; then
    vpcmode="PROD"
zone="xxxxxx"
else
    vpcmode="DR"
zone="xxxxxx"
fi

echo '{"Comment":"Startup registration","Changes":[{"Action":"UPSERT","ResourceRecordSet":{"Name":"'${thishost}'.internal.eps","Type":"A","TTL":300,"ResourceRecords":[{"Value":"'${ip}'"}]}}]}' >/${HOME}/route53.json
aws route53 change-resource-record-sets --hosted-zone-id ${zone} --change-batch file:///${HOME}/route53.json --region us-east-1 --output json >/${HOME}/route53_out.json

On Ubuntu, /etc/systemd/resolved.conf is updated to include Domains=internal.eps so existing references to hostname will resolve to hostname.internal.eps which is the domain name for the Route53 private zones. (Note you don't need valid domain names for private zones.)

We don't have this quite rolled out yet, although hopefully it will be over the next few days, initial testing seems fine. But no warranty expressed or implied. Use at own risk, etc. etc.

Note that there's IAM permission work you'll need to do too to allow the instances to update the Route 53 private zones. Alas, while working on this, I discovered that Veeam is also not assigning the original IAM role to the instance when it's restored. I've been told that's now been added as a feature request as well. In the meantime, to deal with that we'll not automatically start the instances after restoration and manually add the IAM role to each instance before manually starting it. Which just slows down the restoration process and adds to the manual procedures required.

So after much work, procedural changes, configuration changes, and a new startup script, we may be ready to actually run a DR test in a week or so. Pretty far from the advertised "Achieve near-zero RTOs with fast wizard-driven recoveries." We knew there were deficiencies from our POC, but we missed at least the IAM issue during the POC and failed to think through the details of how problematic the issue with different IPs and lack of user data could be. At least at this point I believe / hope we've found all the gaps and Veeam is aware of them and has them at least documented as feature requests. But it does feel more like a version 0.x product than a version 4.0 product at this point.

[nielsengelen]

Hi Scott,

We currently don't have integration with Route53 which may explain what you are seeing and need to adjust.

On the original AMI role not being assigned to the instance, we will look into this as it could be a bug as well.

Thanks for sharing the detailed explanation.

[ScottChapman]

I wasn't suggesting that you should integrate with Route53. As I mentioned in the first line, we use /etc/hosts for name resolution of our servers, but Veeam restores servers to different IP addresses, which means that they can't talk to each other anymore until all the /etc/hosts on all the servers are updated with all the new IPs.

To work around that issue and because /etc/hosts is somewhat cumbersome for increasing numbers of servers, we're moving to using Route53 private zones for name resolution within the VPC. Which does not solve the issue with Veeam restoring to different IP addresses, but it would be a single place to update instead of updating multiple /etc/hosts files. But to better deal with that situation, I came up with the script that would update Route53 with the server's current IP address at startup. So that when Veeam restores a server to a different IP address, at least the server will update Route 53 with the new address and the servers will be able to talk to each other again.

[nielsengelen]

The original IP should be preserved. I will definitely check up on this with our teams to assure this gets looked into.

[nielsengelen]

@ScottChapman to clarify on the original IP, are you talking about the public or private IP? As we do have a hotfix currently related due to an issue with not preserving the private IP which can be obtained via support.

[vincenteberwein]

Hello Niels,

I got the hotfix provided by support 5 days ago (support case ID: 05149445), thanks for that. I installed it on our Veeam Backup for AWS appliance, unfortunately it works only partially.

I have tested the hotfix extensively and found the following behavior:

When I select "Restore to original location" in the "Restore Mode" step of the instance restore, the restored EC2 instance is now assigned the private IP address of the old EC2 instance, but the restored EC2 instance is also assigned a public IP address, even though the old EC2 instance was not. Here we have the requirement that no public IP address is assigned to the restored EC2 instance, only a private IP address. We also have the option disabled in our subnets in AWS to automatically assign a public IPv4 address.

If I select "Restore to new location, or with different settings" in the "Restore Mode" step and restore the EC2 instance to the original location, I can set the option in the "Network" step to not assign a public IP address to the restored EC2 instance. However, if I then execute the instance restore via this variant, the restored EC2 instance is no longer assigned a public IP address, but it is again assigned a randomly generated private IP address and not the private IP address of the old EC2 instance.

This is currently a really important feature that we are missing in Veeam Backup for AWS. I have also addressed this problem in this detail to the Veeam support, but unfortunately, I have not yet received a satisfactory solution.

I would ask you to fix this issue in a timely manner, as this broken functionality causes us a lot of extra work in case of an instance restore.

Thank you very much.

Kind regards,
Vincent

[nielsengelen]

Hi Vincent,

Thank you for the detailed info. I will do some checking on what could be wrong here and once I know more, I will post it here.

[nielsengelen]

Hi @vincenteberwein,

We have identified the behavior (always assign a public IP) as a bug and will resolve this in an upcoming update.

[sinoptik777]

Today, restoring instances (specifically to a new account/region/AZ) requires more manual effort than it should, and requires (at least in our use case) manual effort after the instances are up to get their networking working.

1. There's no way to specify the IP address. Ideally this should be prepopulated with the IP address of the original server.
2. The user data from the original instance is not used on the startup of the new instance. In our case (on Ubuntu 20.04) that means cloud-init notices that it's on a new instance and the hostname gets re-written to a default using the IP address (which due to #1 is some new address). If the original user data was passed through, then at least cloud-init would assign the correct hostname.
3. Ideally, once you've selected the account and region, you'd go out and find the VPCs and subnets and automatically pre-select the same-named VPCs/subnets, with of course an option to override them as need be.
4. When doing tests, it's tedious to go through the process for every server. Ideally we'd select a number of instances (presumably from the same subnet, or even better let us select all the instances in a subnet with a single click) and then select parameters for all of those servers: i.e. we'd like them to go back to the same-named VPC/subnet/security group/etc. Allowing overrides of course, but if we need to move everything from one AZ to another, we generally would want it to all come back up the same way it was.
5. Ideally we'd be able to save our recovery plan for later execution. Ideally that plan would be built generically not instance by instance. I.E. restore all instances from this subnet or with a particular tag, so that when we go to run that recovery plan, we don't have to update it for changes in the instance inventory. I would envision this ideally being a mapping from original to new VPC/subnet/etc. names that is then applied to all instances selected for restoration.

#1 and #2 are already done by at least one of your major competitors and seems to be a very basic requirement that is hard to understand why they're not already there. How do people work around their instances coming up with different IP addresses and different host names?

#3, #4, and #5 are at least partially done by competitors as well. Those also would be easier with VPC recovery which still isn't working for us on v4. (Yes we have a problem open for that.) But at least we can manually create the VPC structures needed.

The goal would be able to quickly and easily restore from one account, region, and/or az to another, which at the moment it's certainly not quick and easy.

We liked the design and usability of Veeam for AWS better than the competitors, but the recovery issues almost pushed us to a competing product. The closest competing product had it's own (less significant) recovery issues, but if it would have been a little bit better we probably would have chosen it. The saving grace at the moment is that we hope to not have to restore on a regular basis, although we do have a goal of performing regular DR tests.

#1 and #2 are very important. Still waiting these feachers

[nielsengelen]

Point 1 is on our roadmap for a future release but this will only work if you use Elastic IPs. If they are dynamically allocated, we cannot preserve them.

Point 2 is a different story as it's unique per case. User data is by default used only once when you launch an instance. It can be altered to always use it. If this is the case for you, it would be good to get more details about the instance setup so we can look into it. You can send these via a direct message here. We don't want to always assign it and potentially do more harm than needed (if it's really a "run only once" type of user data).

[sinoptik777]

Point 1. But what about Private IP, not Public...Elastic IPs use for Public...

[nielsengelen]

Private IPs was already resolved with v4a and is there in v5. If the private IP is free, it should be re-assigned and used.

If this isn’t happening either the IP is still locked (and we warn about this during the restore wizard) or something else is going on in which case, a support case would be needed to analyse logs. If you do open a case and are seeing other things, please let us know the case ID.

[sinoptik777]

Okey...thanks...I will open a case because it is not work. I have last one aws plugin and Veeam backup and restore.

I created EC2 instance and made backup to s3. Then I deleted this instance and restored from backup to original place. But instance got different private IP.
I tried restore to original place in aws but instance got a different private IP address.

[nielsengelen]

The only potential issue I can think of is if you have multiple IPs as that currently isn't supported and targeted to be resolved in a nearby update.

Could you let us know the case ID once opened?

[sinoptik777]

No, One IP
785118633049

The problem is only when I restore from the console veeam R&B 11. If I do so, then another private ip will be assigned to the restored machine. If I do a restore through the browser (veeam backup for aws v5), then the restore occurs with the correct private IP .

[nielsengelen]

Within VBR this is currently not possible. It wasn't clear from your initial post that you were talking about VBR and I thought this was about VBA.

I will pass this on to the VBR team as an enhancement however I have no ETA when this will be resolved as of now. Once I have more details, I will share them.

[sinoptik777]

Thanks!

[sinoptik777]

I have one more question )
About ONEZONE_IA S3 and Veeam aws backup.
Why don't you make it possible to choose when configuring the backup that the storage policy is ONEZONE_IA in S3.
It would save money and ONEZONE_IA is just right for backups

[nielsengelen]

We recommend leveraging standard S3 for short-term backups and Glacier for long-term backups as it's more cost-effective. Restore costs from IA would be higher hence why we currently don't recommend/support it.

[sinoptik777]

Ok, I understood. Thanks