Slow restore from S3 Object storage to EC2 VM

[djenningsNI]

Hello,
With the new Veeam 10 instant offloading to capacity tear storage (object storage) and the alure of the object locking in AWS we are testing backups, copied to AWS when performed, Object locked for 7 days and then if needed restored to a VM in EC2. So far so good until we tried to test the restore tonight.

Downloaded and installed Veeam community edition on a spare laptop
Connected to Object storage
Started VM restore to EC2 instance (from Object storage)
So the data is going from AWS S3 to AWS EC2 directly, confirmed no download/upload of data proxying through my machine

however my restore rate is averaging 5MB/sec and a 60GB machine ( test machine ) is estimating 4 hours for restore. Obviously this is not a workable solution for a DR restore even for production servers with say multiple TB of data. Is something wrong with this setup? Shouldnt the restore from AWS S3 to AWS EC2 being almost instant? Why is this so slow, is there anything I can fix here.

[nielsengelen]

Is this restore within the same region? May I ask in which region u did this test?

[djenningsNI]

Nielsengelen,
Of course, the S3 storage is in region "US West (Oregon)" and the restore was performed to region "us-west-2" which i believe according to research I did is indeed US West (Oregon). I was doing some research on the issue and found this article from AWS about speeding up S3 to EC2 transfers:

https://aws.amazon.com/premiumsupport/k ... -instance/

I intend to try this again after making the modification i am able to make.

Use enhanced networking on the EC2 instance:
I think this is something i can turn on, although it may be per instance and need to be controlled from the Veeam restore process.
Use parallel workloads for the data transfer
I think this is out of my control and should be something the Veeam software or Ubuntu image they use controls.
Customize the upload configurations on the AWS Command Line Interface (AWS CLI)
I think this is out of my control and should be something the Veeam software or Ubuntu image they use controls.
Use an Amazon Virtual Private Cloud (Amazon VPC) endpoint for Amazon S3
Not sure if this is something i can implement or Veeam would, need more research.
Use S3 Transfer Acceleration between geographically distant AWS Regions
Not relevant when data is in the same region.
Upgrade your EC2 instance type
I will try a more powerfull instance for the restore next time, however as the hold up was all from the restore to the Ubuntu image i dont think my selection will make a difference.
Use chunked transfers
I think this is out of my control and should be something the Veeam software or Ubuntu image they use controls.

As of my post last night the job was stuck at 39% in the phase of recovering the snapshot, as of this morning the job actually failed. I have attached some screenshots of the relevant details of the restore job.

https://imgur.com/a/E2H8fn4
https://imgur.com/a/xLvREOS
https://imgur.com/a/jGRp2y7

[dalbertson]

One other test would be to deploy a VBR server in EC2 and mount the object store and perform a direct restore from it to test the speed.

Sometimes it takes a while not so much from restore speeds but the import api that is being used to "convert" the instance.

[djenningsNI]

Is there any recommendations on spec's or instance type to use for a VMB server in EC2? Is there a prebuilt Image for this deployment or is the suggestion just to use a base windows server image and deploy Veeam B&R?

[dalbertson]

Currently there is not an AMI for VBR. For a test of this size i would just use a1.xlarge

[djenningsNI]

Dalbertson,
I performed additional restores today from a t3.large instance running Veeam B&R and found no change. Additional testing with Enhanced Networking support for the Veeam B&R Instance also produced no change. Below are my notes from the testing.


Created new "Microsoft Windows Server 2016 Base - ami-0bd8c602fafdca7b5" EC2 instance of the "t3.xlarge" Instance Type, a1.xlarge was not usable with a Windows X86/X64 operating system;

Connected to the new Instance via RDP and downloaded and installed Veeam B&R v10.0.0.4461, download ran at an average of 35MB/sec. Install of the software required expanding of the root disk, base is 30GB;

Added S3 Object Storage to the Veeam console. Imported backup jobs from the S3 storage, supplying encryption key of the backups. Backup chain listed in Backups > Disk (Encrypted). Supplied decryption key for the job "Test - MCAD04"; Backup chain now listed under "Object Storage (Imported)".

Started restore from backup to Amazon EC2 region US West (Oregon) (us-west-2), EC2 instance type t3.xlarge. Subnet 172.31.0.0/20 (us-west-2c).

Job results the same, average restore speed 6MB/s, left running for 2GB of the data and no change in the speed at any time. Canceled job.

Installed AWS Tools for Powershell and audited the t3.xlarge instance, I had created to run the Windows Veeam server, for enhanced networking configuration. Enabled enhanced networking configuration for the machine. Re-running restore procedure to see if this changes the speed.

Again restore job ran until 2GB of data was restored, still 6MB/s.

[djenningsNI]

One additional note that I would like Veeam confirmation on, it appears the Veeam B&R console is simply issuing commands to the Proxy Appliance via SSH. All of the Data moving seems to happen from the Proxy Appliance. If this is the case the performance bottleneck may be the Proxy Appliance itself. However during my last restore I did change the Proxy Appliance to a t3.xlarge as well and did not see any change.

[djenningsNI]

I have opened Veeam Support case #04032288 for this issue, so far Veeam is blaming AWS. I have now purchased a $100 business AWS support plan and am engaging AWS Support as well. I will get to the bottom of this at some point between these two.

[syscom]

Any news about your support request at Amazon?

We're having some of the same issues. We manage to reach a throughput of about 17 MB/s, it still takes 2 hours for a 70 GB virtual machine, not fabulous!

[djenningsNI]

Syscom, I got nothing but bad news for you on this one... AWS had the following response:


..............................................................................................................................................................................................................................................................................................................................
Traffic between Amazon EC2 and Amazon S3 can leverage up to 25 Gbps of bandwidth. The data transfer rate between an EC2 instance and an S3 bucket depend on several factors, including:
* The AWS Regions that the instance and the bucket are in.
* The medium of the data transfer, such as a transfer through the internet or a transfer through an Amazon Virtual Private Cloud (Amazon VPC) endpoint.
* The size of the objects that are being transferred.
* The resource utilization, CPU utilization, memory, disk I/O, network capabilities, and network limits of the EC2 instance.

The following methods are best practices for improving the transfer speed when you copy, move, or sync data between an EC2 instance and an S3 bucket. Please go through [1] & [2] for more details.
* Use enhanced networking on the EC2 instance.
* Use parallel workloads for the data transfer.
* Customize the upload configurations on the AWS Command Line Interface (AWS CLI).
* Use an Amazon Virtual Private Cloud (Amazon VPC) endpoint for Amazon S3.
* Use S3 Transfer Acceleration between geographically distant AWS Regions.
* Upgrade your EC2 instance type.
* Use chunked transfers.

Additionally you can test with AWS cli “aws s3 cp” , for example below test was run with S3 & EC instance (type c1.medium) both in us-west-2 Region in my test environment. Please go through [3] for more detail of additional steps than can be checked.

$ aws s3 cp s3://test1987testuswest2/file.txt .
Completed 955.5 MiB/1000.0 MiB (38.1 MiB/s) with 1 file(s) remaining

Consider using Amazon EBS–Optimized Instances [5] & Instances that support Enhanced Networking with the Elastic Network Adapter (ENA) [4] further based on you use case.
..............................................................................................................................................................................................................................................................................................................................

My case with Veeam support was unable to get past "Escalated" support because they informed me the product is working, there is no error and the restore speed is "within reason". What they did agree is that the Proxy Appliance may not be utilizing all "Paid" features it could be utilizing, like enhanced networking because not everyone wants to pay for it. My support case was turned into a feature request, i wont hold by breath on that, to allow those to be options during the restore. Ideally this would allow from the Veeam console the option to utilize these paid AWS options to speed up the recovery.

[veremin]

We will investigate the issue internally. I will update, as soon as I have more information (or need your assistance). Thanks!

[veremin]

So, QA team have investigated your case further (kudos to Rostislav Belostok @Belostok):

Let's discuss issues separately:

Slow performance issue

During restore we need to start two data movers. In your case they were started on a backup server: the traffic went from AWS to backup server, then, from the backup server back to AWS. This explains poor performance speed.

To avoid this you need to run proxy appliance in AWS. This will isolate traffic to cloud, which will result in improved performance rates

Failed to import machine to Amazon EC2 issue

The given error is generated whenever the native Amazon import VM procedure fails. Here we just take the error we get from Amazon and provide it to backup server user.

The reason why in your case Amazon Import VM procedure failed might be the fact that restored VM didn't meet the required pre-requisites mentioned here.

So, my further recommendations would be to verify whether the VM to restore meets the referenced requirements and, then, restore it, using proxy appliance.

Thanks!

[djenningsNI]

Veremin,
See my post from 2/12. Performance was still 6MB/sec when being run entirely in AWS EC2 using an XL device for the Windows machine and the appliance mover. So this unfortunately is not a solution. Also when I ran the VMWare console from my machine I did deploy an appliance to AWS which was the data mover, so the data did not come back to my machine and was isolated to the cloud. I confirmed this via network monitoring of my own machine and network.

Thanks,
Daryl J

[veremin]

Unfortunately, the logs attached to the support case show only the restore processes that were executed without proxy appliance, and without debug logs we cannot reliably identify the issue.

So, would you mind reproducing the issue with proxy appliance this time, collecting logs and creating a new ticket?

I will make sure that QA team work in close collaboration with support engineer and that main bottleneck is found.

Thanks!

[dmorenoimsa]

Hi,

My case is: #04092701.

I have this same issue, exactly same (Stuck at 39%, Slow Speed, Same Region S3 Object vs EC2 Region), and this ocurred since updated to V10.
1 - Slow restoration using Object Storage with Proxy Appliance with c4.x8large have a speed of restoration of 15MB.
2 - All my restoration process with Proxy Appliance shows same error Amazon EC2: Firstbootfailure.

But, if i restore without Proxy Appliance, the process complete correctly.

Were you able to solve this problem?

[djenningsNI]

dmorenoimsa,
Honestly I gave up on this for now myself. After putting 40+ hours into it and not getting a lot of help from Veeam support I decided to wait it out for a few months and get back to this later so I havent done any additional work on it.

Thanks,
Daryl J

[dmorenoimsa]

I see, in this case my teamwork is implementing this DRP Scenerario, before install V10 worked as expected, but with V10 everything went complicated.

As example, We have a server with 2TB size, the time to try to restore this server is almost 60 hours, just to read the error Firstbootfailure.

We cannot rollback version to V9.5 because we use the upload to s3 when a incremental is created.

Meanwhile we will work with this case to see if we can solve this or we found something i will update this post.

The first steps we will try this days is install the cumulative patch to see if this works ( https://www.veeam.com/kb3127 ).

Thanks!

[veremin]

My case is: #04092701.

I've asked QA team to follow the support investigation. I will write back, if we need more details or logs from your side. Thanks!

[veremin]

Due to their current business (working on a release for a different product), they will check the case in the beginning of the next week. So, for now, kindly, keep the ticket open. Thanks!

[djenningsNI]

Was there any additional work / information on this issue. We are using something else in the meantime but would love to come back to this if possible ever.

[sgarbe]

I have the same problem in my test environment. Backup runs great with max upload of the internet connection. The restore cloud to locally and directly in AWS runs at 15MB / s (bucket and restore in Europe / Frankfurt). The same test with Azure comes to 22MB / s. A test with AWS VTL Gateway shows a restore of 24MB / s. I wanted to offer AWS or Azure as a replacement for tape libraries. But here I have 250MB / s per drive. Maybe you could make a video, where you explain the configuration of the buckets step by step. When restoring in the cloud, i need a speed of 100MB / s. If such speeds are possible, please make a video

[veremin]

Was there any additional work / information on this issue. We are using something else in the meantime but would love to come back to this if possible ever.

May I ask you to request a hot fix for the issue 236702? QA team believe it should resolve the low restore speed problem. Thanks!

[veremin]

I have the same problem in my test environment. Backup runs great with max upload of the internet connection. The restore cloud to locally and directly in AWS runs at 15MB / s (bucket and restore in Europe / Frankfurt). The same test with Azure comes to 22MB / s. A test with AWS VTL Gateway shows a restore of 24MB / s. I wanted to offer AWS or Azure as a replacement for tape libraries. But here I have 250MB / s per drive. Maybe you could make a video, where you explain the configuration of the buckets step by step. When restoring in the cloud, i need a speed of 100MB / s. If such speeds are possible, please make a video

Hi, Stephan, unfortunately, we cannot help here without support case. Performance issues cannot be investigated via forum correspondence - deep log investigation and environment verification are needed. So, can you log a support ticket and provide its number here? Thanks!

[sgarbe]

#04208378

[veremin]

Thank you, we will follow the investigation. Thanks!