Restore speed from Azure Cool

[frankive]

Hi.
On the object storage capacity repository we have defined a local server on-premise as gateway server. The server has 1Gb internet connection which we have verified with speed test and also is part of a S2d cluster with 10Gb link to the CSV-volumes.

While performing a testrestore from our Object Storage Cool in Azure (Norway East) we only see a performance of appx. 20-25 Mbit/s during restore.

Is this an expected speed? The backupfile we restored was 1 days old (copy tier mode).

[dalbertson]

There are a lot of factors at play here such as bandwidth and distance. Here are a couple of other forum posts about speed that you can review

object-storage-f52/restore-speed-from-s3-t64845.html

object-storage-f52/sofs-evacuate-backup ... ml#p312587

[mkh]

im seeing restore speed from azure storage account - GPv2 - cold at 120-160 Mbit/s - also norway east actually

If i use microsofts' storage explorer connected to the same storage account, it will upload and download the veeam 10 iso at around 900 Mbit/s (the line is a 1 Gbit/s fiber)
both in iso form, and in unpacked form it hits around 900 Mbit/s

Is there something i can tweak regarding how much is downloaded in parallel or something? i have tried changing under network traffic rules to use 20 streams, no difference

these restore speeds makes the storage account unusable for real life restore of large VMs

[veremin]

Do you have an Azure proxy deployed or not? Thanks!

[mkh]

I dont, the setup is purely on-premise except for the storage account.
I was under the impression that the Azure Proxy was used for restores TO Azure.
will it help with data from storage account to on-premise?

[mkh]

i have now tried in another installation where the veeam VM is placed in the same Azure region as the storage account, North Europe - GPv2 - Cold tier
the speed to restore from storage account is around the same 120-160 Mbit/s
Using Azure Storage Explorer on the veeam v10 iso it reached 1500 Mbit/s for upload and download

[veremin]

We will investigate the case internally and see what might be the bottleneck here. Thanks!

[mkh]

thank you very much

[veremin]

The case is being investigated currently by both QA and support teams. I will post back, once we have more information to share. Thanks!

[frankive]

any updates here?

[veremin]

Hi, Frank, can you refer to the issue 236702 to get the hotfix? We believe you might be affected by the similar issue, so, the hotfix should help. Thanks!

[frankive]

the fix seems to be realted to specific hardware Case # 04208359
we have this issue in multiple labs, including running in azure itself

[veremin]

Even though the issue was originally reported on specific storage system, we found out later, that it had bigger scale and created the corresponding hotfix. So, we do believe that applying the hot fix should solve your issue. Thanks!

[mkh]

the hotfix has in my first case resulted in the restore speed increasing to 45 MBps inside Veeam, roughly 200-300 Mbps on the netcard (varying alot during the transfer)
So a nice improvement
im still working with support, hoping for futher increase
If we could reach half line speed (400-500 Mbps), i would be really happy.

[veremin]

Thank you, Michael, that's exactly what we found in our tests - in cases when restore point was stored in object storage, the hot fix improved performance for all types of direct restore to public cloud. Thanks!

[frankive]

I will follow this thread and let @mkh do the the labwork....

[dariusz.tyka]

Even though the issue was originally reported on specific storage system, we found out later, that it had bigger scale and created the corresponding hotfix. So, we do believe that applying the hot fix should solve your issue. Thanks!

Is this hotfix specific to Azure object storage? Or it is suitable for AWS as well? I'm asking because restoration speed from AWS S3 (with no local data on backup server so all data is fetched from S3) is around 120-200 mbps (on 1Gbps link between AWS and on-prem) and maybe there is sth that can be improved.

[veremin]

Suitable for both. Thanks!

[akraker]

I opened a ticket for this and they cannot seem to be able to find the hotfix for me. Are you able to send me a link?

[Gostev]

No, as we don't have access to the support downloads system. Please just ask for your support case to be escalated to a manager, as I find it a bit silly that your engineer cannot find the hotfix that was previously provided to other customers. Be sure to reference issue ID 236702. Thanks!

[akraker]

I finally got the hotfix files. There is a Win and Lin folder. Assuming I just need the Win folder for Windows. Instructions I got are below.
However, the Win folder only contains a veamagent.exe file in a x64 folder and an x86 folder. Which files do I actually need to replace? I don't have a direct replacement for all of the files listed below.

Download and extract hotfix
2. Create backup of original files:
C:\Program Files (x86)\Veeam\Backup Transport\x64\VeeamAgent64.exe
C:\Program Files (x86)\Veeam\Backup Transport\x86\VeeamAgent.exe
C:\Program Files\Veeam\Backup and Replication\Backup\VeeamAgent64
C:\Program Files\Veeam\Backup and Replication\Backup\VeeamAgent
3. Replace these files with new files.

[akraker]

Nevermind, I think I got it. Restore speeds seem better but it looks like it is bursting. It jumps to around 300-400Mbps and down to 0 then back up.

[Gostev]

Please continue working directly with your support engineers for any queries in regards to the correct hotfix installation and its behavior in your environment. They will be able to see from the logs if you installed it properly, and if yes - what is the reason behind such a spiky behavior. Thanks!

[b.vanhaastrecht]

Hello,

I'm testing the VBR => Azure => Azure restore with an Azure Proxy. I like the feature, but the restore speeds are realy bad. Three parallel VM's bearely reaching 10MB/s each. Using two separate storage accounts, Storage V2, both in Western Europe. Source RA-GRS, destination LRS.

Is this the restore speed we could expect?

Kind regards,
Bastiaan

[veremin]

Can you request the said hotfix and see whether it solves the problem? Thanks!

[b.vanhaastrecht]

Ticket created: #04245356

[veremin]

Let's wait and see what support team say after log investigation. Chances are, you're also affected by the issue above and the hot fix will address it. Thanks!

[b.vanhaastrecht]

Applied the hotfix, no improvement of speed. Wil proceed with support to troubleshoot.

[veremin]

OK, we will follow the investigation and get involved, if case is escalated to R&D. Thanks!

[b.vanhaastrecht]

The fix does work for Azure to on-prem, we get 200MB/s+ . However restore from Azure to Azure doenst go passed 18MB/s. According to the engineer this is what his lab also shows as speed and what to expect. Is this really the case as this make it unusable, much to slow restore speeds. And it gets divided by each active restore session.