-
- Service Provider
- Posts: 1092
- Liked: 134 times
- Joined: May 14, 2013 8:35 pm
- Full Name: Frank Iversen
- Location: Norway
- Contact:
Restore speed from Azure Cool
Hi.
On the object storage capacity repository we have defined a local server on-premise as gateway server. The server has 1Gb internet connection which we have verified with speed test and also is part of a S2d cluster with 10Gb link to the CSV-volumes.
While performing a testrestore from our Object Storage Cool in Azure (Norway East) we only see a performance of appx. 20-25 Mbit/s during restore.
Is this an expected speed? The backupfile we restored was 1 days old (copy tier mode).
On the object storage capacity repository we have defined a local server on-premise as gateway server. The server has 1Gb internet connection which we have verified with speed test and also is part of a S2d cluster with 10Gb link to the CSV-volumes.
While performing a testrestore from our Object Storage Cool in Azure (Norway East) we only see a performance of appx. 20-25 Mbit/s during restore.
Is this an expected speed? The backupfile we restored was 1 days old (copy tier mode).
-
- Veeam Software
- Posts: 492
- Liked: 175 times
- Joined: Jul 21, 2015 12:38 pm
- Full Name: Dustin Albertson
- Contact:
Re: Restore speed from Azure Cool
There are a lot of factors at play here such as bandwidth and distance. Here are a couple of other forum posts about speed that you can review
object-storage-f52/restore-speed-from-s3-t64845.html
object-storage-f52/sofs-evacuate-backup ... ml#p312587
object-storage-f52/restore-speed-from-s3-t64845.html
object-storage-f52/sofs-evacuate-backup ... ml#p312587
Dustin Albertson | Director of Product Management - Cloud & Applications | Veeam Product Management, Alliances
-
- Service Provider
- Posts: 64
- Liked: 18 times
- Joined: Apr 20, 2018 6:17 am
- Full Name: Michael Høyer
- Contact:
Re: Restore speed from Azure Cool
im seeing restore speed from azure storage account - GPv2 - cold at 120-160 Mbit/s - also norway east actually
If i use microsofts' storage explorer connected to the same storage account, it will upload and download the veeam 10 iso at around 900 Mbit/s (the line is a 1 Gbit/s fiber)
both in iso form, and in unpacked form it hits around 900 Mbit/s
Is there something i can tweak regarding how much is downloaded in parallel or something? i have tried changing under network traffic rules to use 20 streams, no difference
these restore speeds makes the storage account unusable for real life restore of large VMs
If i use microsofts' storage explorer connected to the same storage account, it will upload and download the veeam 10 iso at around 900 Mbit/s (the line is a 1 Gbit/s fiber)
both in iso form, and in unpacked form it hits around 900 Mbit/s
Is there something i can tweak regarding how much is downloaded in parallel or something? i have tried changing under network traffic rules to use 20 streams, no difference
these restore speeds makes the storage account unusable for real life restore of large VMs
-
- Product Manager
- Posts: 20406
- Liked: 2298 times
- Joined: Oct 26, 2012 3:28 pm
- Full Name: Vladimir Eremin
- Contact:
Re: Restore speed from Azure Cool
Do you have an Azure proxy deployed or not? Thanks!
-
- Service Provider
- Posts: 64
- Liked: 18 times
- Joined: Apr 20, 2018 6:17 am
- Full Name: Michael Høyer
- Contact:
Re: Restore speed from Azure Cool
I dont, the setup is purely on-premise except for the storage account.
I was under the impression that the Azure Proxy was used for restores TO Azure.
will it help with data from storage account to on-premise?
I was under the impression that the Azure Proxy was used for restores TO Azure.
will it help with data from storage account to on-premise?
-
- Service Provider
- Posts: 64
- Liked: 18 times
- Joined: Apr 20, 2018 6:17 am
- Full Name: Michael Høyer
- Contact:
Re: Restore speed from Azure Cool
i have now tried in another installation where the veeam VM is placed in the same Azure region as the storage account, North Europe - GPv2 - Cold tier
the speed to restore from storage account is around the same 120-160 Mbit/s
Using Azure Storage Explorer on the veeam v10 iso it reached 1500 Mbit/s for upload and download
the speed to restore from storage account is around the same 120-160 Mbit/s
Using Azure Storage Explorer on the veeam v10 iso it reached 1500 Mbit/s for upload and download
-
- Product Manager
- Posts: 20406
- Liked: 2298 times
- Joined: Oct 26, 2012 3:28 pm
- Full Name: Vladimir Eremin
- Contact:
Re: Restore speed from Azure Cool
We will investigate the case internally and see what might be the bottleneck here. Thanks!
-
- Service Provider
- Posts: 64
- Liked: 18 times
- Joined: Apr 20, 2018 6:17 am
- Full Name: Michael Høyer
- Contact:
Re: Restore speed from Azure Cool
thank you very much
-
- Product Manager
- Posts: 20406
- Liked: 2298 times
- Joined: Oct 26, 2012 3:28 pm
- Full Name: Vladimir Eremin
- Contact:
Re: Restore speed from Azure Cool
The case is being investigated currently by both QA and support teams. I will post back, once we have more information to share. Thanks!
-
- Service Provider
- Posts: 1092
- Liked: 134 times
- Joined: May 14, 2013 8:35 pm
- Full Name: Frank Iversen
- Location: Norway
- Contact:
Re: Restore speed from Azure Cool
any updates here?
-
- Product Manager
- Posts: 20406
- Liked: 2298 times
- Joined: Oct 26, 2012 3:28 pm
- Full Name: Vladimir Eremin
- Contact:
Re: Restore speed from Azure Cool
Hi, Frank, can you refer to the issue 236702 to get the hotfix? We believe you might be affected by the similar issue, so, the hotfix should help. Thanks!
-
- Service Provider
- Posts: 1092
- Liked: 134 times
- Joined: May 14, 2013 8:35 pm
- Full Name: Frank Iversen
- Location: Norway
- Contact:
Re: Restore speed from Azure Cool
the fix seems to be realted to specific hardware Case # 04208359
we have this issue in multiple labs, including running in azure itself
we have this issue in multiple labs, including running in azure itself
-
- Product Manager
- Posts: 20406
- Liked: 2298 times
- Joined: Oct 26, 2012 3:28 pm
- Full Name: Vladimir Eremin
- Contact:
Re: Restore speed from Azure Cool
Even though the issue was originally reported on specific storage system, we found out later, that it had bigger scale and created the corresponding hotfix. So, we do believe that applying the hot fix should solve your issue. Thanks!
-
- Service Provider
- Posts: 64
- Liked: 18 times
- Joined: Apr 20, 2018 6:17 am
- Full Name: Michael Høyer
- Contact:
Re: Restore speed from Azure Cool
the hotfix has in my first case resulted in the restore speed increasing to 45 MBps inside Veeam, roughly 200-300 Mbps on the netcard (varying alot during the transfer)
So a nice improvement
im still working with support, hoping for futher increase
If we could reach half line speed (400-500 Mbps), i would be really happy.
So a nice improvement
im still working with support, hoping for futher increase
If we could reach half line speed (400-500 Mbps), i would be really happy.
-
- Product Manager
- Posts: 20406
- Liked: 2298 times
- Joined: Oct 26, 2012 3:28 pm
- Full Name: Vladimir Eremin
- Contact:
Re: Restore speed from Azure Cool
Thank you, Michael, that's exactly what we found in our tests - in cases when restore point was stored in object storage, the hot fix improved performance for all types of direct restore to public cloud. Thanks!
-
- Service Provider
- Posts: 1092
- Liked: 134 times
- Joined: May 14, 2013 8:35 pm
- Full Name: Frank Iversen
- Location: Norway
- Contact:
Re: Restore speed from Azure Cool
I will follow this thread and let @mkh do the the labwork....
-
- Enthusiast
- Posts: 57
- Liked: 4 times
- Joined: Jan 21, 2019 1:38 pm
- Full Name: Dariusz Tyka
- Contact:
Re: Restore speed from Azure Cool
Is this hotfix specific to Azure object storage? Or it is suitable for AWS as well? I'm asking because restoration speed from AWS S3 (with no local data on backup server so all data is fetched from S3) is around 120-200 mbps (on 1Gbps link between AWS and on-prem) and maybe there is sth that can be improved.
-
- Product Manager
- Posts: 20406
- Liked: 2298 times
- Joined: Oct 26, 2012 3:28 pm
- Full Name: Vladimir Eremin
- Contact:
Re: Restore speed from Azure Cool
Suitable for both. Thanks!
-
- Enthusiast
- Posts: 45
- Liked: 5 times
- Joined: Feb 11, 2019 6:19 pm
- Full Name: Andrew Kraker
- Contact:
Re: Restore speed from Azure Cool
I opened a ticket for this and they cannot seem to be able to find the hotfix for me. Are you able to send me a link?
-
- Chief Product Officer
- Posts: 31806
- Liked: 7300 times
- Joined: Jan 01, 2006 1:01 am
- Location: Baar, Switzerland
- Contact:
Re: Restore speed from Azure Cool
No, as we don't have access to the support downloads system. Please just ask for your support case to be escalated to a manager, as I find it a bit silly that your engineer cannot find the hotfix that was previously provided to other customers. Be sure to reference issue ID 236702. Thanks!
-
- Enthusiast
- Posts: 45
- Liked: 5 times
- Joined: Feb 11, 2019 6:19 pm
- Full Name: Andrew Kraker
- Contact:
Re: Restore speed from Azure Cool
I finally got the hotfix files. There is a Win and Lin folder. Assuming I just need the Win folder for Windows. Instructions I got are below.
However, the Win folder only contains a veamagent.exe file in a x64 folder and an x86 folder. Which files do I actually need to replace? I don't have a direct replacement for all of the files listed below.
Download and extract hotfix
2. Create backup of original files:
C:\Program Files (x86)\Veeam\Backup Transport\x64\VeeamAgent64.exe
C:\Program Files (x86)\Veeam\Backup Transport\x86\VeeamAgent.exe
C:\Program Files\Veeam\Backup and Replication\Backup\VeeamAgent64
C:\Program Files\Veeam\Backup and Replication\Backup\VeeamAgent
3. Replace these files with new files.
However, the Win folder only contains a veamagent.exe file in a x64 folder and an x86 folder. Which files do I actually need to replace? I don't have a direct replacement for all of the files listed below.
Download and extract hotfix
2. Create backup of original files:
C:\Program Files (x86)\Veeam\Backup Transport\x64\VeeamAgent64.exe
C:\Program Files (x86)\Veeam\Backup Transport\x86\VeeamAgent.exe
C:\Program Files\Veeam\Backup and Replication\Backup\VeeamAgent64
C:\Program Files\Veeam\Backup and Replication\Backup\VeeamAgent
3. Replace these files with new files.
-
- Enthusiast
- Posts: 45
- Liked: 5 times
- Joined: Feb 11, 2019 6:19 pm
- Full Name: Andrew Kraker
- Contact:
Re: Restore speed from Azure Cool
Nevermind, I think I got it. Restore speeds seem better but it looks like it is bursting. It jumps to around 300-400Mbps and down to 0 then back up.
-
- Chief Product Officer
- Posts: 31806
- Liked: 7300 times
- Joined: Jan 01, 2006 1:01 am
- Location: Baar, Switzerland
- Contact:
Re: Restore speed from Azure Cool
Please continue working directly with your support engineers for any queries in regards to the correct hotfix installation and its behavior in your environment. They will be able to see from the logs if you installed it properly, and if yes - what is the reason behind such a spiky behavior. Thanks!
-
- Service Provider
- Posts: 880
- Liked: 164 times
- Joined: Aug 26, 2013 7:46 am
- Full Name: Bastiaan van Haastrecht
- Location: The Netherlands
- Contact:
[MERGED] Azure to Azure restore speed
Hello,
I'm testing the VBR => Azure => Azure restore with an Azure Proxy. I like the feature, but the restore speeds are realy bad. Three parallel VM's bearely reaching 10MB/s each. Using two separate storage accounts, Storage V2, both in Western Europe. Source RA-GRS, destination LRS.
Is this the restore speed we could expect?
Kind regards,
Bastiaan
I'm testing the VBR => Azure => Azure restore with an Azure Proxy. I like the feature, but the restore speeds are realy bad. Three parallel VM's bearely reaching 10MB/s each. Using two separate storage accounts, Storage V2, both in Western Europe. Source RA-GRS, destination LRS.
Is this the restore speed we could expect?
Kind regards,
Bastiaan
======================================================
Veeam ProPartner, Service Provider and a proud Veeam Legend
Veeam ProPartner, Service Provider and a proud Veeam Legend
-
- Product Manager
- Posts: 20406
- Liked: 2298 times
- Joined: Oct 26, 2012 3:28 pm
- Full Name: Vladimir Eremin
- Contact:
Re: Restore speed from Azure Cool
Can you request the said hotfix and see whether it solves the problem? Thanks!
-
- Service Provider
- Posts: 880
- Liked: 164 times
- Joined: Aug 26, 2013 7:46 am
- Full Name: Bastiaan van Haastrecht
- Location: The Netherlands
- Contact:
Re: Restore speed from Azure Cool
Ticket created: #04245356
======================================================
Veeam ProPartner, Service Provider and a proud Veeam Legend
Veeam ProPartner, Service Provider and a proud Veeam Legend
-
- Product Manager
- Posts: 20406
- Liked: 2298 times
- Joined: Oct 26, 2012 3:28 pm
- Full Name: Vladimir Eremin
- Contact:
Re: Restore speed from Azure Cool
Let's wait and see what support team say after log investigation. Chances are, you're also affected by the issue above and the hot fix will address it. Thanks!
-
- Service Provider
- Posts: 880
- Liked: 164 times
- Joined: Aug 26, 2013 7:46 am
- Full Name: Bastiaan van Haastrecht
- Location: The Netherlands
- Contact:
Re: Restore speed from Azure Cool
Applied the hotfix, no improvement of speed. Wil proceed with support to troubleshoot.
======================================================
Veeam ProPartner, Service Provider and a proud Veeam Legend
Veeam ProPartner, Service Provider and a proud Veeam Legend
-
- Product Manager
- Posts: 20406
- Liked: 2298 times
- Joined: Oct 26, 2012 3:28 pm
- Full Name: Vladimir Eremin
- Contact:
Re: Restore speed from Azure Cool
OK, we will follow the investigation and get involved, if case is escalated to R&D. Thanks!
-
- Service Provider
- Posts: 880
- Liked: 164 times
- Joined: Aug 26, 2013 7:46 am
- Full Name: Bastiaan van Haastrecht
- Location: The Netherlands
- Contact:
Re: Restore speed from Azure Cool
The fix does work for Azure to on-prem, we get 200MB/s+ . However restore from Azure to Azure doenst go passed 18MB/s. According to the engineer this is what his lab also shows as speed and what to expect. Is this really the case as this make it unusable, much to slow restore speeds. And it gets divided by each active restore session.
======================================================
Veeam ProPartner, Service Provider and a proud Veeam Legend
Veeam ProPartner, Service Provider and a proud Veeam Legend
Who is online
Users browsing this forum: No registered users and 12 guests