Discussions related to using object storage as a backup target.
frankive
Service Provider
Posts: 1092
Liked: 134 times
Joined: May 14, 2013 8:35 pm
Full Name: Frank Iversen
Location: Norway
Contact:

Restore speed from Azure Cool

Post by frankive »

Hi.
On the object storage capacity repository we have defined a local server on-premise as gateway server. The server has 1Gb internet connection which we have verified with speed test and also is part of a S2d cluster with 10Gb link to the CSV-volumes.

While performing a testrestore from our Object Storage Cool in Azure (Norway East) we only see a performance of appx. 20-25 Mbit/s during restore.

Is this an expected speed? The backupfile we restored was 1 days old (copy tier mode).
dalbertson
Veeam Software
Posts: 492
Liked: 175 times
Joined: Jul 21, 2015 12:38 pm
Full Name: Dustin Albertson
Contact:

Re: Restore speed from Azure Cool

Post by dalbertson » 1 person likes this post

There are a lot of factors at play here such as bandwidth and distance. Here are a couple of other forum posts about speed that you can review

object-storage-f52/restore-speed-from-s3-t64845.html

object-storage-f52/sofs-evacuate-backup ... ml#p312587
Dustin Albertson | Director of Product Management - Cloud & Applications | Veeam Product Management, Alliances
mkh
Service Provider
Posts: 64
Liked: 18 times
Joined: Apr 20, 2018 6:17 am
Full Name: Michael Høyer
Contact:

Re: Restore speed from Azure Cool

Post by mkh »

im seeing restore speed from azure storage account - GPv2 - cold at 120-160 Mbit/s - also norway east actually

If i use microsofts' storage explorer connected to the same storage account, it will upload and download the veeam 10 iso at around 900 Mbit/s (the line is a 1 Gbit/s fiber)
both in iso form, and in unpacked form it hits around 900 Mbit/s

Is there something i can tweak regarding how much is downloaded in parallel or something? i have tried changing under network traffic rules to use 20 streams, no difference

these restore speeds makes the storage account unusable for real life restore of large VMs
veremin
Product Manager
Posts: 20270
Liked: 2252 times
Joined: Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin
Contact:

Re: Restore speed from Azure Cool

Post by veremin »

Do you have an Azure proxy deployed or not? Thanks!
mkh
Service Provider
Posts: 64
Liked: 18 times
Joined: Apr 20, 2018 6:17 am
Full Name: Michael Høyer
Contact:

Re: Restore speed from Azure Cool

Post by mkh »

I dont, the setup is purely on-premise except for the storage account.
I was under the impression that the Azure Proxy was used for restores TO Azure.
will it help with data from storage account to on-premise?
mkh
Service Provider
Posts: 64
Liked: 18 times
Joined: Apr 20, 2018 6:17 am
Full Name: Michael Høyer
Contact:

Re: Restore speed from Azure Cool

Post by mkh »

i have now tried in another installation where the veeam VM is placed in the same Azure region as the storage account, North Europe - GPv2 - Cold tier
the speed to restore from storage account is around the same 120-160 Mbit/s
Using Azure Storage Explorer on the veeam v10 iso it reached 1500 Mbit/s for upload and download
veremin
Product Manager
Posts: 20270
Liked: 2252 times
Joined: Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin
Contact:

Re: Restore speed from Azure Cool

Post by veremin » 1 person likes this post

We will investigate the case internally and see what might be the bottleneck here. Thanks!
mkh
Service Provider
Posts: 64
Liked: 18 times
Joined: Apr 20, 2018 6:17 am
Full Name: Michael Høyer
Contact:

Re: Restore speed from Azure Cool

Post by mkh »

thank you very much :)
veremin
Product Manager
Posts: 20270
Liked: 2252 times
Joined: Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin
Contact:

Re: Restore speed from Azure Cool

Post by veremin »

The case is being investigated currently by both QA and support teams. I will post back, once we have more information to share. Thanks!
frankive
Service Provider
Posts: 1092
Liked: 134 times
Joined: May 14, 2013 8:35 pm
Full Name: Frank Iversen
Location: Norway
Contact:

Re: Restore speed from Azure Cool

Post by frankive »

any updates here?
veremin
Product Manager
Posts: 20270
Liked: 2252 times
Joined: Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin
Contact:

Re: Restore speed from Azure Cool

Post by veremin »

Hi, Frank, can you refer to the issue 236702 to get the hotfix? We believe you might be affected by the similar issue, so, the hotfix should help. Thanks!
frankive
Service Provider
Posts: 1092
Liked: 134 times
Joined: May 14, 2013 8:35 pm
Full Name: Frank Iversen
Location: Norway
Contact:

Re: Restore speed from Azure Cool

Post by frankive »

the fix seems to be realted to specific hardware Case # 04208359
we have this issue in multiple labs, including running in azure itself
veremin
Product Manager
Posts: 20270
Liked: 2252 times
Joined: Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin
Contact:

Re: Restore speed from Azure Cool

Post by veremin »

Even though the issue was originally reported on specific storage system, we found out later, that it had bigger scale and created the corresponding hotfix. So, we do believe that applying the hot fix should solve your issue. Thanks!
mkh
Service Provider
Posts: 64
Liked: 18 times
Joined: Apr 20, 2018 6:17 am
Full Name: Michael Høyer
Contact:

Re: Restore speed from Azure Cool

Post by mkh » 2 people like this post

the hotfix has in my first case resulted in the restore speed increasing to 45 MBps inside Veeam, roughly 200-300 Mbps on the netcard (varying alot during the transfer)
So a nice improvement
im still working with support, hoping for futher increase :)
If we could reach half line speed (400-500 Mbps), i would be really happy.
veremin
Product Manager
Posts: 20270
Liked: 2252 times
Joined: Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin
Contact:

Re: Restore speed from Azure Cool

Post by veremin »

Thank you, Michael, that's exactly what we found in our tests - in cases when restore point was stored in object storage, the hot fix improved performance for all types of direct restore to public cloud. Thanks!
frankive
Service Provider
Posts: 1092
Liked: 134 times
Joined: May 14, 2013 8:35 pm
Full Name: Frank Iversen
Location: Norway
Contact:

Re: Restore speed from Azure Cool

Post by frankive »

I will follow this thread and let @mkh do the the labwork.... :) :) :)
dariusz.tyka
Enthusiast
Posts: 54
Liked: 4 times
Joined: Jan 21, 2019 1:38 pm
Full Name: Dariusz Tyka
Contact:

Re: Restore speed from Azure Cool

Post by dariusz.tyka »

veremin wrote: Jun 03, 2020 7:13 pm Even though the issue was originally reported on specific storage system, we found out later, that it had bigger scale and created the corresponding hotfix. So, we do believe that applying the hot fix should solve your issue. Thanks!
Is this hotfix specific to Azure object storage? Or it is suitable for AWS as well? I'm asking because restoration speed from AWS S3 (with no local data on backup server so all data is fetched from S3) is around 120-200 mbps (on 1Gbps link between AWS and on-prem) and maybe there is sth that can be improved.
veremin
Product Manager
Posts: 20270
Liked: 2252 times
Joined: Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin
Contact:

Re: Restore speed from Azure Cool

Post by veremin »

Suitable for both. Thanks!
akraker
Enthusiast
Posts: 45
Liked: 5 times
Joined: Feb 11, 2019 6:19 pm
Full Name: Andrew Kraker
Contact:

Re: Restore speed from Azure Cool

Post by akraker »

I opened a ticket for this and they cannot seem to be able to find the hotfix for me. Are you able to send me a link?
Gostev
Chief Product Officer
Posts: 31457
Liked: 6647 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Restore speed from Azure Cool

Post by Gostev »

No, as we don't have access to the support downloads system. Please just ask for your support case to be escalated to a manager, as I find it a bit silly that your engineer cannot find the hotfix that was previously provided to other customers. Be sure to reference issue ID 236702. Thanks!
akraker
Enthusiast
Posts: 45
Liked: 5 times
Joined: Feb 11, 2019 6:19 pm
Full Name: Andrew Kraker
Contact:

Re: Restore speed from Azure Cool

Post by akraker »

I finally got the hotfix files. There is a Win and Lin folder. Assuming I just need the Win folder for Windows. Instructions I got are below.
However, the Win folder only contains a veamagent.exe file in a x64 folder and an x86 folder. Which files do I actually need to replace? I don't have a direct replacement for all of the files listed below.

Download and extract hotfix
2. Create backup of original files:
C:\Program Files (x86)\Veeam\Backup Transport\x64\VeeamAgent64.exe
C:\Program Files (x86)\Veeam\Backup Transport\x86\VeeamAgent.exe
C:\Program Files\Veeam\Backup and Replication\Backup\VeeamAgent64
C:\Program Files\Veeam\Backup and Replication\Backup\VeeamAgent
3. Replace these files with new files.
akraker
Enthusiast
Posts: 45
Liked: 5 times
Joined: Feb 11, 2019 6:19 pm
Full Name: Andrew Kraker
Contact:

Re: Restore speed from Azure Cool

Post by akraker » 1 person likes this post

Nevermind, I think I got it. Restore speeds seem better but it looks like it is bursting. It jumps to around 300-400Mbps and down to 0 then back up.
Gostev
Chief Product Officer
Posts: 31457
Liked: 6647 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Restore speed from Azure Cool

Post by Gostev »

Please continue working directly with your support engineers for any queries in regards to the correct hotfix installation and its behavior in your environment. They will be able to see from the logs if you installed it properly, and if yes - what is the reason behind such a spiky behavior. Thanks!
b.vanhaastrecht
Service Provider
Posts: 833
Liked: 154 times
Joined: Aug 26, 2013 7:46 am
Full Name: Bastiaan van Haastrecht
Location: The Netherlands
Contact:

[MERGED] Azure to Azure restore speed

Post by b.vanhaastrecht »

Hello,

I'm testing the VBR => Azure => Azure restore with an Azure Proxy. I like the feature, but the restore speeds are realy bad. Three parallel VM's bearely reaching 10MB/s each. Using two separate storage accounts, Storage V2, both in Western Europe. Source RA-GRS, destination LRS.

Is this the restore speed we could expect?

Kind regards,
Bastiaan
======================================================
Veeam ProPartner, Service Provider and a proud Veeam Legend
veremin
Product Manager
Posts: 20270
Liked: 2252 times
Joined: Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin
Contact:

Re: Restore speed from Azure Cool

Post by veremin »

Can you request the said hotfix and see whether it solves the problem? Thanks!
b.vanhaastrecht
Service Provider
Posts: 833
Liked: 154 times
Joined: Aug 26, 2013 7:46 am
Full Name: Bastiaan van Haastrecht
Location: The Netherlands
Contact:

Re: Restore speed from Azure Cool

Post by b.vanhaastrecht » 1 person likes this post

Ticket created: #04245356
======================================================
Veeam ProPartner, Service Provider and a proud Veeam Legend
veremin
Product Manager
Posts: 20270
Liked: 2252 times
Joined: Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin
Contact:

Re: Restore speed from Azure Cool

Post by veremin »

Let's wait and see what support team say after log investigation. Chances are, you're also affected by the issue above and the hot fix will address it. Thanks!
b.vanhaastrecht
Service Provider
Posts: 833
Liked: 154 times
Joined: Aug 26, 2013 7:46 am
Full Name: Bastiaan van Haastrecht
Location: The Netherlands
Contact:

Re: Restore speed from Azure Cool

Post by b.vanhaastrecht »

Applied the hotfix, no improvement of speed. Wil proceed with support to troubleshoot.
======================================================
Veeam ProPartner, Service Provider and a proud Veeam Legend
veremin
Product Manager
Posts: 20270
Liked: 2252 times
Joined: Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin
Contact:

Re: Restore speed from Azure Cool

Post by veremin »

OK, we will follow the investigation and get involved, if case is escalated to R&D. Thanks!
b.vanhaastrecht
Service Provider
Posts: 833
Liked: 154 times
Joined: Aug 26, 2013 7:46 am
Full Name: Bastiaan van Haastrecht
Location: The Netherlands
Contact:

Re: Restore speed from Azure Cool

Post by b.vanhaastrecht »

The fix does work for Azure to on-prem, we get 200MB/s+ . However restore from Azure to Azure doenst go passed 18MB/s. According to the engineer this is what his lab also shows as speed and what to expect. Is this really the case as this make it unusable, much to slow restore speeds. And it gets divided by each active restore session.
======================================================
Veeam ProPartner, Service Provider and a proud Veeam Legend
Post Reply

Who is online

Users browsing this forum: aurelien-beneteau and 12 guests