Discussions related to using object storage as a backup target.
Post Reply
bwoodard
Novice
Posts: 7
Liked: 2 times
Joined: Jun 30, 2020 3:16 pm
Full Name: Brandon WOodard
Contact:

Bottleneck Restoring From Azure

Post by bwoodard »

Hello,

We are experiencing a performance bottleneck when restoring data from object storage using Veeam and could really use some assistance in identifying the bottleneck and/or better understanding the behavior.

Basically we are using Veeam to backup all of our production VMs and workloads to Azure using a scale-out repository. Our test case involves rescanning the object storage to import the data for a single smaller VM in order to evaluate the scenario of our entire on-premise infrastructure being lost (including the Veeam server). When we issue a rescan the data is imported fairly quickly, with a small stub file of 25MB being added locally for approximately 40GB of source data. The problem comes in when we attempt to restore the data to a Hyper-V server from object storage. The reflected rate in Veeam is only 9MB/s.

In a nutshell here's the data. If we restore from object storage to a local Hyper-V server using the process outlined above the job completes at a rate of 9MB/s (82Mb/s). The same restore of the same VM backup to the same Hyper-V host is approximately 8x that speed at around 70MB/s. When we installed Azure Storage Explorer to the Veeam server and downloaded the same data from object storage we would get transfer rates between 500 Mb/s and 700 Mb/s. We have 1Gb/s links throughout the environment where the backup solution is concerned. The object storage tier is 'Hot'.

We are desperately trying to understand why the restore operations from object storage take so long to complete. We have approximately 10TB of production VM data that we would need to restore from object storage in the event of a catastrophic failure. At a rate of 9MB/s this would give us an optimal restore window of around 13 days - far too long for the environment to be down. If we could get this closer to 500Mb/s (the rough low-end rate of our local restores, outgoing cloud backup jobs, and direct download of data from Azure) this restore window would go down to just under 2 days. We are using Veeam encryption.

Basically what we are wondering is whether or not this is expected behavior. If it is not what could we do to speed things up, or get more insight into what is causing the problem?
nielsengelen
Product Manager
Posts: 5619
Liked: 1177 times
Joined: Jul 15, 2013 11:09 am
Full Name: Niels Engelen
Contact:

Re: Bottleneck Restoring From Azure

Post by nielsengelen »

Hi Brandon, I moved your topic to the relevant section. Did you already have support help you with this and do some investigation on what is causing the difference in speed?
Personal blog: https://foonet.be
GitHub: https://github.com/nielsengelen
bwoodard
Novice
Posts: 7
Liked: 2 times
Joined: Jun 30, 2020 3:16 pm
Full Name: Brandon WOodard
Contact:

Re: Bottleneck Restoring From Azure

Post by bwoodard »

Hi Niels,

Thank you for moving my topic to the appropriate section. We haven't engaged support for this but if your recommendation is to open a ticket and then revisit here on the forums if we don't get an answer that is definitely something we can do. I guess my general question before engaging support was whether or not it is expected that the restore operations from object storage would be so much slower.
nielsengelen
Product Manager
Posts: 5619
Liked: 1177 times
Joined: Jul 15, 2013 11:09 am
Full Name: Niels Engelen
Contact:

Re: Bottleneck Restoring From Azure

Post by nielsengelen »

It shouldn't be the case and with the help of support, things could be cleared out. Maybe the logs are clarifying things already or something else is causing this as a bottleneck. Via the forums this is hard to troubleshoot hence the suggestion to contact support. If you have opened a case, could you let us know the case ID for future reference/discussions?
Personal blog: https://foonet.be
GitHub: https://github.com/nielsengelen
veremin
Product Manager
Posts: 20270
Liked: 2252 times
Joined: Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin
Contact:

Re: Bottleneck Restoring From Azure

Post by veremin »

Kindly, reach our support team and request 10a RTM update. It has a lot of fixes that increase restore to public cloud performance (especially for backups stored in object storage repositories). Thanks!
bwoodard
Novice
Posts: 7
Liked: 2 times
Joined: Jun 30, 2020 3:16 pm
Full Name: Brandon WOodard
Contact:

Re: Bottleneck Restoring From Azure

Post by bwoodard »

Thank you both!

We've opened a support case and uploaded logs - #04306001. We are running Veeam product version 10.0.0.4461. I'll inform them of what you advised so that we can implement the fix and update this thread when we have results from support.
veremin
Product Manager
Posts: 20270
Liked: 2252 times
Joined: Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin
Contact:

Re: Bottleneck Restoring From Azure

Post by veremin »

We made 10a RTM available on July, 23rd - build number is 10.0.1.4854.

So, it's definitely worth updating your installation and re-trying the restore operation.

Thanks!
bwoodard
Novice
Posts: 7
Liked: 2 times
Joined: Jun 30, 2020 3:16 pm
Full Name: Brandon WOodard
Contact:

Re: Bottleneck Restoring From Azure

Post by bwoodard » 2 people like this post

That fixed it! Up from 9 MB/s to 84 MB/s, a nearly 10x improvement that matches what we see for all our other cloud operations on the 1Gb link. Thank you very much!!!
veremin
Product Manager
Posts: 20270
Liked: 2252 times
Joined: Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin
Contact:

Re: Bottleneck Restoring From Azure

Post by veremin »

Thanks, Brandon, for confirmation - appreciated!
Post Reply

Who is online

Users browsing this forum: No registered users and 7 guests