Bottleneck Restoring From Azure

[bwoodard]

Hello,

We are experiencing a performance bottleneck when restoring data from object storage using Veeam and could really use some assistance in identifying the bottleneck and/or better understanding the behavior.

Basically we are using Veeam to backup all of our production VMs and workloads to Azure using a scale-out repository. Our test case involves rescanning the object storage to import the data for a single smaller VM in order to evaluate the scenario of our entire on-premise infrastructure being lost (including the Veeam server). When we issue a rescan the data is imported fairly quickly, with a small stub file of 25MB being added locally for approximately 40GB of source data. The problem comes in when we attempt to restore the data to a Hyper-V server from object storage. The reflected rate in Veeam is only 9MB/s.

In a nutshell here's the data. If we restore from object storage to a local Hyper-V server using the process outlined above the job completes at a rate of 9MB/s (82Mb/s). The same restore of the same VM backup to the same Hyper-V host is approximately 8x that speed at around 70MB/s. When we installed Azure Storage Explorer to the Veeam server and downloaded the same data from object storage we would get transfer rates between 500 Mb/s and 700 Mb/s. We have 1Gb/s links throughout the environment where the backup solution is concerned. The object storage tier is 'Hot'.

We are desperately trying to understand why the restore operations from object storage take so long to complete. We have approximately 10TB of production VM data that we would need to restore from object storage in the event of a catastrophic failure. At a rate of 9MB/s this would give us an optimal restore window of around 13 days - far too long for the environment to be down. If we could get this closer to 500Mb/s (the rough low-end rate of our local restores, outgoing cloud backup jobs, and direct download of data from Azure) this restore window would go down to just under 2 days. We are using Veeam encryption.

Basically what we are wondering is whether or not this is expected behavior. If it is not what could we do to speed things up, or get more insight into what is causing the problem?

[nielsengelen]

Hi Brandon, I moved your topic to the relevant section. Did you already have support help you with this and do some investigation on what is causing the difference in speed?

[bwoodard]

Hi Niels,

Thank you for moving my topic to the appropriate section. We haven't engaged support for this but if your recommendation is to open a ticket and then revisit here on the forums if we don't get an answer that is definitely something we can do. I guess my general question before engaging support was whether or not it is expected that the restore operations from object storage would be so much slower.

[nielsengelen]

It shouldn't be the case and with the help of support, things could be cleared out. Maybe the logs are clarifying things already or something else is causing this as a bottleneck. Via the forums this is hard to troubleshoot hence the suggestion to contact support. If you have opened a case, could you let us know the case ID for future reference/discussions?

[veremin]

Kindly, reach our support team and request 10a RTM update. It has a lot of fixes that increase restore to public cloud performance (especially for backups stored in object storage repositories). Thanks!

[bwoodard]

Thank you both!

We've opened a support case and uploaded logs - #04306001. We are running Veeam product version 10.0.0.4461. I'll inform them of what you advised so that we can implement the fix and update this thread when we have results from support.

[veremin]

We made 10a RTM available on July, 23rd - build number is 10.0.1.4854.

So, it's definitely worth updating your installation and re-trying the restore operation.

Thanks!

[bwoodard]

That fixed it! Up from 9 MB/s to 84 MB/s, a nearly 10x improvement that matches what we see for all our other cloud operations on the 1Gb link. Thank you very much!!!

[veremin]

Thanks, Brandon, for confirmation - appreciated!