Using object storage as a backup target
veremin
Product Manager
Posts: 18003
Liked: 1715 times
Joined: Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin
Contact:

Re: Restore speed from Azure Cool

Post by veremin »

The fix does work for Azure to on-prem, we get 200MB/s+ . However restore from Azure to Azure doenst go passed 18MB/s. According to the engineer this is what his lab also shows as speed and what to expect. Is this really the case as this make it unusable, much to slow restore speeds. And it gets divided by each active restore session.
R&D team is researching the case already. If you don't mind, keep the case open for now.

I will keep you updated on the results of our findings.

Thanks!

Gostev
SVP, Product Management
Posts: 27173
Liked: 4455 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Restore speed from Azure Cool

Post by Gostev »

Since both restore scenarios use the same data movers, I would primarily suspect some throttling on Azure side. For example, AFAIK certain instance types come with disk I/O limitations. Or perhaps there could be some around Azure networking. However, a Veeam bug impacting the data transfer chain should not be excluded from possibilities either.

b.vanhaastrecht
Service Provider
Posts: 669
Liked: 118 times
Joined: Aug 26, 2013 7:46 am
Full Name: Bastiaan van Haastrecht
Location: The Netherlands
Contact:

Re: Restore speed from Azure Cool

Post by b.vanhaastrecht »

Have conducted speed tests. With Azure Proxy on A1 profile gets no more then 170Mb/s with azcopy benchmark. And with a F1 profile we get 3000Mb/s :shock:

So the Azure VM profile does have significant impact on the upload&download speed to Azure Blob, the profile upgrade did not make Veeam Restore to Azure blob any faster. Even funnier, while conducting a restore to a storage account with Veeam, we parrallel conducted the test to the same storage account. Veeam got 11MB/s, and azcopy 3000Mb/s . Yes MB vs Mb, but about 30 times faster with azcopy to same target IP and storage account. Support is currently investigating my findings.
========================================
Veeam ProPartner and Cloud Connect Provider

Gostev
SVP, Product Management
Posts: 27173
Liked: 4455 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Restore speed from Azure Cool

Post by Gostev »

So it looks like we have a bad default value for the number of threads per restored disk. QC were able to get 200MB/s restore speed with just a registry key change. They will play a bit more with this, and let support know good combinations of Azure proxy instance type and this registry value.

Gostev
SVP, Product Management
Posts: 27173
Liked: 4455 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Restore speed from Azure Cool

Post by Gostev »

Here's what our QC suggests based on the initial testing:
1. Create the AzureDiskProcessingThreadCount = 64 (DWORD) registry value on the backup server (default is 8 threads).
2. Use the Standard_F4s_v2 proxy VM size, as the default A2 is throughput-limited. Standard_F4s should also work well.

Using higher thread count or more powerful proxy VMs did not yield additional performance benefits in their testing.

b.vanhaastrecht
Service Provider
Posts: 669
Liked: 118 times
Joined: Aug 26, 2013 7:46 am
Full Name: Bastiaan van Haastrecht
Location: The Netherlands
Contact:

Re: Restore speed from Azure Cool

Post by b.vanhaastrecht »

I've set:
HKEY_LOCAL_MACHINE\SOFTWARE\Veeam\Veeam Backup and Replication\AzureDiskProcessingThreadCount = 64 (DWORD / Decimal) on the VBR server. Then started a new restore using a F4 VM size. Did not change the threadcount and did not improve the speed. Do we need a patch or did we missed something?
========================================
Veeam ProPartner and Cloud Connect Provider

Gostev
SVP, Product Management
Posts: 27173
Liked: 4455 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Restore speed from Azure Cool

Post by Gostev »

From your post it sounds like you used F4 VM size for the restored VM, as opposed to the proxy VM.

b.vanhaastrecht
Service Provider
Posts: 669
Liked: 118 times
Joined: Aug 26, 2013 7:46 am
Full Name: Bastiaan van Haastrecht
Location: The Netherlands
Contact:

Re: Restore speed from Azure Cool

Post by b.vanhaastrecht »

Yeah, thats how I wrote it, sorry for the confusion but we did use a Azure Proxy on F4 VM size. When in parallel of a Veeam restore we do a azcopy in the same Veeam Proxy to the same Azure Blob we get 3000Mb/s. We do see the 64 thread setting coming in at the Azure VM, but the amount of TCP threads initiated from VeeamAgent.exe stays around 10. Support is busy analyzing the logs again.
========================================
Veeam ProPartner and Cloud Connect Provider

Gostev
SVP, Product Management
Posts: 27173
Liked: 4455 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Restore speed from Azure Cool

Post by Gostev »

I would not worry much about azcopy numbers, as they are not entirely fair: azcopy considers zeroed data blocks as if they were transferred in their numbers, without actually transferring them. Azcopy also uses much larger block size, making it an apple to oranges comparison in any case.

But you should definitely be getting close to 200MB/s VM restore speed from blob storage backups now, as this is what our QC is seeing with the settings above, using the very same Azure :D so let's see if support is able to find an issue with your config. By the way, are you sure you're restoring from the restore point in blob storage, and not from your local repository?

b.vanhaastrecht
Service Provider
Posts: 669
Liked: 118 times
Joined: Aug 26, 2013 7:46 am
Full Name: Bastiaan van Haastrecht
Location: The Netherlands
Contact:

Re: Restore speed from Azure Cool

Post by b.vanhaastrecht » 1 person likes this post

We got it, you must apply the hotfix in order for the AzureDiskProcessingThreadCount registry setting to work. We are now getting 150MB/s which is much more acceptable then 30MB/s we had before.

I did make a suggestion to support to try to auto-tune the threadcount to get the best restore speed. I'm testing myself to get the optimum number.

See prerequisite for fast Azure restore:
- Apply hotfix to the Azure Proxy, request from support
- F or higher VM type with at least 4 cores
- Apply registry key to VBR server with a threadcount of 64 decimal
========================================
Veeam ProPartner and Cloud Connect Provider

Gostev
SVP, Product Management
Posts: 27173
Liked: 4455 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Restore speed from Azure Cool

Post by Gostev » 1 person likes this post

Great to hear! I think we reached 200MB/s because it was a late night testing, with Azure less busy.

Did you mean the same hotfix from earlier in this thread?

We didn't see much further improvement beyond 64 threads, so we will likely just make it the new default value. There doesn't seem to be any reason to auto-tune it, at least based on our results.

syscons
Influencer
Posts: 13
Liked: never
Joined: Oct 24, 2017 6:40 pm
Full Name: Jochen Seeger
Contact:

[MERGED] Restore to Azure Performance from Azure Blob

Post by syscons »

Hi!

I just implemented a Solution at a customer with a VBR on Azure which connects to an Azure Blob Storage which is filled by the on-premise VBR (Desaster Recovery on Azure Solution)

We just tested the first direct Restore to Azure from the Azure VBR (not using an Azure Proxy), it worked very nicely, but the Performance from the Blob Storage was around 12 MB/sec.
Is this a realistic Speed or am i missing some Performance Tuning Options?

Gostev
SVP, Product Management
Posts: 27173
Liked: 4455 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Restore speed from Azure Cool

Post by Gostev »

Hi, Jochen

1. You do want to use Azure Proxy if you want restore performance.

2. Veeam Backup & Replication 10a has all the above mentioned fixes included. This release is currently in the Early Availability stage, you can open a support case and request the bits.

Thanks!

syscons
Influencer
Posts: 13
Liked: never
Joined: Oct 24, 2017 6:40 pm
Full Name: Jochen Seeger
Contact:

Re: Restore speed from Azure Cool

Post by syscons »

Hi Gostev,

thanks! i will open up a ticket then.

I was not using the Azure Proxy because of a Blog by Michael Cade where he tested several Scenarios and found the VBR solution the fastest.

https://vzilla.co.uk/vzilla-blog/veeam- ... ot-new-but

Gostev
SVP, Product Management
Posts: 27173
Liked: 4455 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Restore speed from Azure Cool

Post by Gostev »

Indeed, we simply had bad defaults for the Azure Proxy instance type, so its virtual hardware was the bottleneck. This too was changed in 10a.

MichaelCade
Veeam Software
Posts: 308
Liked: 68 times
Joined: Mar 23, 2015 11:55 am
Full Name: Michael Cade
Location: Cambridge, United Kingdom
Contact:

Re: Restore speed from Azure Cool

Post by MichaelCade » 1 person likes this post

Yes, as per my blog and the video showing restore scenarios it shows that the quickest restore scenario was using a VBR server running in Microsoft Azure.

Image

I will update with the hotfix and 10a patch results to show how this will now be greatly improved with the Azure Proxy and Veeam Backup & Replication server running in Azure.
Regards,

Michael Cade
Global Technologist
Veeam Software
Email: Michael.Cade@Veeam.com
Twitter: @MichaelCade1

mkh
Service Provider
Posts: 21
Liked: 5 times
Joined: Apr 20, 2018 6:17 am
Full Name: Michael Høyer
Contact:

Re: Restore speed from Azure Cool

Post by mkh » 1 person likes this post

hi all

tested on Veeam v10a

short summary of setup: onprem server - storage account in azure - 1 Gbps internet connection

copy to performance tier:
now it is 60 MBps in Veeam, 300-700 Mbps on netcard - before with just the fix it was around 26 MBps, just like it was initially

restore directly from storage account (performance tier in maintenance mode):
now it is 110 MBps in Veeam, 1 Gbps on netcard - before with just the fix it was 45 MBps in veeam, initally it was 10-15 MBps

Im happy with the performance at this customer now, and also more confident in using object storage at other customers in the future

Gostev
SVP, Product Management
Posts: 27173
Liked: 4455 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Restore speed from Azure Cool

Post by Gostev »

Thanks for taking time to share your new 10a numbers, Michael.
Good to have the proof it is performing very well now!

mkh
Service Provider
Posts: 21
Liked: 5 times
Joined: Apr 20, 2018 6:17 am
Full Name: Michael Høyer
Contact:

Re: Restore speed from Azure Cool

Post by mkh »

your welcome :)

MichaelCade
Veeam Software
Posts: 308
Liked: 68 times
Joined: Mar 23, 2015 11:55 am
Full Name: Michael Cade
Location: Cambridge, United Kingdom
Contact:

Re: Restore speed from Azure Cool

Post by MichaelCade »

Just an update on some testing so far.

Image

That is a super impressive update on restore performance, granted the machine is small but this is in some places halving the time of restore from v10!

The best restored at speed I have seen is 18 MB/s in my lab network.

I will update again when I have the Azure results.
Regards,

Michael Cade
Global Technologist
Veeam Software
Email: Michael.Cade@Veeam.com
Twitter: @MichaelCade1

MichaelCade
Veeam Software
Posts: 308
Liked: 68 times
Joined: Mar 23, 2015 11:55 am
Full Name: Michael Cade
Location: Cambridge, United Kingdom
Contact:

Re: Restore speed from Azure Cool

Post by MichaelCade »

I in fact went back and tested the top scenario again and it was complete in 7.15. Impressive stuff!
Regards,

Michael Cade
Global Technologist
Veeam Software
Email: Michael.Cade@Veeam.com
Twitter: @MichaelCade1

MichaelCade
Veeam Software
Posts: 308
Liked: 68 times
Joined: Mar 23, 2015 11:55 am
Full Name: Michael Cade
Location: Cambridge, United Kingdom
Contact:

Re: Restore speed from Azure Cool

Post by MichaelCade »

For those that are interested I have completed the testing in comparison to the v10 release.

Image

I also went through this Azure based VBR and Proxy testing here - https://www.youtube.com/watch?v=nBxSrA0zxXg whilst also showing that deployment through the Microsoft Azure Marketplace.
Regards,

Michael Cade
Global Technologist
Veeam Software
Email: Michael.Cade@Veeam.com
Twitter: @MichaelCade1

Post Reply

Who is online

Users browsing this forum: No registered users and 3 guests