Discussions specific to object storage
Post Reply
agrob
Expert
Posts: 190
Liked: 20 times
Joined: Sep 05, 2011 1:31 pm
Full Name: Andre
Contact:

Transfer Speed to Azure Blob

Post by agrob » Feb 20, 2019 1:19 pm

Good Day

I'm testing SOBR with Azure Blob. I'm wondering about what transfer performance Speed others have.

Background:
Azure Storage account used is a BLOB Account, later changed it to StorageV2 LRS.
We have a 1Gbit up/down business internetlink. When a SOBR offload job does move Data to Azure blob, i get transfer speed about 51MB/s. I checked the internet link during this time to check if something else is using bandwith, this is not the case. When i do a restore (vmdk file from a vm) form the blob store, i get transfere speed about 60MB/s.

I checked the the Azure Storage Account specs under https://docs.microsoft.com/en-us/azure/ ... ty-targets
I'm not sure which data does match my questions. There is the Point
"Azure Blob storage scale targets" with the info:
Target throughput for single blob -> Up to 60 MiB per second, or up to 500 requests per second
-> This would match more or less the performance i get in my test.

Now i changed from Blob to StorageV2 LRS. There is a Point "Standart storage account scale limits" with the info:
Max ingress1 per storage account (Non-US regions) -> 5 Gbps if RA-GRS/GRS enabled, 10 Gbps for LRS/ZRS2
-> I'm located in Europe and Storage Account is LRS, this would or could give me max 10Gbps which is over 1GB/s.

so, my transfer to Azure Blob perf data:
Azure Blob Storage: arround 40MB/s
Azure StorageV2: arround 60MB/s

For me, 40-60MB/s is not fast enough, Internetconnection should go up to 100-125MB/s (yes i'm aware it is internet, but i guess with a 1Gbit Link, more than 40-60MB/s should be possible)
Full Backups are over 25TB, so it would take some time until this ammount is on the azure store. I'm not sure if with this ammount of data, azure blob is the right way or if we should go to "cheaper" NAS Storage and offload to Tape.


-> Could other forum members please share their performance data for writing/reading data from Azure Blob store?


Thanks

Gostev
SVP, Product Management
Posts: 24979
Liked: 3629 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Transfer Speed to Azure Blob

Post by Gostev » Feb 20, 2019 1:29 pm

I've noticed about same performance numbers in the adjacent topic that I just responded to.
agrob wrote:
Feb 20, 2019 1:19 pm
Full Backups are over 25TB, so it would take some time until this ammount is on the azure store. I'm not sure if with this ammount of data, azure blob is the right way or if we should go to "cheaper" NAS Storage and offload to Tape.
You should really take a closer look at other object storage providers too. Both Azure and Amazon are fairly expensive in general because their provide a premium offering, but there are plenty of alternatives which can be up to a few TIMES cheaper. Check out this calculator from Wasabi for example.

agrob
Expert
Posts: 190
Liked: 20 times
Joined: Sep 05, 2011 1:31 pm
Full Name: Andre
Contact:

Re: Transfer Speed to Azure Blob

Post by agrob » Feb 20, 2019 1:44 pm

Thanks for the wasabi link, i'll check that. I think i have to take enough time to compare data cost for offloading to cloud providers compared to local nas with offload to tape. tape would have the advantage to have the data offline, compared to the offload to cloud providers where an attacker could delete this backups as well if he has access to the backup server. it is as always, i guess the answer is, it depends... ;)

agrob
Expert
Posts: 190
Liked: 20 times
Joined: Sep 05, 2011 1:31 pm
Full Name: Andre
Contact:

Re: Transfer Speed to Azure Blob

Post by agrob » Feb 20, 2019 2:02 pm

;) the other thing is, offload 25TB of data with 60MB/s takes several days. On the other side it is a one time task when it is afterwards forever incremental.
Maybe there are some other users who share their performance data, always interessting. thanks for your fast feedback anyway Gostev!!

Didi7
Expert
Posts: 343
Liked: 36 times
Joined: Oct 17, 2014 8:09 am
Contact:

Re: Transfer Speed to Azure Blob

Post by Didi7 » Mar 11, 2019 9:41 pm

Hello agrob,

I share the pain with you ;) Considering that we currently have none-Cloud Veeam B&R environments, where backups from SAN storage reaches 900MB/s during active full backups, I am really excited how this develops, when most of our datacenter VMs will be transfered to Azure and finally be backed-up to Azure Blob storage, reaching not more than 60MB/s :shock:

Good luck, when you need to restore full VMs from those 25TB back to your active cloud environment.

Best regards,
Didi7
Using the most recent Veeam B&R in 19 different environments now and counting!

Gbass98
Lurker
Posts: 1
Liked: never
Joined: Aug 28, 2019 6:50 pm
Full Name: Alexandre Labelle
Contact:

Re: Transfer Speed to Azure Blob

Post by Gbass98 » Aug 28, 2019 6:58 pm

Hi Agrob,
Did you fix that issue ?
I want to use Azure Storage to backup 22TB too...

Thank you
Gbass98

agrob
Expert
Posts: 190
Liked: 20 times
Joined: Sep 05, 2011 1:31 pm
Full Name: Andre
Contact:

Re: Transfer Speed to Azure Blob

Post by agrob » Sep 02, 2019 5:18 am 1 person likes this post

Hello Gbass98

I did no further research on this as it was just a test. But we are in the progress for a new backup infrastructure and i guess it will pop-up there again. Maybe it would be worth have a look at aws storage? (i did not so far). I just think of that because in traditional on Prem Backup we also want to keep prod and backup data seperated. so if vms are on azure it would mybe a good idea to backup those vms to an other cloud vendor....

Just checked the latest information about azure storage accounts:(https://docs.microsoft.com/en-us/azure/ ... ty-targets)
Target throughput for single blob -> Up to storage account ingress/egress limits1

Maximum ingress1 per storage account (US, Europe regions) -> 25 Gbps
Maximum ingress1 per storage account (regions other than US and Europe) -> 5 Gbps if RA-GRS/GRS is enabled, 10 Gbps for LRS/ZRS2

But another Source: (https://docs.microsoft.com/en-us/azure/ ... -checklist)
Bandwidth and operations per blob
You can read or write to a single blob at up to a maximum of 60 MB/second (this is approximately 480 Mbps, which exceeds the capabilities of many client-side networks (including the physical NIC on the client device). In addition, a single blob supports up to 500 requests per second. If you have multiple clients that need to read the same blob and you might exceed these limits, you should consider using a CDN for distributing the blob.

For more information about target throughput for blobs, see Azure Storage Scalability and Performance Targets.

-> So i guess it is still 60MB/s...

collinp
Expert
Posts: 152
Liked: 10 times
Joined: Feb 14, 2012 8:56 pm
Full Name: Collin P
Contact:

Re: Transfer Speed to Azure Blob

Post by collinp » Oct 10, 2019 7:19 pm

So can we design around this Azure blob bandwidth cap by creating multiple scaleout repositories pointing to multiple storage accounts so that we can write at 60 MB/s in parallel to multiple Storage Accounts? So if we have 3 Scaleout repositories pointing to 3 Azure Storage Accounts, in theory we could write at 180 MB/s?

Gostev
SVP, Product Management
Posts: 24979
Liked: 3629 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Transfer Speed to Azure Blob

Post by Gostev » Oct 10, 2019 11:43 pm

Wait, I lost you guys here. How did we go from 60 MB/s per a single blob to 60 MB/s per storage account? As per the previous post, storage account ingress limit is 25 Gbps, not 60 MB/s.

And while you can only read or write to a single blob at up to a maximum of 60 MB/second, Veeam certainly does not store all backups in a single blob? More like in millions of blobs, each 1MB or less in size (by default).

agrob
Expert
Posts: 190
Liked: 20 times
Joined: Sep 05, 2011 1:31 pm
Full Name: Andre
Contact:

Re: Transfer Speed to Azure Blob

Post by agrob » Oct 14, 2019 6:20 am

yes, storage account limit is 25Gbps as per Microsoft, also single BLOB is limited to 60MB/s.
Created a storage account with one BLOB Container. This container is configured as azure blob backup repository and linked to a SOBR. I have a test Backup Job with two Windows VMs to this SOBR. Offload to the capacity tier is max arround 60MB/s. We have a 1 Gbit internet connection and if i monitor it during the offload, i see that the connection is arround 50% of bandwith, so it should not limit the offload.
-> So even if Veeam is writing several blobs in parallel to the blob container, we don't get more then arround 60MB/s during offload.
1. it would be interessting if other customer also experience this behaviour
2. what i would to is a second test: creating a second blob container in the same storage account, configure this as azureblob repo an use it in a second SOBR. then running two separate Backups each to one SOBR and then start the offload simultanius. as soon as i find time, i'll test it. -> but for the moment it seems that the limitation is per blob container in a storage account, not per blob IN the container...

agrob
Expert
Posts: 190
Liked: 20 times
Joined: Sep 05, 2011 1:31 pm
Full Name: Andre
Contact:

Re: Transfer Speed to Azure Blob

Post by agrob » Oct 16, 2019 7:25 am

made another test today
Azure
created a new blob container in the existing azure storage account. -> 1 Storage Account with 2 Blob Container

WAN Connection
1Gbit synchron business line (is also used for other traffic, during the test i did other load was about 5-10%)

VBR Config
1. each of the 2 Blob Container are configured as an azure object storage Backup repository
2. created two SOBR and added one of the object storage backup repositories as capacity tier

VBR Jobs
Created two Jobs, one for each SOBR. Source was a VMware Windows VM. Did run both jobs twice (full and active full). As a result, each SOBR had two vbk files.

Offload to Capcity Tier
1. Manually started vbk offload to capacity tier from SOBR1. At the same time i checked the Nic Performance on the gateway server which is used for offloading to the capacity tier. Sent bytes where arround 60MB/s
2. Additionally to Step 1, started the vbk offload to capacity tier from SOBR2. Nic performance on the gateway server increased. Send byte arround 90MB/s

Conclusion
1. From the tests i did, it seems that transfer to a blob container is arround 60MB/s. Veeam is writting several blobs in parallel to the container but this does not seem to increase performance.
2. Performance can be increased by adding one (or maybe more, depends also on your interent connection) blob container to the same storage account (or you can use more than one storage accounts, but from ms side, one storage account can handle up to 25Gbps, so you could configure several blob container until the limit es reached
3. Those are just my test results. It would be nice to have a best practice paper from Veeam with official statements about what can be done to reach max performance when offload or restore from cloud storage.

From those results, i don't know at the moment if it is the best thing to offload many TB (about 30TB) to the Cloud. At least to azure, dont know about other Cloud performance Data. Also the price to store that much data in the Azure cloud is expensive (but thats another story)

Regards

Gostev
SVP, Product Management
Posts: 24979
Liked: 3629 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Transfer Speed to Azure Blob

Post by Gostev » Oct 24, 2019 7:09 pm

@agrob Microsoft is interested to investigate your issue more closely. I have the email thread going with the PM for blob storage performance features - and at this point they want to see the issue live, as well as tweak some settings on your storage account to see if that helps. Could you please open a support case with Microsoft, and let me know the incident number, so that this is tracked appropriately in their workflows?

Also, please PM me your blob storage account name if you don't mind. I will pass it directly into the email thread with Microsoft, as they are standing by ready to start working on this immediately.

Thanks!

agrob
Expert
Posts: 190
Liked: 20 times
Joined: Sep 05, 2011 1:31 pm
Full Name: Andre
Contact:

Re: Transfer Speed to Azure Blob

Post by agrob » Oct 25, 2019 8:54 am

@gostev
thanks for your feedback. i sent you the storageaccount name PM. I have a ms case number now which i'll send you per PM as well.
regards

Post Reply

Who is online

Users browsing this forum: No registered users and 3 guests