Transfer Speed to Azure Blob

[agrob]

Good Day

I'm testing SOBR with Azure Blob. I'm wondering about what transfer performance Speed others have.

Background:
Azure Storage account used is a BLOB Account, later changed it to StorageV2 LRS.
We have a 1Gbit up/down business internetlink. When a SOBR offload job does move Data to Azure blob, i get transfer speed about 51MB/s. I checked the internet link during this time to check if something else is using bandwith, this is not the case. When i do a restore (vmdk file from a vm) form the blob store, i get transfere speed about 60MB/s.

I checked the the Azure Storage Account specs under https://docs.microsoft.com/en-us/azure/ ... ty-targets
I'm not sure which data does match my questions. There is the Point
"Azure Blob storage scale targets" with the info:
Target throughput for single blob -> Up to 60 MiB per second, or up to 500 requests per second
-> This would match more or less the performance i get in my test.

Now i changed from Blob to StorageV2 LRS. There is a Point "Standart storage account scale limits" with the info:
Max ingress1 per storage account (Non-US regions) -> 5 Gbps if RA-GRS/GRS enabled, 10 Gbps for LRS/ZRS2
-> I'm located in Europe and Storage Account is LRS, this would or could give me max 10Gbps which is over 1GB/s.

so, my transfer to Azure Blob perf data:
Azure Blob Storage: arround 40MB/s
Azure StorageV2: arround 60MB/s

For me, 40-60MB/s is not fast enough, Internetconnection should go up to 100-125MB/s (yes i'm aware it is internet, but i guess with a 1Gbit Link, more than 40-60MB/s should be possible)
Full Backups are over 25TB, so it would take some time until this ammount is on the azure store. I'm not sure if with this ammount of data, azure blob is the right way or if we should go to "cheaper" NAS Storage and offload to Tape.


-> Could other forum members please share their performance data for writing/reading data from Azure Blob store?

Thanks

[Gostev]

I've noticed about same performance numbers in the adjacent topic that I just responded to.

Full Backups are over 25TB, so it would take some time until this ammount is on the azure store. I'm not sure if with this ammount of data, azure blob is the right way or if we should go to "cheaper" NAS Storage and offload to Tape.

You should really take a closer look at other object storage providers too. Both Azure and Amazon are fairly expensive in general because their provide a premium offering, but there are plenty of alternatives which can be up to a few TIMES cheaper. Check out this calculator from Wasabi for example.

[agrob]

Thanks for the wasabi link, i'll check that. I think i have to take enough time to compare data cost for offloading to cloud providers compared to local nas with offload to tape. tape would have the advantage to have the data offline, compared to the offload to cloud providers where an attacker could delete this backups as well if he has access to the backup server. it is as always, i guess the answer is, it depends...

[agrob]

the other thing is, offload 25TB of data with 60MB/s takes several days. On the other side it is a one time task when it is afterwards forever incremental.
Maybe there are some other users who share their performance data, always interessting. thanks for your fast feedback anyway Gostev!!

[Didi7]

Hello agrob,

I share the pain with you Considering that we currently have none-Cloud Veeam B&R environments, where backups from SAN storage reaches 900MB/s during active full backups, I am really excited how this develops, when most of our datacenter VMs will be transfered to Azure and finally be backed-up to Azure Blob storage, reaching not more than 60MB/s

Good luck, when you need to restore full VMs from those 25TB back to your active cloud environment.

Best regards,
Didi7

[Gbass98]

Hi Agrob,
Did you fix that issue ?
I want to use Azure Storage to backup 22TB too...

Thank you
Gbass98

[agrob]

Hello Gbass98

I did no further research on this as it was just a test. But we are in the progress for a new backup infrastructure and i guess it will pop-up there again. Maybe it would be worth have a look at aws storage? (i did not so far). I just think of that because in traditional on Prem Backup we also want to keep prod and backup data seperated. so if vms are on azure it would mybe a good idea to backup those vms to an other cloud vendor....

Just checked the latest information about azure storage accounts:(https://docs.microsoft.com/en-us/azure/ ... ty-targets)
Target throughput for single blob -> Up to storage account ingress/egress limits1

Maximum ingress1 per storage account (US, Europe regions) -> 25 Gbps
Maximum ingress1 per storage account (regions other than US and Europe) -> 5 Gbps if RA-GRS/GRS is enabled, 10 Gbps for LRS/ZRS2

But another Source: (https://docs.microsoft.com/en-us/azure/ ... -checklist)
Bandwidth and operations per blob
You can read or write to a single blob at up to a maximum of 60 MB/second (this is approximately 480 Mbps, which exceeds the capabilities of many client-side networks (including the physical NIC on the client device). In addition, a single blob supports up to 500 requests per second. If you have multiple clients that need to read the same blob and you might exceed these limits, you should consider using a CDN for distributing the blob.

For more information about target throughput for blobs, see Azure Storage Scalability and Performance Targets.

-> So i guess it is still 60MB/s...

[collinp]

So can we design around this Azure blob bandwidth cap by creating multiple scaleout repositories pointing to multiple storage accounts so that we can write at 60 MB/s in parallel to multiple Storage Accounts? So if we have 3 Scaleout repositories pointing to 3 Azure Storage Accounts, in theory we could write at 180 MB/s?

[Gostev]

Wait, I lost you guys here. How did we go from 60 MB/s per a single blob to 60 MB/s per storage account? As per the previous post, storage account ingress limit is 25 Gbps, not 60 MB/s.

And while you can only read or write to a single blob at up to a maximum of 60 MB/second, Veeam certainly does not store all backups in a single blob? More like in millions of blobs, each 1MB or less in size (by default).

[agrob]

yes, storage account limit is 25Gbps as per Microsoft, also single BLOB is limited to 60MB/s.
Created a storage account with one BLOB Container. This container is configured as azure blob backup repository and linked to a SOBR. I have a test Backup Job with two Windows VMs to this SOBR. Offload to the capacity tier is max arround 60MB/s. We have a 1 Gbit internet connection and if i monitor it during the offload, i see that the connection is arround 50% of bandwith, so it should not limit the offload.
-> So even if Veeam is writing several blobs in parallel to the blob container, we don't get more then arround 60MB/s during offload.
1. it would be interessting if other customer also experience this behaviour
2. what i would to is a second test: creating a second blob container in the same storage account, configure this as azureblob repo an use it in a second SOBR. then running two separate Backups each to one SOBR and then start the offload simultanius. as soon as i find time, i'll test it. -> but for the moment it seems that the limitation is per blob container in a storage account, not per blob IN the container...

[agrob]

made another test today
Azure
created a new blob container in the existing azure storage account. -> 1 Storage Account with 2 Blob Container

WAN Connection
1Gbit synchron business line (is also used for other traffic, during the test i did other load was about 5-10%)

VBR Config
1. each of the 2 Blob Container are configured as an azure object storage Backup repository
2. created two SOBR and added one of the object storage backup repositories as capacity tier

VBR Jobs
Created two Jobs, one for each SOBR. Source was a VMware Windows VM. Did run both jobs twice (full and active full). As a result, each SOBR had two vbk files.

Offload to Capcity Tier
1. Manually started vbk offload to capacity tier from SOBR1. At the same time i checked the Nic Performance on the gateway server which is used for offloading to the capacity tier. Sent bytes where arround 60MB/s
2. Additionally to Step 1, started the vbk offload to capacity tier from SOBR2. Nic performance on the gateway server increased. Send byte arround 90MB/s

Conclusion
1. From the tests i did, it seems that transfer to a blob container is arround 60MB/s. Veeam is writting several blobs in parallel to the container but this does not seem to increase performance.
2. Performance can be increased by adding one (or maybe more, depends also on your interent connection) blob container to the same storage account (or you can use more than one storage accounts, but from ms side, one storage account can handle up to 25Gbps, so you could configure several blob container until the limit es reached
3. Those are just my test results. It would be nice to have a best practice paper from Veeam with official statements about what can be done to reach max performance when offload or restore from cloud storage.

From those results, i don't know at the moment if it is the best thing to offload many TB (about 30TB) to the Cloud. At least to azure, dont know about other Cloud performance Data. Also the price to store that much data in the Azure cloud is expensive (but thats another story)

Regards

[Gostev]

@agrob Microsoft is interested to investigate your issue more closely. I have the email thread going with the PM for blob storage performance features - and at this point they want to see the issue live, as well as tweak some settings on your storage account to see if that helps. Could you please open a support case with Microsoft, and let me know the incident number, so that this is tracked appropriately in their workflows?

Also, please PM me your blob storage account name if you don't mind. I will pass it directly into the email thread with Microsoft, as they are standing by ready to start working on this immediately.

Thanks!

[agrob]

@gostev
thanks for your feedback. i sent you the storageaccount name PM. I have a ms case number now which i'll send you per PM as well.
regards

[Gostev]

Just to update, as far as I know, this did not go anywhere... Azure team wanted to try some special custom settings on the storage account to see if they help, but eventually lost interest or something (at least I never heard back from them). @agrob if you're still around, can you please share how this ended?

[agrob]

Hello Anton
in short: i had a support case with ms. we tested upload of vbk file with azcopy and this looked ok. so support told me, everything is right. then the case was closed. after that, HTBB should have been enabled but i never heard from support again.
At the moment we decided for several reasons that we don't use azure or other cloud storage as backup repo. so i was not able to spend more time in this...

[archytype]

It sounds to me as if the Veeam software needs to serialise the upload of the resultant files, not simply limit the concurrency.

[Gostev]

We don't upload backup files in principle. There's no file system in object storage, so there are no files. It's all... objects! Which are data and metadata blocks of original backup files.

[andre.simard]

Hi,

Any update on that issue?
is someone as been able to get better performance then the 60 MB/s?
We also have the same issue we are not able to send faster that 60MB/s to azure.

Thank you

[mortenmadsen]

Wait, I lost you guys here. How did we go from 60 MB/s per a single blob to 60 MB/s per storage account? As per the previous post, storage account ingress limit is 25 Gbps, not 60 MB/s.

And while you can only read or write to a single blob at up to a maximum of 60 MB/second, Veeam certainly does not store all backups in a single blob? More like in millions of blobs, each 1MB or less in size (by default).

But veeam creates block blobs in Azure, according to Microsoft's documentation, block blobs does not have a limit other than the storage account for a single block blob. But at the same time Microsoft recommends to write 4MB or larger blocks, so why does veeam use small objects? that is normally never a good idea with an object store.

"Throughput for a single blob depends on several factors, including, but not limited to: concurrency, request size, performance tier, speed of source for uploads, and destination for downloads. To take advantage of the performance enhancements of high-throughput block blobs, upload larger blobs or blocks. Specifically, call the Put Blob or Put Block operation with a blob or block size that is greater than 4 MiB for standard storage accounts. For premium block blob or for Data Lake Storage Gen2 storage accounts, use a block or blob size that is greater than 256 KiB."
https://docs.microsoft.com/en-us/azure/ ... ty-targets

[Gostev]

Large block sizes are hugely inefficient in terms of incremental backup size. Creating 4MB objects means operating with 8MB original image blocks (as average compression is 2x). Going from 1MB (current default) to 8MB will increase the incremental backup size at least 4x (and up to 8x in the worst case scenario for some workload types). And since our object storage integration is forever-incremental, the size of incremental backups is what defines object storage consumption. Now you tell me, do you want to pay Microsoft for storing 100TB or 400TB of data?

I don't imagine so but in case you do, you're more than welcome to enable the 8MB block size option with the special registry value.