Offload Job - Concurrent Tasks

[collinp]

When using Copy mode with a SOBR, the offload jobs to Azure are stacking up under running tasks. Is there a way to limit the number of concurrent offload jobs? In our case, we have 100's of jobs so the offload jobs take up about 3% CPU on the gateway server per offload job. I don't see a place in the object storage extent to limit concurrent offload task.

[dalbertson]

There is a way to limit these. I recommend opening a support case on this and they can provide you with the registry key if its not some other issue.

[collinp]

I see this in the Azure portal and this may be why the offloads are stacking up.

Code: Select all

Active: At 02:16 PM, Thursday, 19 March 2020 UTC, the Azure monitoring system received the following information regarding your storage account resource VeeamRepo1:  
This storage account was throttled because it exceeded Azure Storage partition request per second, partition bandwidth, or IP scalability limits.
Please use the link below to view the latest information about the health event and learn about recommended steps in Azure Resource Health.

[veremin]

Might be the reason indeed. May I ask you to open a support ticket and provide its number here?

We'd like to double check the issue internally, so having support ticket with debug logs attached would help us a lot.

Thanks!

[collinp]

Case #04067752. I also submitted a case to Microsoft. This may be a limit in Azure imposed on Storage Accounts that I may be able to overcome by offloading jobs to multiple Azure Storage accounts.

[collinp]

The offload jobs don't perform well per offload job. We have a 10gbps connection to azure and each job gets about 50-60MB/s. The bottleneck for the offload jobs only ever show source. Even if there is nothing else going on in the local repository. It is a repository with a new serve rand 50 DAS spindle RAID group. Perfmon shows low iops and low queue length. It's almost as if there is a throughput limit per blob.

[veremin]

We will investigate this issue internally; I will update the thread, once I have more information. Thanks!

[Gostev]

Guys, actually 60 MB/s is a known Azure Blob Storage limitation. Here's the existing discussion from last year, so I will lock this one down as duplicate. Thanks!

[tsukraw]

Hey guys,

Wondering if anyone else is running into this.
On version 10 using Azure Blob Storage
The issue we are seeing is not being able to control how many concurrent upload tasks are running.
What i am experiencing is i have say 20VMs backing up with a per-VM backup files. So each backup i get 20 files.

The Azure Blob immediately copy kicks off and all 20 files start uploading at the same time.
This is causing us to get the following errors:
A timeout occurred while waiting for memory resources to execute the query in resource pool 'internal' (1).
WinHttpWriteData: 12002: The operation timed out

I had found a previous article from V9.5 days.
https://rhyshammond.com/winhttpwritedat ... timed-out/

It talks about a registry value AzureConcurrentTaskLimit but this registry value does not seem to have any affect on our V10 installation.

Anyone else run into similar issue or have ideas?
Support Case:04086659

[tsukraw]

I hadnt looked hard enough.
Looks like someone else was having the exact same issue.

object-storage-f52/offload-job-concurre ... 65651.html