Discussions related to using object storage as a backup target.
Locked
collinp
Expert
Posts: 230
Liked: 13 times
Joined: Feb 14, 2012 8:56 pm
Full Name: Collin P
Contact:

Offload Job - Concurrent Tasks

Post by collinp »

When using Copy mode with a SOBR, the offload jobs to Azure are stacking up under running tasks. Is there a way to limit the number of concurrent offload jobs? In our case, we have 100's of jobs so the offload jobs take up about 3% CPU on the gateway server per offload job. I don't see a place in the object storage extent to limit concurrent offload task.
dalbertson
Veeam Software
Posts: 492
Liked: 175 times
Joined: Jul 21, 2015 12:38 pm
Full Name: Dustin Albertson
Contact:

Re: Offload Job - Concurrent Tasks

Post by dalbertson »

There is a way to limit these. I recommend opening a support case on this and they can provide you with the registry key if its not some other issue.
Dustin Albertson | Director of Product Management - Cloud & Applications | Veeam Product Management, Alliances
collinp
Expert
Posts: 230
Liked: 13 times
Joined: Feb 14, 2012 8:56 pm
Full Name: Collin P
Contact:

Re: Offload Job - Concurrent Tasks

Post by collinp »

I see this in the Azure portal and this may be why the offloads are stacking up.

Code: Select all

Active: At 02:16 PM, Thursday, 19 March 2020 UTC, the Azure monitoring system received the following information regarding your storage account resource VeeamRepo1:  
This storage account was throttled because it exceeded Azure Storage partition request per second, partition bandwidth, or IP scalability limits.
Please use the link below to view the latest information about the health event and learn about recommended steps in Azure Resource Health.
veremin
Product Manager
Posts: 20270
Liked: 2252 times
Joined: Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin
Contact:

Re: Offload Job - Concurrent Tasks

Post by veremin »

Might be the reason indeed. May I ask you to open a support ticket and provide its number here?

We'd like to double check the issue internally, so having support ticket with debug logs attached would help us a lot.

Thanks!
collinp
Expert
Posts: 230
Liked: 13 times
Joined: Feb 14, 2012 8:56 pm
Full Name: Collin P
Contact:

Re: Offload Job - Concurrent Tasks

Post by collinp »

Case #04067752. I also submitted a case to Microsoft. This may be a limit in Azure imposed on Storage Accounts that I may be able to overcome by offloading jobs to multiple Azure Storage accounts.
collinp
Expert
Posts: 230
Liked: 13 times
Joined: Feb 14, 2012 8:56 pm
Full Name: Collin P
Contact:

Re: Offload Job - Concurrent Tasks

Post by collinp »

The offload jobs don't perform well per offload job. We have a 10gbps connection to azure and each job gets about 50-60MB/s. The bottleneck for the offload jobs only ever show source. Even if there is nothing else going on in the local repository. It is a repository with a new serve rand 50 DAS spindle RAID group. Perfmon shows low iops and low queue length. It's almost as if there is a throughput limit per blob.
veremin
Product Manager
Posts: 20270
Liked: 2252 times
Joined: Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin
Contact:

Re: Offload Job - Concurrent Tasks

Post by veremin »

We will investigate this issue internally; I will update the thread, once I have more information. Thanks!
Gostev
Chief Product Officer
Posts: 31459
Liked: 6648 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Offload Job - Concurrent Tasks

Post by Gostev »

Guys, actually 60 MB/s is a known Azure Blob Storage limitation. Here's the existing discussion from last year, so I will lock this one down as duplicate. Thanks!
tsukraw
Enthusiast
Posts: 46
Liked: 2 times
Joined: Jul 06, 2012 8:28 pm
Full Name: Tucker Sukraw
Contact:

[MERGED] Azure Blob Storage Upload Concurrent Throttle V10

Post by tsukraw »

Hey guys,

Wondering if anyone else is running into this.
On version 10 using Azure Blob Storage
The issue we are seeing is not being able to control how many concurrent upload tasks are running.
What i am experiencing is i have say 20VMs backing up with a per-VM backup files. So each backup i get 20 files.

The Azure Blob immediately copy kicks off and all 20 files start uploading at the same time.
This is causing us to get the following errors:
A timeout occurred while waiting for memory resources to execute the query in resource pool 'internal' (1).
WinHttpWriteData: 12002: The operation timed out

I had found a previous article from V9.5 days.
https://rhyshammond.com/winhttpwritedat ... timed-out/

It talks about a registry value AzureConcurrentTaskLimit but this registry value does not seem to have any affect on our V10 installation.

Anyone else run into similar issue or have ideas?
Support Case:04086659
Tucker Sukraw
Network Architect
West Central Technology
tsukraw
Enthusiast
Posts: 46
Liked: 2 times
Joined: Jul 06, 2012 8:28 pm
Full Name: Tucker Sukraw
Contact:

Re: Azure Blob Storage Upload Concurrent Throttle V10

Post by tsukraw »

I hadnt looked hard enough.
Looks like someone else was having the exact same issue.

object-storage-f52/offload-job-concurre ... 65651.html
Tucker Sukraw
Network Architect
West Central Technology
Locked

Who is online

Users browsing this forum: No registered users and 9 guests