Traffic Task / Connection throttling for offload job

[cookiejc]

Hi, I've just enabled backup to blob storage using SOBR for a small remote site of ours. The site only has a 20Mbps internet connection. When the Offload job initially ran, the internet connection was overwhelmed which caused an impact. I quickly enabled the Internet throttling rule (set at 10Mbps) and ran the job again but the same impact occurs. It seems despite the rule, when the offload job runs, all VMs in the job immediately have a task within the job and start uploading. Once some of these tasks have failed with IO timeouts to Azure, the remaining are left running at roughly the throttled speed

Is there any way to control this behaviour when the job initially kicks off? I'd rather it queued the tasks inline with the bandwidth limit rather than kick them all of at once

Thanks

[ejenner]

Is it possible you're working with the quoted speed of 20mbps and in fact it's actually running a lot slower? It's fairly normal for the quoted speed of the link from the ISP to be nothing like the actual speed you really experience when using the link? What if you try throttling lower like 1mbps?

[veremin]

Not sure whether I got the question - did the applying the corresponding rule throttle the uploading speed to blob storage or it did not? Thanks!

[cookiejc]

Hi, sorry I should have been clearer. The throttling does kick in but not after many of the tasks in the offload job have already failed due to IO timeouts. Its as if the job kicks off all tasks at once (one per VM within the job) maxes out the internet connection causing some tasks to fail and then once it's been running for 10 minutes or so it throttles the speed down to the correct level.

I have raised a support case as i'd ideally like to limit the amount of tasks an offload job can initiate at once, I just wondered if anyone else had come across this

The circuit is a leased line and definitely reaches the stated bandwidth, its simply the initial process of starting the offload job which seems to ignore any throttle which has been put in place

[ejenner]

I wouldn't have seen this as my strategy is one system per job. It means I have a lot of jobs... but I prefer the granularity, the extra config overhead is worth it.

What might be happening is that the throttling does not apply to the process. It just applies to the data transfer once the session has been established. That makes logical sense to me.

So in your case the job begins - BANG - all sessions negotiations begin simultaneously - this process requires bandwidth but the bandwidth it is using is more for job management and nothing to do with the data transfer process which can be throttled.

If you started one backup of one machine you may not see the same spike in bandwidth (I'm not sure what Veeam transfers at that point?).

[cookiejc]

Yep I see your point and that could certainly be the case. There are multiple backups jobs but they are scooped up by a single copy job which is what uses the SOBR / Blob storage.

I have some info from support which i'm waiting for a window to test. There appears to be a reg key to control the number of tasks to blob storage.

I'll test this and report back whether it controls the behavior!

[veremin]

The described situation does not look expected, so, please, keep working with our support team on addressing this. Kindly, update the topic, once you find the root cause. Thanks!