Proxy threads in v8

[pat_ren]

Hey I think this is more of an R&D question than a support question, so asking here first.

What's the process for managing/increasing threads in our proxy servers since updating to v8, I noticed this option is now gone but I still have the settings in my old proxy configs, new proxy doesn't have any of these settings.

example:
<Source WorkerThreads="64" FixTeamChatSync="True" BandwidthLimit="0" BandwidthLimitUnit="MegabitsPerSecond" />
<WCF OperationTimeout="9600" />

proxy threads used to be able to be edited in the proxy properties, that's gone now.
the time out and fixteamschatsync are things support have asked us to change in the past.

are these settings still used or are they just ignored now, whether they are in the config or not?
I was looking at trying to increase how many threads my proxys are doing as cpu usage is low and jobs are taking longer but I couldn't see the option anymore.

Thanks!

[Polina]

Hi Pat,

Starting v8, proxy threads are managed automatically. The threads parameter has been removed from the proxy settings in the UI, and all custom configurations in XML are ignored.

[pat_ren]

Hi Polina,

Thanks for the update, good to understand a bit better how it works. Documentation on the new proxy's has been a bit light since V8 was released. I'm trying to get a better understanding on how to optimise my environment since implementing a proxy pool. Currently we have ~520 jobs running every 8 hours, I am noticing pretty low resource usage on all my proxy servers. There's currently 8 windows proxy's in a pool and each has 8cpu cores/32GB of RAM, RAM usage is usually around 30-50% and CPU usage is pretty low, rarely gets above 20-30%. My gut feeling is there isn't enough tasks running concurrently and there's a lot of queuing going on instead so the jobs are taking longer to complete than they did before. Would it be better for us to add more smaller proxy servers and assign each one less resources, or should we be seeing the servers we have now handling more load? I'm not sure how best to scale this in V8. Thank you.

[DouglasKhais]

I noticed threads per proxy seem tied to the number of concurrent tasks set in the repo and proxy settings. Tweaking those helped me get better performance without overloading.

[DouglasKhais]

those thread settings just aren’t in the new version anymore. Ended up testing different workloads, and turns out the new system handles threads automatically, so manually tweaking them isn’t needed. If you’re looking for reliable performance for scraping or heavy tasks, datacenter proxies here helped me keep everything smooth without messing with old configs. It makes running multiple jobs way less stressful.

[pat_ren]

I noticed threads per proxy seem tied to the number of concurrent tasks set in the repo and proxy settings. Tweaking those helped me get better performance without overloading.

Proxy threads (like in v7) don't do anything, as you discovered, but there's a few things additional things you can tweak in the proxy configs, support could advise on this. Proxy performance is also tied to available resources, so more cores can help with performance scaling. Linux proxies do perform a little better, and more small proxies are recommended over less big proxies. I'm running almost 1,000 jobs every 8 hours and have spent some time tuning this

This might be of interest to you:
https://bp.veeam.com/vb365/guide/design ... proxy.html