Feature Request: parallel processing on a "per-vm"-basis

[eriktwelker]

We'd like to utiilize parallel-processing - but would like to do so on a per-vm or per-job basis, rather than enabling/disabling parallelism globally.

The reason is that we have loads of backupjobs handling many vms (e. g. 25 win-vms) with only 1-2 vmdks per vm.
On the other hand there are many backupjobs handling only 1 vm - but with up to 10 vmdks and a summarized backupsize of several TB.

Thus, we'd like to use parallel processing for some of the backupjobs while leaving others at the "standard" single processing mode.

As support reported this to be not realized yet, I'd like to file it as a feature request.

[Shestakov]

Hello Erik and welcome to the community!

So the ultimate goal is give a priority to some jobs over others?
You can just start them a bit earlier and the job will get more processing resources, while I don`t see any value of disabling the parallel processing for those jobs with 25 VMs in them.

Could you clarify it a bit?
Thanks!

[eriktwelker]

Hello Nikita (Shestakov),

well, starting some jobs earlier than other is one possibility - but as we perform backups more or less all day long (production-systems during night-times, test&dev-systems during the day) the task of exactly finding the perfect starting point for each backup job is quite "challenging" in our environment: ca. 300 VMs, ca. 100 BackupJobs.

So, we`d like to start using parallelism (for test&dev-systems to start with) and see how this affects other backups.
Thus, it'd be nice to choose which job uses parallelism, rather than playing it by "all or none".

Best regards,
Erik

[Shestakov]

Thanks for the explanation, Erik!
The question turns to be about job priority, rather than parallel processing.
In this case I would deploy more proxies, reduce max concurrent ingress tasks on repositories, or just use dedicated proxy sets for your jobs (or any combination of these approaches).

[eriktwelker]

Hi Nikita,
thanks - I'll have a deeper look at job priority then.

Nonetheless - if we'd like to benefit from parallel processing, what would you suggest to prevent parallel backups from "stalling" the whole environment?
Or to be more precise, what should we check in advance in order to NOT having (then parallel) running jobs interfere with each other?

Best regards,
Erik

[Shestakov]

If the jobs have different targets, you can decrease number of parallel tasks on the repository(target proxy).

[foggy]

Or to be more precise, what should we check in advance in order to NOT having (then parallel) running jobs interfere with each other?

Erik, what do you mean by "interfere"? Here's a good thread discussing how parallel processing works in this regard, if I'm getting right what are you referring to here.