Feature Request - Job level concurrent task limit

[mcdaidh]

Currently you set concurrent task limits at the proxy and repository level. It would be helpful to be able to set a concurrent task limit at a job level to help alleviate the issue of long running jobs taking up resources from other jobs and stopping them from starting.

For example: Backup Copy jobs taking resources away from Backup jobs.
1 x host with 24 concurrent tasks limit (24 CPU cores)
2 x Repositories with 8 concurrent tasks limit on each repository
1 x Backup job targeting repository 1
1 x Backup Copy job copying backup date from repository 1 to 2.

Issue I have experienced. The Backup Copy job is taking a long time to copy the backup data from repo 1 to 2, and is using all 8 available concurrent task treads on both repo 1 and 2. When the backup job tries to start, it has to wait for resources to become available due the the Backup Copy job taking up all the repository concurrent tasks. If we were able to set a job limit, I could set the Backup Copy job to say, use only 4 threads, and would allow my Backup Job to start with some restricted concurrent tasks until the Backup Copy job finishes.

Before any suggest just up'ing the task limit on the repos, our production environment is backing up over 600 VMs using multiple jobs for multiple different cost centers and multiple repositories through 4 physical proxy hosts. The example is there to explain the issue. We have horrible chargeback requirements.

[HannesK]

Hello,
I think it would be helpful if task configuration would not be needed at all

through 4 physical proxy hosts

that sounds like a massive amount of proxies for only 600 VMs. So I make the suggestion you don't want to hear:

1 x host with 24 concurrent tasks limit (24 CPU cores)

normally you can overbook (V12 will also reflect that in the GUI). Proxy up to 2:1 and repository 3:1 to 5:1 with Enterprise Application Plugins. That might solve the issue for you simply with more tasks (assuming that you have CPU cycles available). I also assume, that you run incremental forever (not active full).

Backup jobs have priority over backup copy jobs. So once a task finished for a backup copy job, the backup job would get the repository task. Agree, not always perfect, but Instead of waiting for a new feature, I would really try to solve it by balancing tasks in a way that fits the needs.

Best regards,
Hannes

[mcdaidh]

Due to the way we charge-back the cost of backups to the individual departments, our backup infrastructure is more complex than it could be. The backups are active at 2 different sites with 2 physical proxies per site. There is a separate job per department with it's own repo for each of those jobs. Site 1 replicates to site 2, and site 2 replicates to site 1. We do GFS jobs with synthetic fulls weekly.

[edison5000]

Hello,
I think it would be helpful if task configuration would not be needed at all

Best regards,
Hannes

THIS. Agree. : ) When I first read about SOBR's I (very wrongly) assumed that regardless of SOBR type, Veeam would just automagically deduce resource allocation from either examining the server specs it is installed on + the added storage repos, or, at least ask you to punch in some info and then it would handle it accordingly as best it can. . Currently, SOBRing requires a lot of periodic hands-on management.

[vmtech123]

I'll piggy back on this and request some sort of monitoring of task count within Veeam console .

It would be neat to see that as I use my proxies for tape servers, and as repos. (adding another physical server to migrate tape server role shortly, and maybe add another proxy role)

It would also be cool to see task count next to latency of storage, and CPU usage etc. Does Veeam ONE monitor this stuff?

This is one of the most challenging things as "right sizing" and performance tuning can vary so much. This would be a cool feature and more performance related info can help monitor small changes made.

[HannesK]

Hello,

Does Veeam ONE monitor this stuff?

no. I know a customer that grabs that information directly from the database, but I don't recommend that.

As I mentioned above: the goal is to make the software "self optimizing". For example CDP does not have task settings...

Best regards,
Hannes

[vmtech123]

Right, but manually selecting the amount of tasks per proxy, repo etc, is the opposite. I HAVE to choose and could technically be making things worse or underutilized.

For totally "Self optimizing", Veeam should just increase tasks and monitor until latency gets bad, or CPU hits a unsustainable level then back of "X" amount.
Keep a history in the server somewhere and average it over time for self optimized settings. Heck you could run some AI on backup windows, times, jobs etc. and get the most efficient strategy tailored for fastest window, or based on VM priority etc.

Following best practices my CPU's were hardly touched. Especially when I was using my Proxy's for Repo, Tape, copy jobs etc. Because I had to split the tasks up 99% of the time they were not all full. The only way I could push it was to overprovision my proxies a bit.

Now my target storage is usually the bottle neck which is perfect.

[DaStivi]

I had this feature request too quite some time ago...

post443808.html#p443808

And I love the "AI" idea... I do manually over provisioning the tasks on the proxys... And it take some time to tune everything, but also when some changes occur this might be off again, so an automatic ruleset of how much tasks are used would be great...

[RubinCompServ]

Currently you set concurrent task limits at the proxy and repository level. It would be helpful to be able to set a concurrent task limit at a job level to help alleviate the issue of long running jobs taking up resources from other jobs and stopping them from starting.

For example: Backup Copy jobs taking resources away from Backup jobs.
1 x host with 24 concurrent tasks limit (24 CPU cores)
2 x Repositories with 8 concurrent tasks limit on each repository
1 x Backup job targeting repository 1
1 x Backup Copy job copying backup date from repository 1 to 2.

Issue I have experienced. The Backup Copy job is taking a long time to copy the backup data from repo 1 to 2, and is using all 8 available concurrent task treads on both repo 1 and 2. When the backup job tries to start, it has to wait for resources to become available due the the Backup Copy job taking up all the repository concurrent tasks. If we were able to set a job limit, I could set the Backup Copy job to say, use only 4 threads, and would allow my Backup Job to start with some restricted concurrent tasks until the Backup Copy job finishes.

Before any suggest just up'ing the task limit on the repos, our production environment is backing up over 600 VMs using multiple jobs for multiple different cost centers and multiple repositories through 4 physical proxy hosts. The example is there to explain the issue. We have horrible chargeback requirements.

I'd like to add a +1 to this idea, as well.

[18436572]

This sounds similar to the storage latency control settings ... where VB&R watches my ESX latency and throttles up and down based on observed vcenter latency figures.

I do have times where concurrent tasks are not quite enough. On very rare occasions, I will bump this figure up temporarily.

[YouGotServered]

I could see how this could get very complex.

For example, you'd also have to monitor the ESXi hosts of your hot-add proxies (if you use them). I've seen a scenario where a customer gave a hot-add proxy a bit too much CPU on the ESXi host.They were backing up from local ESXi SSD storage over a 10GB link to a nice big RAID 10 Veeam server. Performance was awesome, but that meant that the proxy would use 100% of it's Windows CPU resources. On the surface, that's not a big deal for a VM with a singular purpose, HOWEVER, the issue was that due to the overallocation of CPU to the VM at the ESXi host level, it actually consumed ALL of the available CPU on the production ESXi host. I actually have a screenshot of the host reporting "1 MHz" of available CPU.

It was actually pretty funny.

[ashleyw]

Incidentally we were seeing copy job failures (with message on the source of the copy job saying it was out of space and couldn't write to the /tmp folder).
What was happening in our case is the destination repository the copy jobs were pushing to was not throttled at the repository level so too many concurrent copys were in progress (more than 100s in our case) and every copy seems to result in a temporary Veeam folder on the source repository (presumably to track the progress etc) and this was sufficient to fill the /tmp file system on the linux source repository causing further errors on the copy job.

Once we throttled the destination repository to 8 concurrent tasks, then reliability was restored and the source repository didn't run out of space any more.

I've attached a screen shot in our case of what the /tmp directory was looking like before we throttled the repository.

So like others, I'd love to see a global Veeam setting for maximum concurrent copy tasks per copy job, to save us having to artificially constrain the repository itself.

https://iili.io/JXP8k9j.png

[MorNor]

Currently you set concurrent task limits at the proxy and repository level. It would be helpful to be able to set a concurrent task limit at a job level to help alleviate the issue of long running jobs taking up resources from other jobs and stopping them from starting.

For example: Backup Copy jobs taking resources away from Backup jobs.
1 x host with 24 concurrent tasks limit (24 CPU cores)
2 x Repositories with 8 concurrent tasks limit on each repository
1 x Backup job targeting repository 1
1 x Backup Copy job copying backup date from repository 1 to 2.

Issue I have experienced. The Backup Copy job is taking a long time to copy the backup data from repo 1 to 2, and is using all 8 available concurrent task treads on both repo 1 and 2. When the backup job tries to start, it has to wait for resources to become available due the the Backup Copy job taking up all the repository concurrent tasks. If we were able to set a job limit, I could set the Backup Copy job to say, use only 4 threads, and would allow my Backup Job to start with some restricted concurrent tasks until the Backup Copy job finishes.

Before any suggest just up'ing the task limit on the repos, our production environment is backing up over 600 VMs using multiple jobs for multiple different cost centers and multiple repositories through 4 physical proxy hosts. The example is there to explain the issue. We have horrible chargeback requirements.

I'd like to +1 to concurrent task limit on joblevel as well. We currently have this issue where the backupjobs might get stuck (not a fault in Veeam, but on the hypervisor), I mean really stuck, cannot even restart the Veeam-service. When this happens all other backup- and copyjobs with same targetrepo suffers because they wait for resources at the repo. The only way to limit concurrent tasks on the affected backupjobs is to setup one virtual proxy per job with this certain issue and limit concurrent tasks on those proxys. It would be more resource saving and neat to limit concurrent tasks in job-setting.