[Feature-Request] per VM or per Job tasks limitation

[DaStivi]

Hello,
i've the request to limit the simultaneous tasks on an individual vm or job.

little explanation when this could be helpfully... imagine an VMware system with around 700 to 800 Vms!
many VMs are larger than 5TB more to 10TB there are even vms with around 50TB!

these VMs contain for example 20-30disks... the issue is now even with big physical Backupproxys (using DirectSAN too) that 1 VMs Backup can use all available Backuprepository Tasks!
Of course you can increase the amount of concurrent tasks on the backup repositorys but sometime you will hit a sweetspot in how many parallel tasks the subsystem can handle, so its not really an option to raise these numbers to around, lets say 50 or more...

now a few vms where backup jobs are running can block all other 650vms from processing...
i've opened an support case, as i have had hoped that there is any hidden regkey to limit such thing on vm/job base but supported told me that there isn't... i've got an maybe workaround, that can help indeed...
you can lower the parallel tasks for a specific backup proxy (worker) and use that in these special jobs... this would help, but in my particular case it's not really possible, as i'm using directSAN, having 3 big machines that are the workers and even more using direct attached storage for primary backup copy, ideally having backup repository proxy affinity activated that no unnecessary network traffic is produced and these 3 backup repos are bundled in one SoBR.... when limiting now one worker to handly only 5 or 10 tasks (simultaneous disk backups) will interrupt also overall performance as a third of the processing power is limited for no use at all....

[foggy]

Hi Stephan, I understand the request, but please note that limiting the number of tasks for a particular VM will result in it running on snapshot for a longer time, which is definitely not desired. You'd want all disks of a single VM to be processed as soon as possible, so that the snapshot does not grow too much and the VM not suffer from snapshot commit (snapshot is triggered for the entire VM, for all its disks at the same moment). Limiting the number of tasks for the job is not that crucial, but still can result in similar issues.

[DaStivi]

Hi, ok this of course something that's not really wanted... But does have no side effect when using directsan for example... Also this is why I've hoped that is maybe something hidden or let's call it "expert settings" that needs to be enabled before using it...

[Gostev]

As a workaround, you could simply order those few biggest VMs to be processed last by the job?

[DaStivi]

sadly not really a solution in my particual case right now, as these vms is an separated job and a full backup (50TB) is running around 4-5 days! (source is only around 150mb/s -> "nearline storage"

[tsightler]

The only thing you can really do today in this case is create a separate repository and point these big jobs to those repos. Perhaps not a perfect solution, but that way you can set tasks per repo and limit the big jobs from using everything while allowing the other jobs to run.

[DaStivi]

thanks for that input... that would be in fact a solution! i will check if this is maybe a way when the big jobs start making trouble.... infact i've opened another case right now because i got "unable to allocate processing resources. Error Repository XXX status in Unavailable" but i don't know if this is now some sideeffect of the "big job" still running because not every vm got this error, its more random, and other backups are running whilst the errors i don't see any other issues on the Proxys or Veeam backup server, and furthermore i suppose normally the jobs should only go "onhold" with "waiting for backup resources to be available" and not failing on particular vms...

[foggy]

Are you seeing this message for all jobs that are targeted to this particular "Repository XXX"? Basically, "unable to allocate" means that there's no proxy or repository slot that could potentially be released in future and assigned to this particular task (for example, all available proxies are offline or disabled, repository is unavailable for some reason, etc.). You're right that typically you should see "waiting for resources" message, so if you're seeing this just for some of the VMs going to the same repo, it is suspicious.

[mkretzer]

Per Job limitation would also help with our problem from veeam-backup-replication-f2/sobr-per-job-t46069.html !

Why are there no such limits? With per-VM these are absolutely necesarry!

[patrickbeau]

sadly not really a solution in my particual case right now, as these vms is an separated job and a full backup (50TB) is running around 4-5 days! (source is only around 150mb/s -> "nearline storage"

Did you try to backup with Synthetic full and two days after, health check once a month? (with no Rollback transform because it is a time consuming process)

[ryan.christensen]

Bumping this. We should be able to limit tasks per job. It should just be another setting in the job itself under advanced options. I frequently see backup copy jobs using all available tasks then when I go to do a restore, the restore sits there saying "Resource not ready: Backup repository". The bandwidth pipe is only so large. For backup copy jobs, once you get to processing two tasks I don't see any benefit in having more than that.

[micoolpaul]

As you’re a service provider, when you say about restores, is this within the customer’s tenant or Veeam Cloud Connect? As restores are the highest priority they’d get the next available resource.

I personally wouldn’t want to have these artificial limits as they invite the risk of issues. Instead I’d rather have proxy/repo settings to reserve a certain number of tasks for restore if anything so you’ve almost got a “break glass” way of always processing a recovery immediately (as a Veeam Cloud Connect Provider).

[ryan.christensen]

I'm referring to on the client side in the Veeam console itself. Backup copy job is running and say it's right after the job starts. It's maxing out the proxy/repo and using all available tasks to process any many disks at once as possible. If you then start a restore you are waiting for one of those tasks on the backup copy job to complete before the restore begins. This seems crazy to me. It's like no one has tested this scenario and only does restores when all other tasks are idle. Same thing if you have a single proxy and hourly backup jobs. If the backup copy happens to be using all available task slots, the hourly job is delayed even if it's a high priority job. Yes, it will get the next available task but there's no guarantee how long that will take.

[micoolpaul]

Hi Ryan,

I see what you mean, do you think that a per VM or per Job limitation would be best, or actually being able to reserve x%/yNumber of slots for recovery at a proxy/repo level?

[ryan.christensen]

I think either would work. If it's per job that gives more flexibility. It could be defaulted to disabled in the advanced storage settings or if you want less items for support to troubleshoot you could reserve slots on the proxy itself for restores. Personally, I'd like to see it per job in advanced settings so I can have backup copy jobs limited to to tasks each, leaving the most tasks open for backup jobs.

[ryan.christensen]

Just wanted to say that I recently ran into this again. I'm surprised this hasn't been a bigger feature request until now.

Situation:

We were restoring a workstation to new hardware. We wanted to take a current restore point of that workstation. Easiest way to do this is to select the workstations job and hit run. That's fast but also triggers 10 other workstations to run as well.

Few minutes later we go to restore that workstation and the repository is still busy with the other workstation backups. Restore was delayed by about 10 minutes waiting for a task slot.

Not critical but it would be nice to always have one or two task slots reserved for restores.

[veremin]

Sounds unexpected, restore tasks should always have a priority over backup ones. So might be worth reaching our support team.

Thanks!

[DaStivi]

sure if there are for example backup jobs running, and you'll start an recovery operation, the recovery operation would start, but only when one of the "all-used" backup tasks slots get finished... it wouldn't start the thex backup operation with that free slot, instead it would run the restore operation.. but thats exactly the issue here.. imaging backing up many huge disks of ah VM, if all slots are used, and you'll start an recovery you need to wait that at least one of the backup tasks (vm-disk, whatever) gets finished...

this needs to be addressed sooner or later if you go want to get more into bigger enterprise workloads!

[foggy]

Hi Stephan, tasks reservation for restores looks reasonable, we will consider this request internally. Thanks!