Backup Repository concurrent tasks not used properly

[LKLD]

Hey guys, I hope you can help me 'cause I am a little confused:

I have limited the maximum number of concurrent tasks on my backup repository to 4 (this was necessary to resolve another issue). We run daily incremental backups for our VMs, grouped into several backup jobs. Additionally, we have a weekly GFS Backup-to-Tape job.

Issue:
Currently, the Backup-to-Tape job is running and processing one of the backup jobs (copying data from the repository to tape). Meanwhile, the daily incremental backup jobs are also scheduled to run, but they all remain in a “pending” state with the message “Resource not ready: backup repository”.

The job currently being processed by the Backup-to-Tape task is stuck at 0% with the status “initializing storage”, and no progress is made.

Why is he only trying to do the daily backup on the job, which is currently processed by the Backup To Tape Job; and all the other jobs are just "pending". As far as I understand, this should only be 1 task for the repository; so why are all the other Jobs waiting for the repository? According to my settings, up to 4 concurrent tasks are allowed on the repository, so why does it seem that just this one Backup-to-Tape job is blocking the whole repository?

In Veeam ONE, the repository shows “4 running tasks”, but I can’t see which jobs or tasks exactly are occupying these slots. How can I find out which tasks are currently using the repository slots?

Is there a way to understand what Veeam counts as a “task”? I want to know why it thinks all 4 slots are already in use even though it looks like only one Backup-to-Tape job is running and processing just one VM at the moment.

Another thing I don’t understand:
If the daily backup job currently processed by the Backup-to-Tape job is effectively blocked (stuck on “initializing storage”), why doesn’t Veeam pause that job and instead start processing other daily backup jobs that are waiting? It seems inefficient that Veeam keeps trying to work on the blocked job instead of using available task slots to process other backup jobs that aren’t blocked.

I’d really appreciate any insights or experiences related to this behavior!

[david.domask]

Hi LKLD,

Indeed I think there's a bit of confusion here on how Concurrent Tasks work.

The number of tasks that Veeam Backup & Replication creates during data protection or disaster recovery jobs depends on the type of backup chains stored on the backup repository:

For regular backup chains, Veeam Backup & Replication creates 1 task per job.
For per-machine backup chains, Veeam Backup & Replication creates 1 task per every VM disk (that is, a disk of a VM added to the job).

Synthetic operations performed in the backup repository (such as synthetic full backup, backup files merge and transformation) are also regarded as tasks. The number of tasks performed during these operations also depends on the type of backup chains stored on the backup repository:

For regular backup chains, Veeam Backup & Replication creates 1 task per job.
For per-machine backup chains, Veeam Backup & Replication creates 1 task per every VM chain (that is, every VM added to the job).

So depending on how you have the repository configured, a task is either a single job (legacy configuration) OR per VM disk for (true) per-VM backups (current default).

There are unfortunately a few variables that need to be checked to give a complete answer on why your jobs are queuing at the moment, such as the parallel processing settings for the media pool, number of drives, other activities that may be occurring, etc.

It would be best to open a Support Case and allow support to review the logs from the Tape and Daily VM Backup job in order to properly review your configuration and understand the reason for the queuing. Use the 1st radio button to export logs from jobs and ctrl+click to select the tape and VM job. That should be enough for Support to start with.

If the daily backup job currently processed by the Backup-to-Tape job is effectively blocked (stuck on “initializing storage”), why doesn’t Veeam pause that job and instead start processing other daily backup jobs that are waiting? It seems inefficient that Veeam keeps trying to work on the blocked job instead of using available task slots to process other backup jobs that aren’t blocked.

The Scheduler follows this Job Priority logic, and if the job priorities are equal based on the first two checks, StartTime is used to determine priority.

Additionally, I'm guessing you have the option to Prevent this Job from being interrupted enabled; by default, if a Tape job has locked a resource a primary backup requires, the tape job will abort and surrender its lock to the primary job. With this option enabled, the primary job queues.

As mentioned before, a few too many variables to give you a complete answer on why in your environment the jobs are queuing in an undesirable way, so please do proceed with a Support case and let Support review the behavior.

Please share your case number once created. Thanks!