Longstanding Scheduling issue around overlapping days

[ashleyw]

hi,

Currently jobs in a job chain need to have their ActiveFull schedule set independently of each other.
The problem with this is that if an ActiveFull job (or any other job type for that matter) runs into the following day then the scheduling needs to be changed.
e.g;
job-A: 6 hours: ActiveFull 1st Friday month, scheduled at 7pm every night
job-B: 3 hours: ActiveFull 1st Friday month, chained to run after job-A
job-C: 5 hours: ActiveFull 1st Friday month, chained to run after job-B

In this scenario ActiveFulls never happen on job-B or job-C as they always start on the Saturday (as job-A only finishes on the Saturday and job-B and job-C are chained after job-A).
Once we know the run times of an ActiveFull in a job chain then we can figure out the starting days and manually adjust the start day accordingly, but our backup workloads are quite dynamic which makes this hard.

For reliability what we'd really like to see an option on each job.
Backup job>Advanced Settings>Backup>ChainedBehavior
Synthetic/ActiveFull Behavior determined by ChainedJob Parent? Y/N (default N)

If Y then if a backup chain was started on the first job, and this results in a synthetic/active full, the the current job will also be a synthetic/active full.

If this option could be added to the UI then this would help to improve the scheduling reliability for common scenarios where a backup job flows into the following day and multiple jobs are chained together.

thanks
Ashley

[haslund]

What is the reason for the chained jobs?

If you are concerned about load issues, it would be recommended to simply leverage the load controls implemented in the product instead of daisy chaining the jobs. For example, backup I/O control or simply managing the number of proxy/repository tasks.

[ashleyw]

thanks, Chained jobs have been a product feature for as long as I can remember.
We already leverage the various controls for managing the workload parallelism.
We used to have a configuration where we'd trigger all the jobs off at 5 minute intervals and then managed the scheduling using the controls as you mentioned, but we found this wasn't practical and caused reliability issues for us (our workload size is about 500 development VMs split over 8 backup jobs), particularly on job retries.

When we switched to chained jobs reliability was more consistent and the behavior was more predictable.
We would like to continue to use chained jobs in our situation so if Veeam is unable to make changes to the scheduling, then we'll just need to work around the process I guess by adjusting the start days on each job (for the active fulls) to prevent day overlap from impacting on the schedules.

I guess we could trigger the behavior we want by triggering the jobs using "Start-VBRJob" cmdlet but then that sort of defeats the purpose of having UI based scheduler.

[mjr.epicfail]

Im still curious at why you would use daisychaining, its a headache and a disaster waiting to happen.
If one job stalls the next wont start is an example i've seen dozens of times. This is the same for the issue you are having with the active full not starting.

Veeam software it self can process parallel backups just fine if you don't tweak it too much.
500 vm over 8 jobs = ~62 vm's per job. Bit high but doable.

Is your back-upstorage quick enough to do syntethic fulls? or is that the reason for the active fulls.
Maybe we can help thing of a different setup If you can clarify a bit more what your requirements are.

[ashleyw]

thanks, IMHO, if job chaining no longer has relevance then it should be removed form the product, otherwise in the case of chained jobs then any job in the chain should have its run type determined by the head of the chain unless its overridden by the job stage.
We do use block clone file system (XFS) so we do leverage synthetic fulls etc and this gives predictable fantastic performance improvements, but for reasons such as the recent CBT scare on 8 update 2, there is still merit to run regular active full backups as part of the backup strategy.
When we started using Veeam over a decade ago, because of the de-dupe job file based architecture, the performance slowed down as the jobs got larger (due to the size of the de-dupe tables), and the only solution at the time was to split the jobs.
We now use per VM de-duplication option, so we sacrifice some of the de-dupe benefits to gain consistent performance.

Our workloads are quite dynamic (being a dev shop) so no solution is going to be a perfect for us.
I was just hoping a relatively small UI change to the job schedulers would help many people utilising job chains have a more predictable experience with the product.