Job Contention - Backup Copy Jobs

[mcsmithSOP]

I'm trying to set it up so that my most important backups get shipped off-site first. Due to limited bandwidth, the on-site backups finish first so the off-site backup copy jobs get queued.

I've set the copy job intervals on the less important jobs to be after the important jobs and that works in the sense that my most important backups (which run last on-site) start shipping off-site before the less important jobs. I've set the repositories on both sides to only process 3 jobs at a time. (It was 2 and from a speed perspective was faster getting everything off-site, so I'll probably switch it back if I can.)

What I'm finding though is that when a job finishes transferring the data, the next queued job starts transferring right away, locking out the first job from doing its file merge.

I'm ending up with one or two jobs sitting at 99% waiting for resources to do the merge while other jobs are transferring.

Is there any way I can prevent a queued job from starting to transfer until a previous job is fully complete (i.e. data transferred and files merged)?

[foggy]

Jason, considering your concern to get backups offsite asap, I'd say that the way it works is inline with your requirements. You're getting data transfer tasks completed first (new restore points created and ready for restore) and then retention applies when repository slots become available. Why do you want to prevent this behavior?

[mcsmithSOP]

My preference (and to be honest, expectation) would be that the job would fully complete (i.e. take the 5 minutes to merge the oldest increment) rather than sit at 99% for 1-2 hours waiting for the next job to transfer its data.

However, as long as the backup chain is in state where it can be used for recovery during the period when the newest increment has been copied but the oldest increment has not yet been merged, it's fine. (From a data standpoint.)

My concern is that let's say something happens to the network during that 1-2 hour period and the job ends up failing, do I lose the ability to recover from the most recent backup?

[foggy]

My preference (and to be honest, expectation) would be that the job would fully complete (i.e. take the 5 minutes to merge the oldest increment) rather than sit at 99% for 1-2 hours waiting for the next job to transfer its data.

The fact is it could be far longer than 5 minutes, depending on the storage and the increment size, since merge is a heavy random I/O process.

My concern is that let's say something happens to the network during that 1-2 hour period and the job ends up failing, do I lose the ability to recover from the most recent backup?

No, you will still have the ability to recover.

[mcsmithSOP]

Thanks. I was most concerned about losing the ability to restore while it was waiting for resources to due to the merge.