Chained Jobs Not Starting

[nunciate]

Hoping someone has had a similar issue and can quickly direct me to an answer. I am super busy this week so I don't have the time for the back and forth with support right now.
If I can't find a quick solution here I will open a ticket later this week or next week.

We setup a new tape library in our DR facility where I already send backup copies of all my local disk data.
I cloned all of my tape jobs here in production. I have quite a few jobs and I have 4 sets of media pools to go with my 4 tape drives.

Here is how it works in production.
Set first tape job in media pool 1 to start at a specific time.
Set all other tape jobs in the same media pool in a chain starting after the scheduled job finishes.
I do this for all 4 media pools and this works fine in production and prevents tape drive locking due to multiple jobs waiting on free drives.

I simply cloned all of the jobs and changed the source to be the backup copy job and the destination to the same pool but in the library that is in the backup copy destination.
I also updated the schedule chain on each to match the new jobs I created. I have quadruple checked the chain of jobs and all is correct.
If I run the jobs manually one by one they work fine. Data is going to tape and all is well.

When my first scheduled jobs fire off they run just fine but the first chained job never starts and so none of the chains get run. Only the first schedule job in each of the 4 media pools.

[foggy]

Alan, do both Veeam B&R servers (production and DR) have similar versions? There was a change in v8 behavior regarding running chained jobs: they do not run if the parent job is started manually (which seems to be similar to your case).

[nunciate]

Yes both Veeam servers are exactly the same. They are physical HP DL380G8 servers with 16 cores and 128Gb of ram. They both have large external arrays attached to use as repositories. Almost 70Tb.
They are both running Veeam B&R 8.0 Update 2. I am aware of the manual start issue and I am not manually starting the initial jobs. I am setting the jobs to start in 5 minutes from the present and the first jobs do start but nothing happens with the remaining jobs.

I did notice when I manually started a backup job with a tape job chained after I was given the option to start the chained job after. I wonder why that wasn't added to all chained job types. Or maybe it was and it isn't working for me...

[foggy]

They are both running Veeam B&R 8.0 Update 2. I am aware of the manual start issue and I am not manually starting the initial jobs. I am setting the jobs to start in 5 minutes from the present and the first jobs do start but nothing happens with the remaining jobs.

Thanks for clarifying the issue. I recommend to let support take a look at your setup then, since it does not look as expected behavior.

I did notice when I manually started a backup job with a tape job chained after I was given the option to start the chained job after. I wonder why that wasn't added to all chained job types. Or maybe it was and it isn't working for me...

Similar prompt will be added to tape jobs as well in the next version.

[nunciate]

I will try and get a ticket open today. It is taking forever to export the logs even with it set to only pull the last 2 days.

I wonder if I am hitting some job limitation if there is one. I am not talking running jobs as I don't have that many jobs running all at once. As a matter of fact with this issue I have no other jobs running.
I break down my VMs into many jobs so that the VBK files don't get larger then 500Gb where possible.
This leads to quite a few jobs on the server though. Here is the break down. Anyone know if this could be an issue? I have noticed somewhat slow response clicking on tape jobs now that I duplicated all of them for DR.
My SQL Server is a separate VM running Windows 2012 R2 and SQL 2012 Standard.
I also rebooted the server today to see if that would help. Still seems slow clicking on tape jobs.

70 Backup Jobs (These run on schedules starting at 5:00pm and run throughout the night)
70 Backup Copy Jobs to DR (These run in continuous mode and fire off right after the backup job completes)
70 Tape Jobs for Production (4 Jobs start at 10:00pm and the others chain off of the first 4 throughout the night)
70 Tape Jobs for DR (Cloned and edited from the Production Jobs).

[foggy]

There's no fixed limitation on the number of jobs that a single Veeam B&R can manage, however, depending on the database size and available resources, I admit the console might be somewhat slow with this number of jobs.

[nunciate]

Case ID: 00938277 was opened for this. Still have not been able to get these new chained jobs to run on schedule. All of the jobs have been successfully run manually at least once. Now I am just manually running them in order when I can.

[nunciate]

It looks like there is a bug that is being worked out right now in Veeam 8. As an interim solution I did the following based on advice from support.

I created a batch file that will start each job. So that was 70 batch files I created.
The command line for starting a job can be found on the summary tab of each job when you edit it.
To do them all I ran a powershell command that dumped a list of all job settings for tape jobs.
Then I pulled out the Job Name and Job Unique ID. The Name is used for naming the batch file and the unique ID is used in the command line.
I used excel to concatenate the command line together for each job.
Then I used excel again to concatenate a command line which echoed my job start command into batch files. Only took a few minutes to generate all 70.

Once I had all my batch files I edited each job and put a post job script pointing to the next jobs startup batch file.

It isn't perfect but it works.