Job Chaining

[nunciate]

I read the community digest today and saw where you all are planning to get rid of job chaining. I assume you are talking about the scheduling feature that allows you to start a job after another job finishes.

I really hope you don't decide to get rid of this feature. I use it to chain my tape out jobs to my backup jobs. One of the features missing from our previous backup application was this ability and it was one of the factors for us to move to Veeam. I knew going in that this feature could cause trouble if I didn't schedule my backup jobs carefully. I didn't want to have a lot of reverse incremental jobs running to disk at the same time my tape backups were going on. I did have to space out my backup jobs to prevent this. We have almost 40Tb worth of full & reverse incremental files on this server and quite a bit of changed data backed up weekly. We don't have any issues chaining the jobs. I love that my tape backup starts right after my backup jobs completes. It saves time because it is impossible to predict exactly when the backup job will finish. Especially on the weekends when we do synthetic full backups. If other folks out there don't have the forethought to properly schedule their jobs please don't punish the rest of us by removing a useful feature.

[guillermo.lozano]

Hi Veeam, I'v just read the digest too.
I didn't found the post that Gostev was pointing out, so I'll post my scenario here:

My big usage for job chaining is post script commands to call the powershell snap-in in a remote VB&R Server that handles my replication jobs.
One Backup Job is completed & that job post-script action launches a cmd that calls a remote powershell script that starts a replication job for the same VM's on a remote VB&R server (Offsite DRP Location).

It would be awesome if you add support to control job execution of a second or more VB&R Servers from a central VB&R Server.

Thank you.
Guillermo.-

[sketchy]

My response is per the request of Gostev to provide feedback on the apparent problematic feature of job chaining.

I have to say that is certainly an interesting response from the Veeam engineering team. I’m sure many of us would like to know more context and detail around the internal conversations. I wish to offer a slightly different perspective.

I do use the job chaining feature. While I’m not completely married to it, I do find it to be very useful, and have not had any issues with it. I do work for a Software Development company, and am very familiar with how products are developed, and how features are included.

I’m guessing that the “job chaining feature” was added not because Veeam couldn’t think of anything else to add, but rather, it was a part of a “feature backlog” that derived from customer requirements. I suspect that customers had a specific need, and job chaining was the “solution” to address the need. Well, clearly, removing job chaining does NOT remove a specific customer need. Permanently removing a feature due to reasons provided sounds almost Dilbert-esque “Congratulations, we removed all need for Technical support, because we removed all product features” Humorous perhaps, but a cautionary tale for all Software Development companies. If a feature is not quite ready for prime-time, then it is totally understandable to pull it back temporarily, but permanently? That is something to be do with extreme caution.

If there is a shortcoming with job chaining, then perhaps it needs to be refined and fortified. Gostev’s comments were somewhat vague on what the partners and customers were actually experiencing. I would love to know more specifics (please share with the community if you can). For instance, why do some customers have problems with it, and others do not? And how do the partners play in this? Was it configuration issues on behalf of the customer, the partner, or both? Can we agree that poor performance and behavior can exist on almost any feature if not used properly? Then if that is the case, perhaps there are better fail-safe mechanisms to make the job chaining more robust, or prevent serialized failures from occurring. Considering the default concurrent limits on job processing, I am struggling to imagine a scenario in which one environment went from a 12 hour backup window with chaining to a 4 hour window without chaining. More detail on this would be helpful.

Good Software Development is all about understanding the “what” from the customer. Once that is understood, Veeam’s job is to then determine the “how.” Henry Ford once said, “If I listened to the customer, I would have built a faster horse” This showcases the “what” versus the “how.” So perhaps what needs to be reviewed are the customer scenarios and workflows that drove the need for the customers to ask for “job chaining” Then Veeam can look at this with a fresh set of eyes and come up with a great solution - whatever you choose to call it.

I hope you find my comments useful. They are written with the utmost respect for the team at Veeam, and the great products they produce.

- Pete
http://vmpete.com
@vmpete

[snarlywino]

I would love to hear more about why job chaining is bad and what the pitfalls have been. I have been using this feature to run my entire backup set daily with no issue since upgrading to v7 upon its release. Like the OP stated, this saves time and minimizes the backup window. I think that job chaining is a valuable feature that many of your customers use great success and I would have to see the masses punished for the failings of the few.

That being said, I do not have the specific requirements of the OP so I will experiment with scheduling by backups all at once and letting the load balancing work to see if it reduces my backup window as Gostev suggested.

[lp@albersdruck.de]

I do use job chaining as well.

Two reasons:
Reason 1 is that I just have to change the "starting point" to move/change my backup window.
Reason 2 is that I did this setup before backup copy jobs were introduced and I needed todays full backup to be taken off-site, and for that I use reverse incremental and then a simple file copy script.

I do want to experiment a bit with backup copy jobs to move from reverse incremental to incremental+synthetic full and then copy job to external storage in the future, though.

[LaurentZ]

I do use job chaining for Exchange 20123 backup.
With this option, I can control that each dag node is backed up and databases are not moving during VMware snapshot remove before backing up another node.
My other problem is that I don't have enough empty storage space for 4 LeftHand snapshots in the same time. With this option, I can run storage snapshot one after the other.

[veremin]

My big usage for job chaining is post script commands to call the powershell snap-in in a remote VB&R Server that handles my replication jobs.

Hi, Guillermo, may I ask you how job chaining helps you to use post-job activity? Thanks.

[yizhar]

Hi.

I would like to suggest a simple and effective solution for job chaining.

To Veeam stuff - please consider this feature request.

The needed feature is "Job Grouping".

Create several jobs as needed, without schedule.

Create a single group of jobs, the group properties will have the following parameters:

List of jobs.

An option (per group) for the admin to choose if the jobs will run in sequence one after the other, or parallel (inteligent load balancing) - It is important to let us choose what is best for us. Me for example prefer running in sequence when I have enough time (longer backup window but less stress on production storage).

A schedule.

An option for summary email report - A table with list of jobs, result, date/time, duration, GB.
(The admin can choose to get the summary, and/or per job report).

This simple to implement feature will be very useful for most of us.

Yizhar

[CharlesGillanders]

Gostev had a request in this weeks community forums digest to post any use case for daisy chaining. We use it for some of our jobs so here's our case for keeping the functionality...

We have a linux pacemaker HA cluster that runs in VMware and supplies NFS service to some other systems that need to see a single shared NFS accessible folder hierarchy (these other systems are running a very proprietary application).

We use a pre-freeze-script and a post-thaw-script to put the HA nodes into standby before taking the snapshot and then put the node back into online mode afterwards. This ensures that taking the snapshot doesn't trigger a pacemaker HA heartbeat failure - without the scripts the brief timeout caused by the snapshot is enough to make pacemaker trigger it's fencing functionality which essentially does a hard reset on the VM that's being backed up, this can obviously cause problems.....

Because the scripts force the node being backed up into an offline state - it's crucial that the other node doesn't also simultaneously get backed up - this would result in the whole cluster going offline which would cause our application to crash.

We therefore use daisy chaining to ensure that the VMs are backed up sequentially and not in parallel, we actually have four jobs that are daisy-chained together - Backup VM1, Backup VM2, Replicate VM1 and finally Replicate VM2. We aren't worried about optimum performance for these jobs we are far more concerned about making sure the NFS cluster stays up and running right through all the backups and replications.

All our other backup jobs on all our other systems are run in parallel and we leave the scheduling up to Veeam, it's just these four where the order of and serial nature of the jobs are crucial to us.

[dannythake]

Hi.

I would like to suggest a simple and effective solution for job chaining.

To Veeam stuff - please consider this feature request.

The needed feature is "Job Grouping".

Create several jobs as needed, without schedule.

Create a single group of jobs, the group properties will have the following parameters:

List of jobs.

An option (per group) for the admin to choose if the jobs will run in sequence one after the other, or parallel (inteligent load balancing) - It is important to let us choose what is best for us. Me for example prefer running in sequence when I have enough time (longer backup window but less stress on production storage).

A schedule.

An option for summary email report - A table with list of jobs, result, date/time, duration, GB.
(The admin can choose to get the summary, and/or per job report).

This simple to implement feature will be very useful for most of us.

Yizhar

This a nearly perfect - exactly what we would like to see.

We use job chaining at we are a 24 hour operations, so we do this in order to minimize stress on Production storage

[gdstark]

Regarding job chaining, leave it the hell alone. It works. I want my jobs to run concurrently and in the order I chose. If you have another way of doing this, make it clear before threatening to remove the future.

[dproffer]

I have a relatively small environment that I am backing up using Veeam. I have around two dozen VMs with about 1.75 TB of data spread between them. I use job chaining to simplify scheduling. In my mind, I was creating efficiency by not stressing the production storage or backup storage.

I am not married to the feature one way or another. It has not caused me any problems that would not have already existed had I been scheduling each job myself.

The thing that struck me most was the assertion by Gostev that I should turn off chaining and "starting all jobs at the same time and letting the intelligent load balancing manage the required concurrency took the backup window down dramatically in every deployment".

Really??? Have all of my jobs kick off at the same time? Sure seems like I will be creating an IO problem at the very least and resource problems all throughout the infrastructure at most.

[sisteminformativi]

I am as well posting with regards to the request from Gostev about daisy chaining of jobs.

One of the main reason we use it is to be sure that a tape job for offsiting backups starts AFTER all backup to disk jobs, and this is achieved with daisy chaining jobs, and obviously putting the tape jobs at the end of the chain.

Another reason, is that for some application landscapes, we want to make sure we back up things in the right order: for example, we have a large application server server pool where 1 database is used, and then there are 15 systems (iis, jboss, java, etc. etc.), and we use chaining to make sure we save data in the right order (the DB first, application servers following, front-end web servers, and so on).

All of this because we are never sure about how much time every job takes, and chaining helps us preventing to schedule things with at least 30 minutes interval between each other, especially on the weekends when we do full backups.

If anybody has a better solution to satisfy these requirements, I am good with pulling out the feature, otherwise we are more than interested in keeping the feature.

Is it possible to know instead which are the reasons why chaining is not recommended ?

Thanks.

[cstemaly]

I absolutely loved the job chaning feature when it was introduced, and I really hope that you DO NOT take it away.

I have 5 backup jobs. These jobs need to be split up in order to add the jobs to SureBackup jobs that utilize different network settings to test the backups.
Also, I find it easier to change the start time of the jobs - with chaining I only have to change one time.

Before job chaining came along, I used Powershell scripts to run the subsequent job. I'll go back to this if I have to, but keeping track of it in the GUI is much easier.....

As others have requested, I would also like to know WHY chaining is bad: Is it only bad under certain circumstances? What could go wrong?

[dspjones]

Guess I'll add my 2 cents....

I use chaining as well. Our production vm cluster starts the ball rolling and once it finishes the exchange environment kicks off. Works 100% of the time. Never had a problem. Please do not remove it. If anything enhance it or fix what you think is wrong with it but please don't remove it. I currently back up our environment 4 times per day and every once in a while the change rate for the non exchange environment may cause the backup window to go a little longer than normal. It is really nice that I don't have to set specific start times for the next job. The first job can take whatever time it needs to complete and then the other jobs run after and don't step over each other. In the past without chaining I had the occasional issue of other jobs starting while one was already running and having to wait for the first to finish and release the proxy servers so it could use them. Chaining resolves that for us.

[arsprod]

I too saw this in the email digest and use job chaining for both backups and replicas and would like to know the downsides. I've been using it for quite awhile (when you used to have to do it as a post job activity in the advanced tab) and was thrilled when Veeam added it as a scheduling option. I use it for 11 replica runs. What gives?

[lobo519]

I am also using it for chaining replica jobs - I would add all the replicas to one job but I don't want a problem with one replica to stop/slow them all.

[jtupeck]

I would like to know more about why this feature is not recommended now, as well. I hadn't been using the job chaining features until just this last week. The reason for my usage is to chain my jobs together so they didnt sit in a waiting status for hours and hours waiting for backup resources. This skews the actual amount of time that each job reports it took to run, data which is important in my opinion. So now, I start one job and chain them together. My jobs start at 6pm, run in series and report the appropriate amount of time each one took. Is this not what it was meant for?

[tsightler]

As the SA that started the internal discussion, I guess I'll chime in here with a little background. I think one thing that's important to remember is that the native ability to chain jobs is a relatively new feature added relatively recently to the product. Prior to supporting it directly in the scheduler it was of course possible to chain jobs using simple post-job activity (some people used scripts, but it was actually as easy as adding a single line to the post-job that called the next job). Overall, I always warned against chaining, but it wasn't a big deal, people that needed it figured out how to do it via post-job and in many cases provided feedback as to why they were doing it.

However, there was one big side effect to having chaining available natively in the scheduler GUI, people started using it because that how they think Veeam is supposed to work if there are mulitple jobs, meaning they are not using one of Veeam's most powerful features, the automatic scheduling of job resources. This became even more important in V7 with parallel processing since jobs will not just all run in parallel anyway, but rather attempt to leverage the tasks available.

It's certainly a little harsh to just outright say they are "bad", but the issues we see are mostly:

1. Inconsistent handling of error cases (for example a tape drive offline keeping all further jobs from executing, meaning no backups at all, retries holding up jobs, etc.).
2. Poor use of available backup resources (long running VM in one job leaving tasks slots available for use but with no tasks, or a VM taking a long time to remove a snapshot, leading to long backup windows).
3. Unpredictable behavior of synthetic processing due to jobs starting after midnight
4. Administrative errors when users disable jobs or add VMs to other jobs leading to unexpected consequences in the scheduling
5. Unpredictable behavior with backup copy due to inability to coincide interval with job start time since job start time varies for every job each night.
6. Chaining doesn't scale. The larger the environment the more difficult it is to know exactly when each job should be run and how many resources should be assigned and which jobs should get new VMs or when new jobs should be added.
7. Some repositories perform better with mulitple jobs as that's the only way to get multiple I/O streams.

Admittedly, some of these shortcomings have been addressed via various error handling methods of the chaining schedule, yet time and again I get customers that are complaining about their backup window only to find they have a chain of jobs and a long running VM or snapshot removal holding up everything.

In almost every case where I've run across a customer that is using job chaining and is having problems with the backup window I've asked why they are using chaining and the answer is "Because I saw it in the GUI and I had mulitple jobs so I assumed I had to". They don't even consider the possibility of scheduling jobs to run together (or with slight offset if they want to control the order) and letting the automatic resource allocation handle when the jobs run. When I explain how task management works in V7 a lightbulb goes off and they realize that they didn't need chaining at all.

So perhaps instead of saying "Chaining is bad" I should say "Chaining for no reason is bad". If an admin understands how automatic resource management works, but still chooses to chain a job, then I really don't have an issue with that. It's the choice to use chaining over automatic resource management because they see chaining in the GUI and don't know that Veeam is capable of automatically scheduling the resources without chaining. When the chaining option wasn't sitting there in the GUI, people who needed chaining would still seek out how to do it.

Even in this thread it appears that many people are using chaining because they don't realize that the automatic scheduler can already do things like automatically monitor for backup job completion to start copy to tape, chaining is not required to do this. This implies to me that our GUI is not doing a good enough job on emphasizing the features of the automatic scheduler options. In other cases it appears that chaining is being used as a "fix" for something that the automatic scheduler can't quite accomplish natively so it seems like there are plenty of things that could be improved there.

But that's the reason we ask for feedback. I knew this would be a hot topic because chaining was a highly requested feature, and I wasn't even opposed to it's inclusion because I could see several use cases, but now that it's in the GUI people are using it because they don't know about the automatic scheduling capabilities. Keep the feedback coming!

[jtupeck]

tsightler - Fantastic response and many great things to keep in mind that I hadn't thought of when implementing it to test performance on my Sunology CIFS repository - pertaining to #7 in your list.

[zoltank]

I use job chaining and it works very well for me and our environment, and has ever since we upgraded to v.7. In fact, I was one of the customers asking for GUI based job chaining in the first place.

It sounds to me like the problem here isn't a product feature issue, but rather a user education issue, and it's frustrating that there's talk of removing a very useful feature due to uneducated users. If you absolutely have to go that direction, then at least please make it a feature which is off by default but can be turned on

[Gostev]

Absolutely agree, I was never going to remove this feature to start with (there are a few corner cases when it comes useful), but I wanted to throw this in to provoke people to really start thinking about this, and posting their usage scenarios. As you can see, this totally worked

However, it is still fair to say that 90% of the users use this capability for no good reason. For example:

Have all of my jobs kick off at the same time? Sure seems like I will be creating an IO problem at the very least and resource problems all throughout the infrastructure at most.

This comment is perfect demonstration why ability to chain jobs is hurting most of our customers. Job chaining is easy to understand, so everyone just goes ahead with that, not realizing that the B&R is far more advanced than your typical backup product from the past, and is able to maintain the required concurrency at the task scheduler level. Not that I blame anyone for that, because in the end it is quite an advanced concept to grasp indeed.

And sorry for not commenting here earlier, I was tied up in the meetings for the whole day today. But Tom has provided the perfect response above anyway.

[nholman]

So are you saying we should schedule all our jobs for the same time - Veeam will then backup each vm as it sees fit?
Do we need set our Direct SAN mode to run more than 2 concurrent tasks to achieve what you are saying?

[Gostev]

Same time, or slightly different if you care about priority (some VMs being backed up before other), and yes Veeam will then backup each vm as it sees fit, using the available proxy and repository task slots (up to maximum concurrency you have allowed). For example, you can still have 1 task processed at a time if you want to (effectively same as chaining), by limiting your backup repository to 1 concurrent task.

I don't understand the last sentence in your post though.

[Yuki]

Gostev, a few comments:
1) your example shows that people may be using the product incorrectly based on the lack of information about proxies and job processing. Basically, they cant reliably predict the behavior because they dont know details (a black box to them, even if it is just a perception), causing them to play it safe. You want people to just trust Veeam that you have already implemented logic to handle everything in the most efficient way.

2) we use chaining at clients as well. The primary reason for this is to separate VMs that may be affected by different scheduled maintenance windows or other potential downtime (like dealing with snapshot issues on 20TB vms, etc). We can't stop backups and replication of all systems, but don't want to receiving notifications of failures during scheduled/unscheduled maintenance. Give us ability to pause/disable/delay processing of VMs in existing jobs without adding or removing them and give us ability to enable/disable notifications per VM for maintenance reasons.

3) Chaining is useful for organizational reasons as well, with no technical reason for it. One of our clients wants to see all SQL servers complete first, then app servers, etc. They don't care about doing it faster, then want order and a system behavior they can understand and predict.

[scd]

I am using chaining as we were having issues with some of our offsite replicas kicking off before all of the backups were finished which pushed those backups into the next morning. We previously used the Post Job Activity option. I would also agree with jtupeck that it is important to see the actual time the job ran; info I wish was available in the main program window as job start and end times.

tsightler -

5. Unpredictable behavior with backup copy due to inability to coincide interval with job start time since job start time varies for every job each night.

This is precisely why I think backup jobs should work like normal jobs and be able to be chained to kick off after the backup is finished. I still don't grasp the reasoning behind the 24 hour backup windows and start time of backup copy jobs as I don't see what problem that feature was created to address.

[nunciate]

Here is how I use chaining:
I break up my servers based on O/S. So all W2K3 in one group, WS08 in another and so on. This is so I get the best deduplication ratio across similar operating systems.
I also don't want my VBK files to get too big for backup jobs where I have many servers in the job and I don't want more than 25 server per backup job.
So I break things down further by putting up to 25 servers per job and try to keep the VBK to around 1Tb total or as close as I can get.

So I might end up with something like this. W2K3-Job-A, W2K3-Job-B, W2K3Job-C and so on for each O/S. I chain these together so when one completes the other starts and at the end of the chain I have the last job kick off a single tape out job for the entire series.

Jobs that have static start times like the first job in the chain get manually scheduled at different times. This is where I try to guess when the first series will finish and the next will start. It isn't always perfect but I do rely on the Veeam load balancing logic to determine if things can run. I have never had an issue with too many jobs running.

[foggy]

1) your example shows that people may be using the product incorrectly based on the lack of information about proxies and job processing. Basically, they cant reliably predict the behavior because they dont know details (a black box to them, even if it is just a perception), causing them to play it safe. You want people to just trust Veeam that you have already implemented logic to handle everything in the most efficient way.

We probably need an extensive user guide section with a thorough explanation of how resource scheduling works in Veeam B&R, covering all the aspects mentioned in this thread. There's currently not much information about that in user guide.

I am using chaining as we were having issues with some of our offsite replicas kicking off before all of the backups were finished which pushed those backups into the next morning.

As it is explained above, you do not need jobs chaining for that.

[Ian Cook]

We have used chaining in the past for running replication jobs after the backup job has completed. eg Windows 2008 server backup job runs, when it finishes the Windows 2008 server replica job kicks off, this has always been to prevent Veeam trying to replicate before its backed up. Are you saying that this is no longer required ?

We have moved all of the backup jobs to kick off at about the same time now (since V7 came out) and allow Veeam to sort their scheduling out, but still do the same with regards replication job runs, in this instance would running the replication job slightly later than the backup give us the same end result but with a much quicker run time ? or am i misunderstanding your point?

[arsprod]

I'm sure veeam is great at allocating resources but there's a hitch in this idea - maybe I'm not getting it but seems to me that once a job starts and picks a proxy it will sit with that proxy until it (the proxy) is available. I've seen jobs sit there for hours waiting for "resource not ready" - when I daisy chain it doesn't seem to happen as often.