Job Chaining

[ethan.cain]

For those who can view cases 00526002 (w/ screenshots)

Good example of why NOT to use job chaining. In this case, the job proceeding runs Synthetic fulls on Sat at 10pm, and takes 4 hours to run, therefore the chained job that is also set for Sat never runs on a Saturday and therefore never ran a synthetic. Hit 200 restore points out of a 30 point retention policy before it ran out of space. Reverse Incremental, or chaining Replications to Backups are the only two things I see as a good/needed use, whereas the situation I noted above can come up without fully understanding the issues that can come up.

[nunciate]

I have seen the issue where I might have 3 jobs chained one starting after the other. The last one didn't run it's Friday synthetic full because it didn't get fired off until after midnight. When I notice those I simply set the synthetic full on that last job to run on Saturday instead of Friday and bingo problem fixed. You can even choose to have it run the full on Friday & Saturday because it will only run when it is triggered by the previous job completing so it will not just run on its own. If it happens to be fired on Friday night it runs its full. If it fires on Saturday morning it still catches it. It is just a matter of watching your jobs and setting them up correctly. Not really all that hard.

[scd]

Foggy - As a test, I changed the schedule for my jobs last night to unchain them and schedule them with 5 minute offsets on the start times. I DID have the same issue where the backups did not backup the VMs in the exact order I wanted and even though all of the replicas were scheduled to start after all of the backups, one replica still kicked off before the backups were finished and ruined the schedule for the night. That backup reports it took 8 hours where it normally only takes 20 minutes and it didn't snapshot the VMs until Tuesday, meaning I don't have a backup for Monday (the date matters).

[foggy]

Foggy - As a test, I changed the schedule for my jobs last night to unchain them and schedule them with 5 minute offsets on the start times. I DID have the same issue where the backups did not backup the VMs in the exact order I wanted and even though all of the replicas were scheduled to start after all of the backups, one replica still kicked off before the backups were finished and ruined the schedule for the night. That backup reports it took 8 hours where it normally only takes 20 minutes and it didn't snapshot the VMs until Tuesday, meaning I don't have a backup for Monday (the date matters).

Probably, 5 minutes was not enough in your case for all the tasks of the said backup job to be registered in the scheduler and assigned processing resources prior to replication job tasks. Resources are assigned after the job VM list is built ('Building VM list' record in the job session log), which, in its turn, requires a lot of communication with virtual infrastructure. In case of a job with large number of VMs and/or not rocket vCenter, replication job, even being started 5 minutes later, could build the VM list faster and get priority for its tasks over the backup job tasks, causing the observed behavior.

I would try to increase the interval between jobs start times.

[guillermo.lozano]

Hi, Guillermo, may I ask you how job chaining helps you to use post-job activity? Thanks.

Even though I do not use job chaining itself, I do chain jobs between VB&R servers.I took this opportunity to explain and request if it would be possible to use job chaining between servers as a developed feature.

Thank you.
Regards,
Guillermo.-

[veremin]

Got it. Thanks, Guillermo, for providing the explanation. For now, the best option will be to use Enterprise Manager in order to orchestrate the work of two backup servers. Thanks.

[ntw]

We had our jobs scheduled to start at the concurrently and "fight" for resources, but we were getting "Unknown status of async operation Another shadow copy creation is already in progress" errors (see Case 00498538). Though as a rule I don't like the idea of making two changes at once, Jerimy suggested: a) turning off "Enable parallel VM and virtual disk processing" and b) chaining our jobs instead of firing them at once. He alluded to the idea that "Enable parallel VM and virtual disk processing" and allowing jobs to fight for resources are incompatible. Nevertheless, while I'd prefer to not chain, Veeam support recommended chaining.

[ITWWT]

I will feed my results in here.
I have 35 VM's all being backed up using the daisy chaining option. The backups usually take 5 hours to run.
Yesterday I turned off the daisy chaining and started all the jobs at the same time (6pm). To my surprise all the backups finished in around 2.5 hours. This is a significant difference.
Regards
John

[arsprod]

We had our jobs scheduled to start at the concurrently and "fight" for resources, but we were getting "Unknown status of async operation Another shadow copy creation is already in progress" errors (see Case 00498538). Though as a rule I don't like the idea of making two changes at once, Jerimy suggested: a) turning off "Enable parallel VM and virtual disk processing" and b) chaining our jobs instead of firing them at once. He alluded to the idea that "Enable parallel VM and virtual disk processing" and allowing jobs to fight for resources are incompatible. Nevertheless, while I'd prefer to not chain, Veeam support recommended chaining.

Another good reason NOT to remove this feature, I've also experienced instances where support has suggested/recommended things differently from developers including chaining!

[Gostev]

Nathan and Aaron, it is important to understand that some support engineers tend to provide recommendation which only purpose is to quickly close the support case, despite it may put the product in the non-optimal configuration. In many cases, this is caused by the fact that support engineers simply do not possess the Solution Architect level knowledge of the product, at least those on lower tiers.

The correct way to treat this type of recommendations is to refuse the proposed workaround, and insist that the product is made working in your configuration (as long as you have deployed the product in the supported way). Such response from you would trigger escalation of the issue into R&D, investigation of the real issue behind your problem, and production of the hot fix addressing the real issue.

[Gostev]

I'm sure veeam is great at allocating resources but there's a hitch in this idea - maybe I'm not getting it but seems to me that once a job starts and picks a proxy it will sit with that proxy until it (the proxy) is available. I've seen jobs sit there for hours waiting for "resource not ready" - when I daisy chain it doesn't seem to happen as often.

Why do you see job sitting for hours waiting for the resources as a problem? Generally speaking, this is not an issue at all. Moreover, those last jobs in the daisy chain effectively behave the same way (sit there for hours waiting), so really there is no difference at all.

If the issue is that you want the job to start and finish processing sooner, change the start time to a few minutes before all other jobs to give it a priority. And, of course, even simpler fix would be to add more proxies or repositories, depending on what backup infrastructure resource is lacking. Or, increase max concurrent tasks value on the existing ones, if they still have some spare CPU and I/O resources.

[arsprod]

Problem I see is once a job is initiated the proxy is assigned so whether it's waiting for that proxy or VM is locked, other jobs are now waiting too. I see advantages both ways but one relies on the software. To be fair the support tech I spoke with today also said to add more proxies.

[sketchy]

As one of the "early responders" to this post, I'm interested in the continued hesitation of Veeam to recognize some of the legitimate business cases stated here. Engineering seems to have some legitimate scenarios in which it may not work well, but those in the trenches who are ultimately responsible for trying to protect their systems according to particular business requirements and constraints also have very valid reasons. Lets suppose a customer gets rid of job chaining, and lumps all of the jobs together under the stated recommendation. Based on the responses here, you can see that isn't exactly generating a lot of enthusiasm. I have a hunch one reason is because you have no idea when, or in what order a VM or group of VMs was protected. If a person has someone to answer to, that isn't going to be a fun conversation.

[tsightler]

As one of the "early responders" to this post, I'm interested in the continued hesitation of Veeam to recognize some of the legitimate business cases stated here. Engineering seems to have some legitimate scenarios in which it may not work well, but those in the trenches who are ultimately responsible for trying to protect their systems according to particular business requirements and constraints also have very valid reasons. Lets suppose a customer gets rid of job chaining, and lumps all of the jobs together under the stated recommendation. Based on the responses here, you can see that isn't exactly generating a lot of enthusiasm. I have a hunch one reason is because you have no idea when, or in what order a VM or group of VMs was protected. If a person has someone to answer to, that isn't going to be a fun conversation.

Due to travel this week I haven't been able to follow this thread as closely as I would have liked, but I will eventually be able to read each post in detail and I'll try to do a summary. However, I don't think it's correct to say that we (Veeam) haven't acknowledged there are legitimate use cases for this feature. Both Gostev and I have both stated above that we do recognize those cases.

However, this thread has also highlighted that quite a number of people are using it, not to meet business requirements, but simply because they see it and think that's what they need to do. This implies that we are not doing a good enough job in the GUI or in our guides explaining how job resource scheduling works.

I would say that even your example case somewhat continues to show this. If the customer in your example has 5 jobs, all chained, with the first job starting at 6PM, they really only know when the first one starts anyway, there's no guarantee what time the other 4 jobs would start, other than "after the first one". Heck, if something went really wrong with the first job perhaps those other jobs would never start. Perhaps a more common issue would be a Storage vMotion operation on a large VM that's currently processed by the first job. Now that night the VM can't use CBT and takes 6 hours to backup, those other jobs will not start for 6 hours while proxy resources sit around idle.

If the same customer has those same 5 jobs, but they are set without chaining to start 6:00, 6:01, 6:02, 6:03, and 6:04, they will still all start in order, but as long as the 1st job is using all of the resources jobs 2-5 will have to wait. However, as soon as the 1st job can no longer fully utilize the available proxy resources, the second job will be allowed to start using those available resources. If job 1 has a sudden long running VM, it won't keep jobs 2-5 from using the resources that become available and running concurrently.

The likely scenario in the second case is that the backup window will be significantly reduced (typical is around 50%), while still backing up the VMs in the order required, although with a little overlap across jobs to keep the resources fully utilized. It's hard for me to see how this isn't actually meeting business requirements in the same way as chaining, although I'm interested to hear your reply.

Thanks again for the feedback.

[guillermo.lozano]

Got it. Thanks, Guillermo, for providing the explanation. For now, the best option will be to use Enterprise Manager in order to orchestrate the work of two backup servers. Thanks.

You're welcome.

Where can I find more info to orchestrate with enterprise manager?
Would it allow to chain jobs between servers without using powershell as a post job activity?

Thank you!
Guillermo.-

[emachabert]

After installing hundreds of Veeam B&R servers, I can confirm:

- Job chaining is useful in very few cases (most of them explained above), but these cases are real so don't remove that feature.
- Job chaining, when misused, is a sort of nuclear weapon on your backup infrastructure resulting in real and important problems

Most of the time, scheduler logic coupled with loadbalancing logic is all you need to reduce your backup window at its minimum while preserving processing order.

<Feature Request>
It would be very useful to have the choice of enabling or disabling chaining when starting a job manualy (without editing the chain). When you test your jobs for guest processing (after adding a VM). I think many people posting here know what I mean.
</Feature Request>

[Gostev]

Agreed.

<Feature Request>
It would be very useful to have the choice of enabling or disabling chaining when starting a job manualy (without editing the chain). When you test your jobs for guest processing (after adding a VM). I think many people posting here know what I mean.
</Feature Request>

That's a messy way to test your guest credentials... sounds almost like buying an airplane ticket just to get that on-board meal? I'd rather have devs spend the same time implementing such test as a part of the wizard.

[lp@albersdruck.de]

Nowhere did he say he wanted to test credentials. He wanted to test guest processing, which could mean pre-freeze scripts and whatnot.

[Gostev]

I meant the same thing. Bad or lacking credentials is simply by far the primary cause for guest processing failures... anyhow, the test would perform full cycle anyway, triggering the exact same guest processing function as during the normal job run.

[veremin]

Where can I find more info to orchestrate with enterprise manager?

More infromation regarding this component can be found in the corresponding section of our Help Center.

Would it allow to chain jobs between servers without using powershell as a post job activity?

No, it wouldn't. However, what it will give is a single point of view, using which you will be able to manage backup/replication jobs of two backup servers. This is pretty useful tool for those who have more than one backup servers.

Thanks.

[emachabert]

I agree with you Anton, but for the moment there is only that "messy way"

That testing feature would be really useful and could save a lot of time. I hope it will exist someday.

[evander]

I have been reading this thread with interest and in all the examples and explanations I have yet to understand if the logic behind running the different backups at the same time applies to the same VMs or different, or if that even matters?

For example:

Assume I have two jobs:
Job1 will backup 20 VMs to local site.
Job2 will backup THE SAME VM's to remote site. (100mb link)

If the backup jobs are running at the same time, will it be clever enough to not snapshot a machine that is already in snapshot mode and being backed up? I would prefer my machines to not run multiple snapshots as I have had plenty snapshot issues in the past.
So my question is, is it intelligent on resources only or also on which VM is currently being backed up/Snapshotted. (Is that a word? )
A daisy chain would ensure they would never be backed up at the same time.

[Vitaliy S.]

If the backup jobs are running at the same time, will it be clever enough to not snapshot a machine that is already in snapshot mode and being backed up? I would prefer my machines to not run multiple snapshots as I have had plenty snapshot issues in the past.

Yes, Veeam B&R server is clever enough. VMs that are processed by one job, will be skipped by the second job and will start processing once first job releases those VMs.

Reading back through your setup and requirements of storing VMs in the offsite location, I would recommend using backup copy jobs. Backup copy jobs do not "touch" production VMs at all, they retrieve VM data from the backup files produced by the main backup job.

[evander]

Yes, Veeam B&R server is clever enough. VMs that are processed by one job, will be skipped by the second job and will start processing once first job releases those VMs.

You mention "one job" will be skipped. Do you mean one VM in the job or the whole job?
If it is indeed "One job" that is skipped as opposed to one VM that is skipped, isn't that effectively chaining anyway?
Apologies if I am not understanding you correctly.

Reading back through your setup and requirements of storing VMs in the offsite location, I would recommend using backup copy jobs. Backup copy jobs do not "touch" production VMs at all, they retrieve VM data from the backup files produced by the main backup job

Thanks for the tip Vitaliy. I had intended to look at the Copy job when I plan to upgrade from v6.5 to v7 sometime soon, although I like the idea of separate backups (as opposed to copy jobs) as it gives me options to schedule different retention periods and allows me to manage my disk space etc. I also like the idea of two separate "successful backups"

[Vitaliy S.]

You mention "one job" will be skipped. Do you mean one VM in the job or the whole job?

Just the VM in the job, not the entire job.

Thanks for the tip Vitaliy. I had intended to look at the Copy job when I plan to upgrade from v6.5 to v7 sometime soon, although I like the idea of separate backups (as opposed to copy jobs) as it gives me options to schedule different retention periods and allows me to manage my disk space etc. I also like the idea of two separate "successful backups"

Well... you can set different retention policies for backup and backup copy jobs. Backup copy job does not copy backup files, it synthetically create VMs based on the existing backup files.

[evander]

Thanks Vitaliy - much appreciated. I will look into the Copy Job options in more detail going forward.

[yizhar]

Hi.

As a reminder about the "job group" feature I've suggested as solution to several limitations of Veeam, here is more info as discussed last year:

Veeam feature request - job group view topic
http://forums.veeam.com/veeam-backup-re ... 17393.html

Yizhar

[scott.anderson]

Must admit that the chaining has been very useful.

I prefer to keep things simple.

Backup job scheduled.

Replication Job set to start as soon as the Backup job finishes.

Tape job set to start as soon as the Backup job finishes.

Simple, whether its incremental during the week or full (synthetic or or otherwise) at the weekend I don't need to worry about the timing.

If the Chaining option were removed then I'd have to schedule the Replication and Tape jobs to run hours after they currently run during the week as they would need to be set to allow for the weekend full processing time.

Its like most features. If you use it wrong then it causes problems.

Use it right and it has loads of advantages.

My vote is to keep the feature!

[foggy]

If the Chaining option were removed then I'd have to schedule the Replication and Tape jobs to run hours after they currently run during the week as they would need to be set to allow for the weekend full processing time.

Scott, just to note, chaining is not required to address this scenario. If jobs are started with short intervals, automatic scheduler will monitor the completion of the backup jobs and start replication and tape jobs accordingly.

[dproffer]

So I have one week of testing data under my belt. I changed all of my backups to begin at the same time as recommended earlier in this thread.

My daily backups finish roughly 30 minutes earlier every day. Because I was chaining jobs, I had already spread out my synthetic full backups over Friday, Saturday, and Sunday evening. All of those jobs are finishing 30 minutes to an hour earlier.

It was not the result I expected. I still see a lot of value for job chaining in many environments, but in mine, the perceived need seems to be dispelled. Only time will tell the true tale.