Enhancement request -- job priority

[averylarry]

At least I couldn't find any way to do this.

I have some local backups, some local replications, and some WAN replications that run each night. With v.7 I've turned on parallel processing. What I am experiencing is the local replication and local backup jobs "eating up" all the parallel processing "slots". This means my WAN replications jobs will at times sit idle while the other jobs are running multiple VMs.

So I'd like to request a job priority feature. I could put my WAN replications jobs on a higher priority than my local jobs. That way the WAN replications jobs would always be using at least some of the parallel processing "slots" available (except for the normal conflicts). I suppose I could disable the parallel processing, but what fun would that be?


On a side note -- when a VM finishes processing, how does Veeam decide which VM in which job gets processed next (after eliminating VMs that have a conflict)?

[averylarry]

I'll do some experimenting. I think it might process the jobs in alphabetical order.

[Gostev]

Jobs are processed according to their start times. Make sure your high priority jobs are set to start earlier than other.

I don't disagree job priorities can still be helpful in certain scenarios. We will look at v7 in the wild, and decide how to best implement those. Meanwhile, issues like above can be easily resolved by deploying more proxies, or reducing max concurrent ingress tasks on repositories, or just using dedicated proxy sets for all your jobs (or any combination of these approaches).

[dellock6]

Also, be careful about VMs with several disks, since a single VM with let's say 5 disk is going to use 5 processing slots. You may want to place the jobs with those VMs to be started after others, but if for example a VM fails, is placed in the retry queue, so somehow your schedule is not going to be fully followed by Veeam. Unless you configure retries differently.

I'm in the phase of tuning our backup infrastructure after a couple of weeks using V7, right because of these kind of findings.

Luca.

[averylarry]

Gostev --

Jobs begin based on their start times, sure. But I end up with 5-10 jobs running all at the same time. They are not processed in series based on their start times. They are processed in parallel. Now with the additional feature of parallel processing, I want to know how the Veeam server decides which VM in which job to process when a VM in whatever job finishes.

Also -- I'm a small business. Cannot afford more proxies (I can't even afford a dedicated proxy). I don't understand how the other suggestions would make any difference.


Luca --

Note that hotadd mode only does 1 disk at a time, so parallel processing is strictly for multiple simultaneous VMs.


Ultimately I want the ability to make sure my WAN is always being used to avoid running into the work day. A dedicated WAN replication proxy would accomplish this (and a good idea I didn't consider). I just don't consider that feasible for many small businesses (I have minimized the number of proxies I use because they are production machines and making them a proxy will break hotadd mode and direct SAN mode also costs more money in infrastructure and network mode is generally slow/inefficient).

[foggy]

Now with the additional feature of parallel processing, I want to know how the Veeam server decides which VM in which job to process when a VM in whatever job finishes.

This thread is likely to answer your question.

[JeremiahS]

You may want to try chaining jobs to avoid resource contention and allow manual job prioritization. Under schedule, configure highest priority job to at desired time with next highest priority time starting after job1 finishes and so on. With limited proxies and single repository design, this approach seems to be working reasonably well.

[averylarry]

Jeremiah --

Thanks for the idea. The problem is I can come up with a dozen different workarounds. I would just like something that I would consider more of a "fix" to directly address an issue I've found, instead of figuring out a way to circumvent (what I consider) a limitation.

[larry]

I would like this as well.
I found with backupcopy jobs at times will do 3 VMs from one job, a VM from job 3 then back to job 1 for a vm so alot of jobs endup open and running. The jobs are continuos but the backup job I start in the order I want them to run.

[veremin]

Hi, Larry. You can add particular VMs, instead of backup jobs, as a source for backup copy job and explicitly set the processing order, using “Up” and “Down” buttons. Thanks.

[larry]

I like the add by job too much. With 7 all my backup jobs have 2 backupcopy jobs to send to DR and tape, same name with -DR added. This way once you place a VM into a backupjob the backup to tape and offsite is automatic. Before we would get a vm not going to tape or offsite but getting backed up, this way it will just happen. Our jobs are VM A-C ( all vms who names start with A-C, VM D-G and so on. We name the backup location, tape job and DR spot the same. We do pull out a couple servers that get treated special, exchange and SQL. With one quick look you know where the VM should be as well as its offsite copy. thats my reasoning.

[ekisner]

I'm presently in a situation where I've got my replication jobs disabled... going to be doing some vcenter maintenance "at some point". Since some jobs can take a long time, and I'd rather not take vcenter offline while there's a job running, disable seems like the logical choice.

I've just pushed back the maintenance however, and re-enabled the jobs in the short term. Since they were disabled and out of date I initiated a manual run to get everything current. And since I have jobs on a 1:1 basis with VMs, I now have quite a few active jobs.

I could see many other situations where this would be applicable too, but that's how I came to the thought that I'm now suggesting.

It would be nice to be able to prioritize resources to jobs.. the "more important" jobs can hence finish sooner, and the "less important jobs" can finish whenever.

[foggy]

Thanks for your feedback. Currently you can prioritize jobs according to their start time (the job that started first will be processed first).

[bsoosay]

I've configured my Backups by datastores. I have about 35 datastores.

I have 6 proxy servers with 4 cores, which totals up to 24 connections to the repository.

Now when I execute all 35 jobs, how does Veeam prioritise which job runs first?

1. Is it by alphabetical order, sequential order or some other method?

2. Is there a way for me to prioritise which job runs first?

[foggy]

Jobs are executed right in the order you start them (literally click Start on the job) and tasks are assigned to proxy servers upon availability of processing slots.

[bsoosay]

hi Foggy.

You mentioned this.....Jobs are executed right in the order you start them

What happens if I have 35 jobs that kick start at the same time. Which takes the priority or run order?

I've observed that when I highlight all the jobs and run them immediately, there doesn't seem to be an order.

So what order exactly is Veeam using to prioritize which jobs run first.

[foggy]

You're right, there's no particular order. Jobs are started simultaneously, on the "first come - first serve" basis. The one that builds VM list first, will get resources first.

[brupnick]

I would also love to see an official Priority setting on jobs, but until that happens (I'm always an optimist), I've been using the scheduler as a workaround. However, this doesn't seem to be working as described, at least in version 8. My situation:

My storage repositories are 2xDD7200s, each limited to 40 tasks for a total of 80. My proxies are physical, direct SAN attached DL380G8s with 2x10 core CPUs. I have 4 of these, configured for 20 tasks each, again for a total of 80 concurrent tasks. I have 7 production jobs and 7 development jobs, each with an average of 35 VMs (I'll round these up to 250 VMs each for easy math). My goal is to have the production jobs run first and have the development jobs wait until all production VMs have gone at least once (I know that retries of production VMs will put them at the bottom of the queue, behind development VMs; another argument for true job prioritization). In order to accomplish this, my 7 production jobs start at 7:00PM and my 7 development jobs start at 7:15PM.

At 7:00PM when all of the production jobs kick off, the 80 concurrent tasks are handed out according to which job builds its VM list the fastest and is ready to process; I'm fine with this as I don't really care about the order of the production jobs. All the production VMs that want to start after the 80 tasks are running have to wait; again, this is fine.

However, the problem that I observe starts after 7:15PM when the development jobs kick off. Initially, the 250 development VMs are all waiting for resources which is to be expected because the 250 production servers ran first and have tied up the 80 tasks. Now what I would expect to observe is that no development server would process before every production server has gone at least once, since the development servers should be #251-500 in the queue, but that is not the case. It seems as though once a task becomes available, it is not necessarily assigned to a production server, which should be higher in the queue since it was started earlier. Instead, it is assigned to a development VM, which doesn't seem to match everything that I've read about how this is supposed to work.

If I look through my jobs, I see, for example, that I have a dev server waiting 25 minutes for resources, but also a production server waiting an hour or more. To me, this should never happen because that production server should be given the free tasks before the development server, based on its earlier start time, and therefore higher position in the queue.

I'm hoping that there's something the covers that can explain this behavior, or maybe it's a bug in v8. Either way, any information to help me understand what's happening would be greatly appreciated.

[foggy]

My storage repositories are 2xDD7200s, each limited to 40 tasks for a total of 80.

Brian, are production and development jobs targeted to different repositories? There could be a situation, when there're no available tasks on the production repository, while development repository (in case they are different and dedicated to their corresponding set of jobs) is fully available, which makes priorities shift - tasks from development jobs will be assigned proxy resources even if there are production tasks waiting.

[brupnick]

Hey, foggy. All of my jobs, development and production, write to the same backup repositories.

[foggy]

But still, different repositories are involved (since one job can write to a single repository only). So probably one of them gets used up when another one has only tasks from development jobs in the queue. The number of assigned/free slots on repository at particular moment in time can be monitored through the logs, so you can ask support to review this case.

Another guess is storage I/O control comes into action and stops assigning new tasks involving (overloaded) production datastore.

[dellock6]

Maybe you have hit the limits of snapshots per ESXi or datastore so even if there are productions jobs in the queue, they cannot progress until some snapshots are removed. I would guess dev VMs use different datastores at least, if not even different ESXi, so free processing slots are assigned by parallel processing to jobs that can be executed like those of the test/dev, and production jobs can progress only when other "production" VMs are completed.

Luca.

[brupnick]

But still, different repositories are involved (since one job can write to a single repository only). So probably one of them gets used up when another one has only tasks from development jobs in the queue. The number of assigned/free slots on repository at particular moment in time can be monitored through the logs, so you can ask support to review this case.

Another guess is storage I/O control comes into action and stops assigning new tasks involving (overloaded) production datastore.

foggy, is there any way for me to see if storage I/O control is coming into play, possibly through the logs?

Maybe you have hit the limits of snapshots per ESXi or datastore so even if there are productions jobs in the queue, they cannot progress until some snapshots are removed. I would guess dev VMs use different datastores at least, if not even different ESXi, so free processing slots are assigned by parallel processing to jobs that can be executed like those of the test/dev, and production jobs can progress only when other "production" VMs are completed.

Luca, I have seen the message in the job details indicating that a VM is waiting because the maximum number of snapshots for a datastore has been reached, but that is not what's being displayed most of the time. Usually what I see is that it's waiting on proxy or repository resources to become available. Also, development and production VMs are using the same datastores.

Thanks!

[dellock6]

Hi Brian,
sorry for late reply, we are having our own Kickoff this week. If dev and prod VMs are sharing the same datastores, then the limits if any have been hit maybe somewhere else. I would check job logs to have detailed informations on how parallel processing is distributing resources.

[brupnick]

Hey Luca-

Not a problem at all! Which log file would contain information the information about the queues that I'd be interested in?

Thanks!

[veremin]

Mostly job logs (those that can be exported via menu -> help -> support information). Thanks.

[brupnick]

Mostly job logs (those that can be exported via menu -> help -> support information). Thanks.

And what should I be looking for to find details about the queue?

[foggy]

Brian, for assistance in reading logs, please contact technical support.

[ChuckS42]

I'm in need of a real backup priority system as well. I've got a job that must start at a specific time (2:00AM). If it starts processing too late it causes problems. The "first scheduled first run" concept won't work here. It seems I can create a dedicated proxy server just for this job, but that seems wasteful, and doesn't take into account repository slots. Really surprised this hasn't made it into the product yet.

[foggy]

Chuck, could you please share the reasons for starting this job exactly at this time and the problems that the later job start causes?

As another workaround, think of using backup window for other jobs to restrict their execution times when this job needs to start.