Enhancement request -- job priority

Availability for the Always-On Enterprise

Re: Enhancement request -- job priority

Veeam Logoby bsoosay » Thu Oct 16, 2014 8:28 am

hi Foggy.

You mentioned this.....Jobs are executed right in the order you start them

What happens if I have 35 jobs that kick start at the same time. Which takes the priority or run order?

I've observed that when I highlight all the jobs and run them immediately, there doesn't seem to be an order.

So what order exactly is Veeam using to prioritize which jobs run first.
bsoosay
Enthusiast
 
Posts: 34
Liked: never
Joined: Thu Oct 02, 2014 7:47 am
Full Name: Benjamin Soosay

Re: Enhancement request -- job priority

Veeam Logoby foggy » Thu Oct 16, 2014 11:22 am

You're right, there's no particular order. Jobs are started simultaneously, on the "first come - first serve" basis. The one that builds VM list first, will get resources first.
foggy
Veeam Software
 
Posts: 15384
Liked: 1140 times
Joined: Mon Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson

Re: Enhancement request -- job priority

Veeam Logoby brupnick » Wed Jan 21, 2015 9:52 pm

I would also love to see an official Priority setting on jobs, but until that happens (I'm always an optimist), I've been using the scheduler as a workaround. However, this doesn't seem to be working as described, at least in version 8. My situation:

My storage repositories are 2xDD7200s, each limited to 40 tasks for a total of 80. My proxies are physical, direct SAN attached DL380G8s with 2x10 core CPUs. I have 4 of these, configured for 20 tasks each, again for a total of 80 concurrent tasks. I have 7 production jobs and 7 development jobs, each with an average of 35 VMs (I'll round these up to 250 VMs each for easy math). My goal is to have the production jobs run first and have the development jobs wait until all production VMs have gone at least once (I know that retries of production VMs will put them at the bottom of the queue, behind development VMs; another argument for true job prioritization). In order to accomplish this, my 7 production jobs start at 7:00PM and my 7 development jobs start at 7:15PM.

At 7:00PM when all of the production jobs kick off, the 80 concurrent tasks are handed out according to which job builds its VM list the fastest and is ready to process; I'm fine with this as I don't really care about the order of the production jobs. All the production VMs that want to start after the 80 tasks are running have to wait; again, this is fine.

However, the problem that I observe starts after 7:15PM when the development jobs kick off. Initially, the 250 development VMs are all waiting for resources which is to be expected because the 250 production servers ran first and have tied up the 80 tasks. Now what I would expect to observe is that no development server would process before every production server has gone at least once, since the development servers should be #251-500 in the queue, but that is not the case. It seems as though once a task becomes available, it is not necessarily assigned to a production server, which should be higher in the queue since it was started earlier. Instead, it is assigned to a development VM, which doesn't seem to match everything that I've read about how this is supposed to work.

If I look through my jobs, I see, for example, that I have a dev server waiting 25 minutes for resources, but also a production server waiting an hour or more. To me, this should never happen because that production server should be given the free tasks before the development server, based on its earlier start time, and therefore higher position in the queue.

I'm hoping that there's something the covers that can explain this behavior, or maybe it's a bug in v8. Either way, any information to help me understand what's happening would be greatly appreciated.
brupnick
Expert
 
Posts: 196
Liked: 13 times
Joined: Sat Feb 05, 2011 5:09 pm
Location: New York, USA
Full Name: Brian Rupnick

Re: Enhancement request -- job priority

Veeam Logoby foggy » Thu Jan 22, 2015 3:23 pm

brupnick wrote:My storage repositories are 2xDD7200s, each limited to 40 tasks for a total of 80.

Brian, are production and development jobs targeted to different repositories? There could be a situation, when there're no available tasks on the production repository, while development repository (in case they are different and dedicated to their corresponding set of jobs) is fully available, which makes priorities shift - tasks from development jobs will be assigned proxy resources even if there are production tasks waiting.
foggy
Veeam Software
 
Posts: 15384
Liked: 1140 times
Joined: Mon Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson

Re: Enhancement request -- job priority

Veeam Logoby brupnick » Thu Jan 22, 2015 3:31 pm

Hey, foggy. All of my jobs, development and production, write to the same backup repositories.
brupnick
Expert
 
Posts: 196
Liked: 13 times
Joined: Sat Feb 05, 2011 5:09 pm
Location: New York, USA
Full Name: Brian Rupnick

Re: Enhancement request -- job priority

Veeam Logoby foggy » Thu Jan 22, 2015 5:03 pm

But still, different repositories are involved (since one job can write to a single repository only). So probably one of them gets used up when another one has only tasks from development jobs in the queue. The number of assigned/free slots on repository at particular moment in time can be monitored through the logs, so you can ask support to review this case.

Another guess is storage I/O control comes into action and stops assigning new tasks involving (overloaded) production datastore.
foggy
Veeam Software
 
Posts: 15384
Liked: 1140 times
Joined: Mon Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson

Re: Enhancement request -- job priority

Veeam Logoby dellock6 » Fri Jan 23, 2015 10:48 am

Maybe you have hit the limits of snapshots per ESXi or datastore so even if there are productions jobs in the queue, they cannot progress until some snapshots are removed. I would guess dev VMs use different datastores at least, if not even different ESXi, so free processing slots are assigned by parallel processing to jobs that can be executed like those of the test/dev, and production jobs can progress only when other "production" VMs are completed.

Luca.
Luca Dell'Oca
EMEA Cloud Architect @ Veeam Software

@dellock6
http://www.virtualtothecore.com
vExpert 2011-2012-2013-2014-2015-2016
Veeam VMCE #1
dellock6
Veeam Software
 
Posts: 5157
Liked: 1386 times
Joined: Sun Jul 26, 2009 3:39 pm
Location: Varese, Italy
Full Name: Luca Dell'Oca

Re: Enhancement request -- job priority

Veeam Logoby brupnick » Mon Jan 26, 2015 2:32 pm

foggy wrote:But still, different repositories are involved (since one job can write to a single repository only). So probably one of them gets used up when another one has only tasks from development jobs in the queue. The number of assigned/free slots on repository at particular moment in time can be monitored through the logs, so you can ask support to review this case.

Another guess is storage I/O control comes into action and stops assigning new tasks involving (overloaded) production datastore.

foggy, is there any way for me to see if storage I/O control is coming into play, possibly through the logs?

dellock6 wrote:Maybe you have hit the limits of snapshots per ESXi or datastore so even if there are productions jobs in the queue, they cannot progress until some snapshots are removed. I would guess dev VMs use different datastores at least, if not even different ESXi, so free processing slots are assigned by parallel processing to jobs that can be executed like those of the test/dev, and production jobs can progress only when other "production" VMs are completed.

Luca, I have seen the message in the job details indicating that a VM is waiting because the maximum number of snapshots for a datastore has been reached, but that is not what's being displayed most of the time. Usually what I see is that it's waiting on proxy or repository resources to become available. Also, development and production VMs are using the same datastores.

Thanks!
brupnick
Expert
 
Posts: 196
Liked: 13 times
Joined: Sat Feb 05, 2011 5:09 pm
Location: New York, USA
Full Name: Brian Rupnick

Re: Enhancement request -- job priority

Veeam Logoby dellock6 » Tue Jan 27, 2015 10:01 pm

Hi Brian,
sorry for late reply, we are having our own Kickoff this week. If dev and prod VMs are sharing the same datastores, then the limits if any have been hit maybe somewhere else. I would check job logs to have detailed informations on how parallel processing is distributing resources.
Luca Dell'Oca
EMEA Cloud Architect @ Veeam Software

@dellock6
http://www.virtualtothecore.com
vExpert 2011-2012-2013-2014-2015-2016
Veeam VMCE #1
dellock6
Veeam Software
 
Posts: 5157
Liked: 1386 times
Joined: Sun Jul 26, 2009 3:39 pm
Location: Varese, Italy
Full Name: Luca Dell'Oca

Re: Enhancement request -- job priority

Veeam Logoby brupnick » Wed Jan 28, 2015 2:13 pm

Hey Luca-

Not a problem at all! Which log file would contain information the information about the queues that I'd be interested in?

Thanks!
brupnick
Expert
 
Posts: 196
Liked: 13 times
Joined: Sat Feb 05, 2011 5:09 pm
Location: New York, USA
Full Name: Brian Rupnick

Re: Enhancement request -- job priority

Veeam Logoby v.Eremin » Wed Jan 28, 2015 2:31 pm

Mostly job logs (those that can be exported via menu -> help -> support information). Thanks.
v.Eremin
Veeam Software
 
Posts: 13856
Liked: 1037 times
Joined: Fri Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin

Re: Enhancement request -- job priority

Veeam Logoby brupnick » Mon Feb 02, 2015 2:58 pm

v.Eremin wrote:Mostly job logs (those that can be exported via menu -> help -> support information). Thanks.

And what should I be looking for to find details about the queue?
brupnick
Expert
 
Posts: 196
Liked: 13 times
Joined: Sat Feb 05, 2011 5:09 pm
Location: New York, USA
Full Name: Brian Rupnick

Re: Enhancement request -- job priority

Veeam Logoby foggy » Mon Feb 02, 2015 5:11 pm

Brian, for assistance in reading logs, please contact technical support.
foggy
Veeam Software
 
Posts: 15384
Liked: 1140 times
Joined: Mon Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson

Re: Enhancement request -- job priority

Veeam Logoby ChuckS42 » Mon Feb 01, 2016 6:56 pm

I'm in need of a real backup priority system as well. I've got a job that must start at a specific time (2:00AM). If it starts processing too late it causes problems. The "first scheduled first run" concept won't work here. It seems I can create a dedicated proxy server just for this job, but that seems wasteful, and doesn't take into account repository slots. Really surprised this hasn't made it into the product yet.
ChuckS42
Enthusiast
 
Posts: 71
Liked: 5 times
Joined: Wed Apr 24, 2013 8:53 pm
Full Name: Chuck Stevens

Re: Enhancement request -- job priority

Veeam Logoby foggy » Tue Feb 02, 2016 11:50 am

Chuck, could you please share the reasons for starting this job exactly at this time and the problems that the later job start causes?

As another workaround, think of using backup window for other jobs to restrict their execution times when this job needs to start.
foggy
Veeam Software
 
Posts: 15384
Liked: 1140 times
Joined: Mon Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson

PreviousNext

Return to Veeam Backup & Replication



Who is online

Users browsing this forum: Google [Bot], Google Feedfetcher and 1 guest