I would also love to see an official Priority setting on jobs, but until that happens (I'm always an optimist), I've been using the scheduler as a workaround. However, this doesn't seem to be working as described, at least in version 8. My situation:
My storage repositories are 2xDD7200s, each limited to 40 tasks for a total of 80. My proxies are physical, direct SAN attached DL380G8s with 2x10 core CPUs. I have 4 of these, configured for 20 tasks each, again for a total of 80 concurrent tasks. I have 7 production jobs and 7 development jobs, each with an average of 35 VMs (I'll round these up to 250 VMs each for easy math). My goal is to have the production jobs run first and have the development jobs wait until all production VMs have gone at least once (I know that retries of production VMs will put them at the bottom of the queue, behind development VMs; another argument for true job prioritization). In order to accomplish this, my 7 production jobs start at 7:00PM and my 7 development jobs start at 7:15PM.
At 7:00PM when all of the production jobs kick off, the 80 concurrent tasks are handed out according to which job builds its VM list the fastest and is ready to process; I'm fine with this as I don't really care about the order of the production jobs. All the production VMs that want to start after the 80 tasks are running have to wait; again, this is fine.
However, the problem that I observe starts after 7:15PM when the development jobs kick off. Initially, the 250 development VMs are all waiting for resources which is to be expected because the 250 production servers ran first and have tied up the 80 tasks. Now what I would expect to observe is that no development server would process before every production server has gone at least once, since the development servers should be #251-500 in the queue, but that is not the case. It seems as though once a task becomes available, it is not necessarily assigned to a production server, which should be higher in the queue since it was started earlier. Instead, it is assigned to a development VM, which doesn't seem to match everything that I've read about how this is supposed to work.
If I look through my jobs, I see, for example, that I have a dev server waiting 25 minutes for resources, but also a production server waiting an hour or more. To me, this should never happen because that production server should be given the free tasks before the development server, based on its earlier start time, and therefore higher position in the queue.
I'm hoping that there's something the covers that can explain this behavior, or maybe it's a bug in v8. Either way, any information to help me understand what's happening would be greatly appreciated.