Veeam 8 Load Balancing Issue + Job Priority

[isolated_1]

Environment breakdown:
- Single Veeam proxy server
- Two repositories
- 30+ backup jobs
- Veeam version = 8.0.0.917 (recently performed upgrade from Veeam 7)
- Veeam Support case # 00870705

Situation:
In Veeam 7, we needed a couple of multi-vm backup jobs to finish ahead of others. Since there is no job priority feature in Veeam, I had to start those special jobs prior to the others, which all start at 9PM. So, I configured these jobs to start at around 8:45-8:50PM and it works great. These 3-4 jobs get the available resources due to them starting ahead of the others. Each VM in these jobs would be processed in turn and when 9PM comes around, the latter jobs would start and if there were no available resources available, they would have to wait. This is fine with us because priority should be given to the jobs we started earlier.

The Problem:
After upgrading to Veeam 8 the other week, I believe the new auto load balancing feature is messing things up. From what I have gathered in the What's New document, this feature is suppose to give the necessary resources to jobs based on start time. However, the jobs that I have started earlier are now actually getting paused once 9PM comes around because Veeam is assigning the resources to all the other jobs! What happens is from 8:45-8:50PM, the 3-4 jobs start and everything is kosher. Once 9PM rolls around, all the other 30+ jobs start and what will happen to these earlier jobs is Veeam will finish backing up the current VM in the backup job. Once that is finished, it will assign the resources to another job leaving this one in a "paused" state. In fact, these 3-4 jobs I've started earlier are actually now the last one's to finish out of all the jobs in Veeam!

Things Tried:
Working Veeam support, here are things we have tried:
- Increasing the concurrent tasks in both repositories to match that of my single Veeam proxy server
- Enabled Parallel Processing, which actually made things worst

Can anyone figure out why this is happening? I want things to go back like how it did in Veeam 7. With the load balancing feature, it seems like the infrastructure resources are being randomly assigned to backup jobs without taking job start time into consideration.

[foggy]

Simon, how many source datastores do you have? How VMs/jobs are distributed among them?

[Gostev]

Job start time is the only priority system we have, and it should definitely work in the way that What's New document explains (and your understanding above is correct).

Assuming we are talking about daily jobs, the scheduling is super straight forward: internal list that tracks the required VM processing order is populated as new jobs start, and starting jobs they can only APPEND their VMs to the existing list. There is simply no way for them to "inject" them somewhere in the middle of the list, or jump in front of it. Now, every time new backup infrastructure resource becomes available, the task scheduler goes from the top of the list until it hits the VM it is able to process with the newly acquired infrastructure resource - and then, it starts processing it.

As you can see from that, the ONLY way how a VM2 from a job B (that was started later) can get processed before a VM1 from a job A (that was started earlier), is when VM1 is waiting for backup infrastructure resources required for its processing, while VM2 already has those available (for example, because of jobs A and B going to different backup repositories). So, see if this could be the issue.

[isolated_1]

Foggy - Did you mean repositories? We have two. Currently there are 16 jobs going to repository 1 and 19 to repository 2.

Gostev - Thank you for the explanation but here is where I am a bit confused. What you said makes perfect sense. However was this always the case with Veeam 7? Because it seems like in Veeam 7, rather than "VMs" getting appended to the list, it was more like "backup jobs" that got appended instead. From what I can remember, once a backup job got started, ALL VMs in that job would be processed sequentially regardless of how many jobs started after. The only way the other jobs would be able to get resources is if the jobs, not VMs, that started earlier were completely finished.

Also, based on what you have said, wouldn't the jobs that got started earlier would have all of their VMs on top of the list and be processed first regardless of what happens? Lets say I have 4 jobs. Jobs A and B goes to repository 1 and C and D to repository 2. All four jobs got started at 8:45 PM. In Veeam, I can see that the VMs in the jobs appear in the backup window and backup is going normally for all four jobs. When it's 9PM and all other jobs start, wouldn't these VMs take a back seat (assuming I don't have anymore resources to give) since the jobs in all four of the earlier jobs got their VMs on top of the list? This is what I expect to happen but it's not! What you said that shouldn't happen about VMs being injected in front of other VMs in the list is actually happening!

Please let me know if you guys need more info. And thanks for helping!

[tsightler]

One important thing to remember is that "resources" can refer to things like datastores as well. By default Veeam will only run 4 snapshots on the same datastore within a job, so if a job has a lot of VMs from a single datastore, while the job stated later has VMs on another datastore, it's possible that the second job will get task assignments even though the first job is not finished. Also, if you've enabled storage I/O control it can play a role here as well since it can stop tasks being assigned to a datastore that is already overloaded which means proxy task resources can be assigned to jobs datastores that aren't as busy.

[brupnick]

I had a similar issue with v8 as FIFO was not working as I understood it to. In my case, I had my production job starting at 7:00PM and my non-production jobs starting at 7:15PM. The goal was to have the production VMs in the queue ahead of the non-production VMs, but in actuality, this is not what was happening - non-production VMs were being assigned resources ahead of production machines. I opened a ticket with support (00744805 for those who have access to it) and was told that "QA is confirming incorrect behavior and planning to fix it in major release due to complexity of the fix implementation."

Ultimately, being able to control priority through some sort of high/medium/low mechanism rather than start time is really something that needs to be added sooner rather than later.

[foggy]

Foggy - Did you mean repositories? We have two. Currently there are 16 jobs going to repository 1 and 19 to repository 2.

I meant source VMware datastores, for the reasons explained by Tom above.

[isolated_1]

One important thing to remember is that "resources" can refer to things like datastores as well. By default Veeam will only run 4 snapshots on the same datastore within a job, so if a job has a lot of VMs from a single datastore, while the job stated later has VMs on another datastore, it's possible that the second job will get task assignments even though the first job is not finished. Also, if you've enabled storage I/O control it can play a role here as well since it can stop tasks being assigned to a datastore that is already overloaded which means proxy task resources can be assigned to jobs datastores that aren't as busy.

Most of our VMs are spread across multiple datastores. But this was never an issue with Veeam 7, only in Veeam 8. Storage I/O is not enabled.

I meant source VMware datastores, for the reasons explained by Tom above.

We have around 20-22 datastores in VMware.

I had a similar issue with v8 as FIFO was not working as I understood it to. In my case, I had my production job starting at 7:00PM and my non-production jobs starting at 7:15PM. The goal was to have the production VMs in the queue ahead of the non-production VMs, but in actuality, this is not what was happening - non-production VMs were being assigned resources ahead of production machines. I opened a ticket with support (00744805 for those who have access to it) and was told that "QA is confirming incorrect behavior and planning to fix it in major release due to complexity of the fix implementation."

Ultimately, being able to control priority through some sort of high/medium/low mechanism rather than start time is really something that needs to be added sooner rather than later.

Thank you for chiming in! Knowing another person that is facing the same issue sort of confirms that its not just my configuration. I am also hoping for a proper job priority feature we can customize rather than just basing it on start time. it works for simple environments like ours but going forward, I'm sure everyone will welcome it.

Can someone from Veeam confirm what brupnick said about QA confirming this incorrect behavior and is working to patch it? We have some high-maintenance clients that randomly request backup reports and its hard to explain how their backup jobs went from finishing at 11 PM to now completing at 2:30 AM. If Veeam is working on this issue, then we can at least relay that to the client.

Thanks!

[Gostev]

I've requested additional info.

Answering your question on v7 scheduling, it was pretty chaotic (newly appearing resources were split between all running jobs). Main idea behind v8 changes was to make scheduler's top priority to complete every started job as soon as possible, by ensuring that newly started jobs do not impact availability of resources to already running jobs.

[dellock6]

Curious too to read further info.

Just a note, as Tom and others explained, is the queue itself is populated sequentially, but the processing of a VM depends on the availability of resources to process it. So it can happen a VM that is 5th in the list is processed first, because is the only one in a new datastore while 1-4 are on the same datastore where there are already 4 snapshots of other VMs (thus hitting one of the limits, that is 4 snapshots on the same datastore).

The new logic of v8 was however exactly not to assign resource "too evenly" to any open job, and instead to try and respect as much as possible the queue (as limits allow).

[isolated_1]

All the info I've gotten so far from the support technician I am working with via email to the What's New document and now here on the Veeam forums points out that Veeam 8 is now supposed to have better logic and a better job priority system such as jobs that start later not impacting jobs that started earlier is exactly the opposite from what I am seeing in Veeam 8! Gostev mentioned that Veeam 7 was chaotic but I actually found it to be a lot more predictable in terms of job priority! In V7, when a multi-vm job was started, as soon as the first VM in the job was getting processed, the job kept going smoothly until completion. It didn't matter how many jobs started afterwards. In V8, jobs that started earlier can now have ALL VMs stuck in the "Pending" state even though Veeam started to process it prior to other jobs starting. Resource assignment seems all over the place.

@dellock6 - Sorry, when I said "sequentially", I didn't exactly mean that every VM in the backup list was processed from top to bottom in the order shown. What I meant to say was that when that multi-vm job got started, Veeam would process every single VM in that job one after another as soon as just one VM within that job was started by Veeam.

[Gostev]

In V7, when a multi-vm job was started, as soon as the first VM in the job was getting processed, the job kept going smoothly until completion

This was solely driven by unique availability of backup infrastructure resources in your case then, because with "infinite" resources all running v7 jobs competed with each other for resources, resulting each one running for the entire backup window in most cases. Most users had an issue with this approach, because restore points of each VM within a given job were too far away from each other in time (like, 6-8 hours difference).

[Gostev]

I have reviewed the internal notes for your support case, Simon. Looks like the issue was simply postponed for deeper research for after Update 2. According to the debug logs, incorrect task assignments do happen occasionally in your case, but the reason for them is not immediately obvious - as they happen against the main task scheduling algorithm (which I have explained earlier). After a quick research, devs could not find an obvious bug in that algorithm. This is usually a sign of a rare bug due inter-process synchronization issue (or something along these lines), and these typically require significant time to research, while resulting in potentially dangerous fixes, which we just could no be possibly squeezed last minute into the Update 2. Due to the imminent Update 2 release and dev/QC focus on this release, the plan is to perform a deeper research and produce the hot fix on top of Update 2 once it is out. Thanks!

[isolated_1]

Thanks Gostev for confirming and everyone else for chiming in! This is definitely something I can at least relay to our clients if the issue comes up.

[PaulNSW]

Is there a way to make sure a job you start manually, gets priority for the available Veeam resources?

I started a backup job on my Veeam server at 7:50, it contains 6 VMs.
My Replication jobs (all 10) began at 8:00 on schedule.
However my backup job is now stuck waiting for resources as the replication jobs keep stealing them!

[Shestakov]

Hello,
As stated above there is no other priority besides job start time.
"The backup job is waiting for resources" means probably that one datastore accomplishes tasks faster than another. Please review the thread for the detailed explanation.
Thanks!

[bleyers]

Hi,

The following happens when 2 jobs runs simultaneously

- 2 backup jobs, job 1 starts at 10PM, and 2 at 11PM.
Both backup jobs have 30 VMs with at least 1 disk.

- 2 proxy servers, 2 concurrent task per proxy (4 total)

- 10PM
job 1 starts and consumes all proxy slots
-11PM
job 2 starts and waits for backup resources (proxy slot)
- after 11 PM
proxy slots are granted to both job 1 and 2


My question is:
Why does veeam give resources to job 2 when job 1 is not completed ?
Is there a way to force veeam to give job 1 all the resources first?

thanks

[PTide]

Hi,

Why does veeam give resources to job 2 when job 1 is not completed ?

That could happen, for example, if Job_1 had only 1 disk to process so Job_2 kicks in and gets three remaining slots. Could you please check if that's the case and how many of Job_1 disks are to be processed when the second job gets the resources?

Is there a way to force veeam to give job 1 all the resources first?

That can be accomplished with either job chaining or PowerShell pre-run script, the latter is more preferable.

Thank you.

[bleyers]

There are plenty of disks that need processing in job 1 when job 2 gets slots...
I would indeed suspect that job 2 gets slots when all remaining disks of job 1 have resources but this is not the case.

I don't want to do chaining because i want job 2 to begin as soon as there are resources available but only after
job 1 does not need them anymore.
With chaining you lose time if there is 1 large disk in job 1, then job 2 cant start although there are resources available.

am i making any sense ?

[PTide]

How many backup repositories do you have? If more than 1 - do both jobs go to the same repo? If that's the case please review the thread for more detailed explanation.

[bleyers]

We have multiple repositories.

yesterday i started 4 jobs. 1 and 2 went to repo A, 3 and 4 went to repo B
the slots were allotted to all 4 jobs. Not the one that started first.

[PTide]

Please open a case with support for deeper investigation and post your case ID here.

Thank you.

[bleyers]

Opened a case and i got this as answer...

If you run the jobs concurrently, the resources will be automatically distributed between all running jobs and it's not possible to force it manually.
However, if you want the first job to process first, you could schedule the second job to run after the first one.
In the schedule settings you'll see "After this job:": http://helpcenter.veeam.com/backup/80/v ... le_vm.html

According to this response there is no priority based on start time...

Case ID 01219389

[PTide]

However, if you want the first job to process first, you could schedule the second job to run after the first one.

That could strictly hold in case you had one repository. I suggest you to either escalate the case by pressing "Talk to manager" button or try to provide more details to your support engineer (e.g. number of proxies, numbert of repos, proxy mode etc).

Thank you.

[Gostev]

According to this response there is no priority based on start time...

There is. Every freeing up backup infrastructure resource is first offered to jobs which started earlier. Please see my first post in this topic.