Wacky Job Queuing in Version 8

[jleblanc]

Today I upgraded by B&R from Version 7 to Version 8. I noticed a strange new way that concurrent resource constrained jobs run that is completely different than how it did in version 7.

Here is an example of what I've noticed:

My backup proxy is configured to process only 1 concurrent task. I have no I/O Control configured. Lets say I have two jobs.
Job 1 (4 VM's) - Scheduled for 2:05 PM
Job 2 (5 VM's) - Scheduled for 2:10 PM

In version 7, job 1 would run to completion, and job 2 would be pending until job 1 finished.
Now with version 8, job 1 backs up its first VM, then job 2 backs up its first VM, then job 1 backs up its second VM, then job 2...you get the point.

This flip flop behavior strikes me as strange and unnecessary. I opened a case with support (00676283) and got the response that the job queuing for processing vms has been modified for Version 8. Is anyone else noticing this behavior and can anyone from the Veeam product management team speak as to why the functionality has changed? The workaround provided was to either chain my backup jobs, or to expand the schedule to when the job before it usually completes. Both of these workarounds are less than ideal, as the first could introduce missed backups and the later expands my backup window by quite a bit.

[Gostev]

Yes, it has been modified and the changes are described in details in the What's New document. Please, ask your support engineer to read that document to see where you see the issue

Basically, per new v8 scheduler behavior, in your specific scenario, I would expect Job 1 to complete processing ALL of its VMs before Job 2 is even allowed to start processing of it's first VM. Unless perhaps you have some very special scenario when there is only 1 proxy capable of processing Job 1 VMs.

As such, it would be a good idea to not accept the response you've got as the solution, and have them continue investigating this case.

In version 7, job 1 would run to completion, and job 2 would be pending until job 1 finished

This observation is strange though, because the problem we had in v7 was exactly opposite: all appearing processing slots were spread evenly across all running jobs, so each particular job was taking forever to complete, resulting in all jobs included in the VM to have significant time difference between captured states. And changes to scheduled logic in v8 were aimed at fixing exactly this issue.

Thanks!

[dellock6]

I was going to reply exactly as Anton, this behaviour is not expected at all as per how we designed the new processing method.
And cuncurrency=1 should not be an edge case, but simply allow to process job1 completely without wasting processing slots with other queued jobs.

[jleblanc]

Unless perhaps you have some very special scenario when there is only 1 proxy capable of processing Job 1 VMs.
Thanks!

I only have 1 Proxy in my environment, and it is configured for 1 concurrent job; so I don't think I'm running into any fringe case scenarios.

I wasn't able to find the "Whats New" document, however I pointed the Support Engineer assigned to my case to this forum thread. Hopefully we can get this fixed; as its so bizarre I am experiencing the exact behavior the change was designed to fix. Everything worked flawlessly in V7.

[Gostev]

On veeam.com, click Resources > Technical Documentation.

With your setup, you are actually very well may be running in fringe case scenarios. Especially if you also have parallel processing disabled, because looking at your v7 job behavior description (NOW it makes sense to me) I am guessing you have that disabled as well? But this basically makes all your settings non-default, and such setups are never well covered in testing.

Another question is why did you put together such a strange setup? If you tell me more about your requirements, then perhaps I can suggest you a better way to setup our product. For example, while I can imagine some legitimate reasons behind limiting each proxy to 1 concurrent task in extreme cases, there is not a single good reason for disabling parallel processing with 1 proxy and 1 concurrent task, as in such setup parallel processing will simply remove "dead time" between VMs without ANY negative impact on your backup infrastructure.

[jleblanc]

Okay so here is an update.

Last night I went into I/O Control and enabled parallel processing. I then kicked off two jobs, this time it functioned as desired; Completing job 1 entirely before moving to job number 2.

In regards to why I have such a configuration, basically I'm trying to limit I/O during my business hours by only letting one job run at a time. As to why I had both the max concurrent task set to 1 and Parallel processing disabled,I can't really answer that, but it only became an issue in Version 8. I'll keep an eye on my jobs over the next couple days but I think that enabling parallel processing resolved this weird bug(?) when max concurrent tasks are set with parallel processing enabled.

Thanks All for your insight.

[dellock6]

Also thanks you for the update on the topic and for the tests you are doing.
A wild guess from me: since the new priority logic is part of parallel processing, by disabling it the logic reverted back to the previous mode, thus assigning even resources to both jobs that were in the execution queue.
This would explain the behaviour going back to an expected outcome when re-enabling parallel processing.

Luca.

[Gostev]

I'm trying to limit I/O during my business hours

This is exactly what new v8 Backup I/O Control feature is designed to do