v6 Limit on Number of Jobs (64)?

[jkcouch]

I think we have found a limit on the number of jobs in v6. Has anyone on the forum tried to deploy more than 64 jobs on a single VBR server? If so, were you successful?

After working with support/development, we have been advised to back down the number of jobs from 70 to below 64. We are receiving these errors in the Veeam backup service logs:

Code: Select all

Error    Failed to track states of jobs. Waiting before restart states tracking [5 sec]
Error    The number of WaitHandles must be less than or equal to 64. (System.NotSupportedException)
Error       at System.Threading.WaitHandle.WaitAny(WaitHandle[] waitHandles, Int32 millisecondsTimeout, Boolean exitContext)
Error       at System.Threading.WaitHandle.WaitAny(WaitHandle[] waitHandles, TimeSpan timeout, Boolean exitContext)
Error       at Veeam.Backup.Service.CJobStateTracker.TrackJobs()
Error       at Veeam.Backup.Service.CJobStateTracker.RetryableJobsTrackingCallback()

Case number is 5162899. This was a merged case from other issues, but is currently being worked on for this issue.

What we see as symptom of this is jobs not starting as scheduled, and retries not working. The current workaround wed have is rebooting the Veeam server daily when backups are not running. We are waiting on word from development on if a fix is possible, but the current word is it could be a .net limitation.

[Sethbartlett]

Yeah does seem to be a limitation of System.Threading: http://msdn.microsoft.com/en-us/library/tdykks7z.aspx

"This method returns when any handle is signaled. If more than one object becomes signaled during the call, the return value is the array index of the signaled object with the smallest index value of all the signaled objects. On some implementations, if more that 64 handles are passed, a NotSupportedException is thrown."

Not stating that something can't be done about it on how we design how scheduling works possibly but definitely a limitation of Threading/Mutex.

[wolfgang.reisert]

At the moment we have 76 jobs running in v5 without any problems. Does this realy mean that we have the reduce the jobs below 64 when upgrading to v6? We are running .net 3.5SP1. When this should be a .net limitation but v5 is running without problems can I expect that v6 will run in the same way?

[Sethbartlett]

Are you running 76 jobs all at the same time? This is more of a running all these synchronously.

[wolfgang.reisert]

No, the 76 jobs aren't running all at the same time. I schedule the jobs in that way that not more than 5-7 jobs are running at the same time.

[Sethbartlett]

Gotcha, this is a limitation of running 64+ jobs all at the same time:) Takes a VERY beefy Veeam Box haha.

[wolfgang.reisert]

Ok thank you. 64+ jobs is really a bit heavy.

[tsightler]

Jeff,

I have a couple of clients that have more that 64 jobs, in some cases >100 jobs, but none of them are attempting to run them all at the same time, they simple stagger the start of the jobs so that only 20-30 are ever in "started" state. Are you saying you have this issue even if you stagger the schedule?

[jkcouch]

We haven’t tried that Tom. We stagger them, but very close together so that we get the maximum use of the queuing abilities of v6. I will try scaling back the schedule some to see if we can keep it under that 64 count. I will let you know how it turns out.

Good suggestion!

[Gostev]

Yes, this limitation is for concurrent jobs only, not total amount of jobs. My plan is to "enlighten" the job scheduler with the knowledge of this OS limitation, so that it does not try to start more than 64 jobs at once - but I did not discuss the possibility of this with R&D yet.

[jkcouch]

I can confirm that backing off the concurrent job count below 64 fixes the issue. Thanks guys for the clarification. I would like to see this fixed so we can queue up as many jobs as we like, sense I see that as the key to making our backup jobs complete as fast as possible. With v5 not having queuing capabilities, we had hours of dead time between backups. I don’t want to do that again .

[tsightler]

Thanks for the verification Jeff. I'm quite sure that development will address this issue, although admittedly it might be a simple "64 active job" limit for the immediate workaround.