Comprehensive data protection for all workloads
Post Reply
jkcouch
Enthusiast
Posts: 30
Liked: never
Joined: Jun 26, 2011 7:02 pm
Full Name: Jeff Couch
Contact:

v6 Limit on Number of Jobs (64)?

Post by jkcouch » Jan 09, 2012 1:00 am

I think we have found a limit on the number of jobs in v6. Has anyone on the forum tried to deploy more than 64 jobs on a single VBR server? If so, were you successful?

After working with support/development, we have been advised to back down the number of jobs from 70 to below 64. We are receiving these errors in the Veeam backup service logs:

Code: Select all

Error    Failed to track states of jobs. Waiting before restart states tracking [5 sec]
Error    The number of WaitHandles must be less than or equal to 64. (System.NotSupportedException)
Error       at System.Threading.WaitHandle.WaitAny(WaitHandle[] waitHandles, Int32 millisecondsTimeout, Boolean exitContext)
Error       at System.Threading.WaitHandle.WaitAny(WaitHandle[] waitHandles, TimeSpan timeout, Boolean exitContext)
Error       at Veeam.Backup.Service.CJobStateTracker.TrackJobs()
Error       at Veeam.Backup.Service.CJobStateTracker.RetryableJobsTrackingCallback()
Case number is 5162899. This was a merged case from other issues, but is currently being worked on for this issue.

What we see as symptom of this is jobs not starting as scheduled, and retries not working. The current workaround wed have is rebooting the Veeam server daily when backups are not running. We are waiting on word from development on if a fix is possible, but the current word is it could be a .net limitation.

Sethbartlett
Expert
Posts: 282
Liked: 25 times
Joined: Nov 10, 2010 6:51 pm
Full Name: Seth Bartlett
Contact:

Re: v6 Limit on Number of Jobs (64)?

Post by Sethbartlett » Jan 09, 2012 4:18 am

Yeah does seem to be a limitation of System.Threading: http://msdn.microsoft.com/en-us/library/tdykks7z.aspx

"This method returns when any handle is signaled. If more than one object becomes signaled during the call, the return value is the array index of the signaled object with the smallest index value of all the signaled objects. On some implementations, if more that 64 handles are passed, a NotSupportedException is thrown."

Not stating that something can't be done about it on how we design how scheduling works possibly but definitely a limitation of Threading/Mutex.
Skype: Sethbartlett88 - Make sure to label who you are and why you want to add me ;)
Twitter: @sethbartlett
If my post was helpful, please like it. Sometimes twitter is quicker to hit me up if you need me.

wolfgang.reisert
Novice
Posts: 6
Liked: never
Joined: Jan 26, 2010 7:41 am
Contact:

Re: v6 Limit on Number of Jobs (64)?

Post by wolfgang.reisert » Jan 09, 2012 3:00 pm

At the moment we have 76 jobs running in v5 without any problems. Does this realy mean that we have the reduce the jobs below 64 when upgrading to v6? We are running .net 3.5SP1. When this should be a .net limitation but v5 is running without problems can I expect that v6 will run in the same way?

Sethbartlett
Expert
Posts: 282
Liked: 25 times
Joined: Nov 10, 2010 6:51 pm
Full Name: Seth Bartlett
Contact:

Re: v6 Limit on Number of Jobs (64)?

Post by Sethbartlett » Jan 09, 2012 3:30 pm

Are you running 76 jobs all at the same time? This is more of a running all these synchronously.
Skype: Sethbartlett88 - Make sure to label who you are and why you want to add me ;)
Twitter: @sethbartlett
If my post was helpful, please like it. Sometimes twitter is quicker to hit me up if you need me.

wolfgang.reisert
Novice
Posts: 6
Liked: never
Joined: Jan 26, 2010 7:41 am
Contact:

Re: v6 Limit on Number of Jobs (64)?

Post by wolfgang.reisert » Jan 09, 2012 3:47 pm

No, the 76 jobs aren't running all at the same time. I schedule the jobs in that way that not more than 5-7 jobs are running at the same time.

Sethbartlett
Expert
Posts: 282
Liked: 25 times
Joined: Nov 10, 2010 6:51 pm
Full Name: Seth Bartlett
Contact:

Re: v6 Limit on Number of Jobs (64)?

Post by Sethbartlett » Jan 09, 2012 3:51 pm

Gotcha, this is a limitation of running 64+ jobs all at the same time:) Takes a VERY beefy Veeam Box haha.
Skype: Sethbartlett88 - Make sure to label who you are and why you want to add me ;)
Twitter: @sethbartlett
If my post was helpful, please like it. Sometimes twitter is quicker to hit me up if you need me.

wolfgang.reisert
Novice
Posts: 6
Liked: never
Joined: Jan 26, 2010 7:41 am
Contact:

Re: v6 Limit on Number of Jobs (64)?

Post by wolfgang.reisert » Jan 09, 2012 3:56 pm

Ok thank you. 64+ jobs is really a bit heavy.

tsightler
VP, Product Management
Posts: 5405
Liked: 2230 times
Joined: Jun 05, 2009 12:57 pm
Full Name: Tom Sightler
Contact:

Re: v6 Limit on Number of Jobs (64)?

Post by tsightler » Jan 09, 2012 4:10 pm

Jeff,

I have a couple of clients that have more that 64 jobs, in some cases >100 jobs, but none of them are attempting to run them all at the same time, they simple stagger the start of the jobs so that only 20-30 are ever in "started" state. Are you saying you have this issue even if you stagger the schedule?

jkcouch
Enthusiast
Posts: 30
Liked: never
Joined: Jun 26, 2011 7:02 pm
Full Name: Jeff Couch
Contact:

Re: v6 Limit on Number of Jobs (64)?

Post by jkcouch » Jan 09, 2012 7:08 pm

We haven’t tried that Tom. We stagger them, but very close together so that we get the maximum use of the queuing abilities of v6. I will try scaling back the schedule some to see if we can keep it under that 64 count. I will let you know how it turns out.

Good suggestion!

Gostev
SVP, Product Management
Posts: 24656
Liked: 3471 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: v6 Limit on Number of Jobs (64)?

Post by Gostev » Jan 10, 2012 2:42 am

Yes, this limitation is for concurrent jobs only, not total amount of jobs. My plan is to "enlighten" the job scheduler with the knowledge of this OS limitation, so that it does not try to start more than 64 jobs at once - but I did not discuss the possibility of this with R&D yet.

jkcouch
Enthusiast
Posts: 30
Liked: never
Joined: Jun 26, 2011 7:02 pm
Full Name: Jeff Couch
Contact:

Re: v6 Limit on Number of Jobs (64)?

Post by jkcouch » Jan 15, 2012 4:50 am

I can confirm that backing off the concurrent job count below 64 fixes the issue. Thanks guys for the clarification. I would like to see this fixed so we can queue up as many jobs as we like, sense I see that as the key to making our backup jobs complete as fast as possible. With v5 not having queuing capabilities, we had hours of dead time between backups. I don’t want to do that again :).

tsightler
VP, Product Management
Posts: 5405
Liked: 2230 times
Joined: Jun 05, 2009 12:57 pm
Full Name: Tom Sightler
Contact:

Re: v6 Limit on Number of Jobs (64)?

Post by tsightler » Jan 15, 2012 2:58 pm

Thanks for the verification Jeff. I'm quite sure that development will address this issue, although admittedly it might be a simple "64 active job" limit for the immediate workaround.

Post Reply

Who is online

Users browsing this forum: No registered users and 22 guests