Feature request: Scheduler options (overall)

[glennsantacruz]

Having reviewed multiple posts regarding best practices for B&R Server maintenance ( Windows patching, reboots, etc. ), and other posts in the forum requesting ability to pause/resume jobs, would Veeam consider adding a feature for overall scheduler options? Presently the 'safe' way to ensure a proper maintenance window for B&R is to ensure no jobs are running -- and to ensure no jobs are running, each job needs to be modified in some fashion ( disable, change schedule, etc. ). That modification can certainly be scripted via automation, but realistically this is a workaround to the actual problem. Alternatively, Veeam services could be stopped, effectively preventing any jobs from running. However, that has the negative impact of not allowing administrators to use the B&R console, and is also really just a workaround to the actual problem.

Why not have a feature to allow "overall" control of the Veeam job scheduler process itself? This would provide an ability to pause/resume the entire set of jobs from one location, and could be leveraged to establish maintenance windows ( either ongoing, or on specific date/times, etc. ). Alerts could be included in this feature, to send notifications if an on-demand 'pause' was issued without a 'resume' following in a configurable amount of time.

For us, this would be far more preferable than having to build/maintain scripts and workaround processes to stop jobs, alter job schedules, disable jobs, etc. In addition, any jobs that may have been disabled purposefully (for reasons other than maintenance windows) could lead to confusion if an administrator is tasked with manually altering jobs for an upcoming maintenance window, then having to remember which jobs need to be reverted to original settings when the maintenance is complete. Again, these are really just workarounds to the actual problem -- namely, no built-in ability to control the overall scheduler for B&R.

[Gostev]

So like "maintenance mode" for the backup server?

[glennsantacruz]

Yes, "maintenance mode" for the backup server to put it simply.

Ideally this would be configurable with a schedule as well as "on-demand". For example, we may want to schedule maintenance for an upcoming date/time, or we may want to schedule maintenance on a recurring basis. Providing an "on-demand" maintenance could allow an administrator to optionally stop (immediate or graceful) all running jobs, and have the effect of preventing any scheduled jobs from starting.

A scheduled maintenance could be written to take into consideration the time spent in earlier job runs and begin tapering prior to the window beginning. For example: suppose we have 10 jobs that are all scheduled to run at midday, and a maintenance window is defined to begin 30 minutes past midday. If those 10 jobs normally complete within 30 minutes (based on historical average of the job duration), then those jobs could be allowed to run. However, if those 10 jobs normally take 3 hours to complete, then the maintenance window start time will be delayed due to those jobs. I could see benefits of providing options to the administrators, allowing selection of "force running jobs to stop, either gracefully or immediately" as part of the schedule, or to "notify if maintenance window cannot begin due to running jobs".

[Gostev]

Thanks, it's clear now.

[micoolpaul]

+1 for this feature

[skrause]

Maintenance Mode would be a cool feature, I agree.

Especially if it would gracefully stop things like BackupCopy jobs in a way that they don't show up as "failed" in the last run status.

[glennsantacruz]

To add a few more "hey this would be nice" anecdotes to this feature:

* It would allow for clean maintenance of the Veeam B&R SQL server instance, whether that instance is co-installed on Veeam B&R Server or running as a separate server altogether. This includes maintenance of the server installation bits, as well as maintenance of the underlying database/index/object structures without fear of impacting running Veeam jobs.
* It would allow for clean maintenance of Veeam infrastructure servers ( proxies, wan accelerators, tape machines, guest processing, mount servers, repositories, etc. spread throughout an enterprise deployment )
* It would allow for clean updates/installations of Veeam product patches. It is important to point out that maintenance windows are not just specific to the underlying OS, but can include applications as well -- Veeam not being the least of those applications.
* The feature would not intrude on existing automation processes that may be in place for clients. If a client chooses to continue using workaround scripts/intervention to allow for 'server maintenance', then they could most certainly continue using workarounds for this missing feature. In other words, addition of the feature would not negatively impact existing client workarounds.

By "clean maintenance" I'm referring to a "known good state" of Veeam jobs/servers/services, such that maintenance could be performed on dependent components/services without fear of damaging or interfering with any running Veeam jobs.

[Aaron Wells]

Being able to schedule a maintenance window and gracefully stop any running tasks would be a great feature.
Adding SMTP notification emails for scheduled maintenance for customers who use our Cloud Connect & Service Provider Console would be awesome.

+1 for this feature from us

[mhunzi]

We also would like to have this feature and make maintenance windows easier to handle!

+1 for this feature from our side

[Samba222]

sounds nice ...
+1

[DanielJ]

This would be very useful.

'+1' if someone is counting...

[klingk]

I've been wanting this for years.

The closes I got was recently support told me to stop all the Veeam services and they would "gracefully" terminate any running jobs.

I've been doing this, but it still feels like there should be at the very least a "pause" button on the scheduler that would just not start a new job until resumed.

That way running jobs could finish and then maintenance could be done.

This is also important for patching proxy servers and data stores while it is paused without breaking any in process jobs.

Having a scheduled window would be great,but I can see that would be more difficult because it would need to prevent jobs from starting that would run too long and that is not always predictable but a pause would be simpler maybe ?

[nmdange]

You can just Ctrl+A to select all the jobs, right-click, and select "Disable" to prevent any jobs from starting. Then it's pretty easy to stop any jobs that are currently running if you don't want to wait for them to finish.

[AlexHeylin]

If this is going to get traction, please can I suggest VCSP needs are considered from the outset? Easier to make it cover all the use cases in one go, than write it and have to come back later to make the CSP (VCC) use cases work / not get gnarly. Thanks

[adelamora]

+1 for this as well.

Instead of it stopping/disabling all jobs, ideally it would simply pause or freeze every task in a state that can be resumed. Like how hypervisors can freeze a vm, reboot, and then resume a VM without the guest OS even knowing the hypervisor rebooted (well, sort of).

There really should be no reason why we cannot pause something that may take a long time to complete (like a synthetic full or merge) without having to cancel it.

[Gostev]

I can tell that all kind of resumes are in general extremely costly functionality from R&D perspective (and particularly the QC part). There's just too of a big potential for data losses and corruptions. However, I personally would be more worried about pausing jobs in light of snapshots, which will keep growing while the job is paused.

Easier to make it cover all the use cases in one go

Not really... 10x use cases means 10x longer dev and QC time required! I've seen features taking years to the initial release when a PM was trying to "cover all the use cases" right with v1. This is why moving in small steps is actually always preferred. It takes the same few years to arrive to the "cover all the use cases" state, but all these years some customers already benefit from a limited number of supported use cases, and are providing real-world feedback that helps to stabilize and mature the engine behind the feature. While without this, in a few years you finally get a feature that covers all possible use cases, but then still needs another few years to stabilize (because those resuming jobs corrupt your backup files etc.)

[neilvbell]

+1 for something along these lines, but don't necessarily need it to pause mid-job. One of the awkward aspects that I have is that when disabling all jobs for the maintenance, I have a few jobs that might be disabled for some other reason (currently working on freeing up repository space, fixing a backup issue, etc.). I now have to try and note down or remember which were disabled for this reason and should not be re-enabled. Maybe some other category for a job other than just disabled so that I can disable all but only re-enable those that were just disabled, not those that were disabled for another reason. Maybe a maintenance category for a job... dunno.

Regarding approach for getting maintenance time in for Veeam servers, could we maybe trigger something that says to disable all jobs, cancel any backup copy jobs in progress (as they can usually be picked back up easily and I don't want to wait on them) and send a notification email as soon as the current running jobs have completed. I am then not having to constantly check the progress of the currently running jobs, waiting for them to finish.

Kind regards,
Neil

[Gostev]

One of the awkward aspects that I have is that when disabling all jobs for the maintenance, I have a few jobs that might be disabled for some other reason (currently working on freeing up repository space, fixing a backup issue, etc.). I now have to try and note down or remember which were disabled for this reason and should not be re-enabled.

Yes, this use case is quite clear. Nothing awkward about this, actually it's the first thing I myself thought about when I saw OP's request

[18436572]

+1 for this feature.

I remember two Backup products that had a similar feature. One had a job queue and you could pause it. The other allowed you to select a running job and pause it. Then you could do your maintenance (patching etc) and reboot any of the servers. Then resume the queue or jobs.

Currently we have jobs that run constantly and maintenance can be an annoyance. I will multi-select all jobs and disable them.

[a-tome]

I agree, I really hate crashing backup jobs so I can patch servers, or so the server being backed-up can be patched and rebooted. Occasionally patches take longer than the planned maintenance windows. "Pause" would be a very helpful feature.

[18436572]

The ideal solution in my mind is allowing current jobs complete, but prevent new jobs from starting. Then an email to notifying that all jobs are complete.