Staggering start times

[LeeMackie]

I have recently made some changes within our Microsoft 365 environment and as a result we now have a large amount of jobs starting within the same hour.

I have staggered the start times based on the minutes option, but ideally I need to stagger the start times across the environment within the 8 hour window (having 70+ jobs kick off in the same hour is far from ideal, let alone the overlap of jobs within the same organization).

There doesn't seem to be any way to do this either via Powershell or console - it seems your locked in to starting within the hour of which the job was created regardless of whether this is desired or not. Does anyone have any suggestions or workarounds for this limitation?

[Mildur]

Hi Lee

Thanks for the request.
We got this request from time to time to allow a higher offset. I shared your topic with the team.

Does anyone have any suggestions or workarounds for this limitation?

There is currently no workaround besides creating more jobs. 3 daily jobs every 8 hours instead of a single one scheduled periodically.
Thread Settings on the proxy server will limit how many objects are processed at once per proxy server. This may help to reduce the load on the server:
https://helpcenter.veeam.com/docs/vbo36 ... tml?ver=70

Best,
Fabian

[LeeMackie]

Hi Fabian,

Thanks for the reply - We have split our jobs from a single backup job to 3 seperate jobs for each tenant executing every 8 hours but as I had to modify a large amount of jobs I scripted the change and hence now they all kick off at the same hour (with the offset in minutes between them that was randomly set via the script).

Hope to see something in the future.

[LeeMackie]

Sorry to bring up an old thread - just wanted to confirm if there is a registry key that can silence the alerts related to job overlap?
Specifically:
Team posts are already being processed by another job

[Polina]

Hi Lee,

Sorry, but there's no such option. You need to either reconfigure your jobs to make sure they don't include objects that can overlap, or update jobs scheduling to run them at different times.

Thanks!

[LeeMackie]

Thanks Polina - as the jobs are set to run periodically every 8 hours there is no ability to change the timing that they run as per the above messages. The jobs that overlap run for more than 1 hour so just manipulating the minutes is not sufficient unfortunately.

[Polina]

Makes sense. We'll consider extending the offset period in future releases.

[MaartenA]

Or add the option to set a start time when setting the run every x hours would be even better

[kirderf]

I would like to add a vote for this feature.

We currently have 70 jobs, and this will probably almost double by the time we have enabled protection of all objects. The large amount of jobs is needed due our object count. We create jobs automated using scripts and randomly sets the offset. Best case scenario is that we have two jobs starting every minute (~120 jobs), when we are limited to a 1 hour offset.

As others have mentioned, I think there's two solutions to this:

1. Allow PeriodicallyOffsetMinutes to have a value between 0 and (60*PeriodicallyEvery-1). I guess also some equivalent would be needed for situations where someone chooses Minutes5 through Minutes30.
2. Keep PeriodicallyOffsetMinutes between 0 and 59, but allow for setting DailyTime in combination with Type=Periodically.

[Gostev]

Fredrik, once you switch to V8 you will no longer need to create multiple jobs to work around scalability problems, so this should remove the very issue.

[LeeMackie]

I don't think that's true at all Gostev?

If you are an internal org with 1000 users - best practice would still dictate you don't want to run a single Exchange backup job with those 1000 user mailbox objects as the backup time could be obscene depending on the amount of change you're seeing. I would think you would probably want to break it into 5 jobs at a minimum, this also helps prevent one mailbox with an issue hanging up your 1000 mailbox job for 2 weeks (seen this way too often, and still see it from time to time).

This also doesn't help people like ourselves in the SP land with 500+ organizations in a VBO pod all kicking off within the same hour.

So from my perspective, version 8 has done nothing to help with this issue.

[Gostev]

Your perspective is wrong. The whole design goal for the new V8 architecture was to remove the need for customers to split large orgs into multiple jobs for scalability and performance reasons, as well as to ensure service providers can seamlessly process large number of client orgs with a single VBO server (if I remember correctly, our QA tested up to 300K mailboxes per server). Remember Veeam is also a VBO service provider these days so we need all these enhancements just like you to reduce our COGs. Having to manually split large orgs into multiple jobs, or to micromanage start times of those many jobs is simply not sustainable... our VDC team really struggled with that on V7.

[edh]

Your perspective is wrong. The whole design goal for the new V8 architecture was to remove the need for customers to split large orgs into multiple jobs for scalability and performance reasons, as well as to ensure service providers can seamlessly process large number of client orgs with a single VBO server (if I remember correctly, our QA tested up to 300K mailboxes per server). Remember Veeam is also a VBO service provider these days so we need all these enhancements just like you to reduce our COGs. Having to manually split large orgs into multiple jobs, or to micromanage start times of those many jobs is simply not sustainable... our VDC team really struggled with that on V7.

Is this true? As best pratice we understood that we need at least 4 jobs for each task, (Exchange/OneDrive/Teams/Sharepoint) maybe this is a missconception that comes from v7 architecture. Can any developer confirm that v8 can handle jobs with thousand objects (+10K objects) in a single job without problem? In our case our proxys got 240cores. Right know splitting the jobs cause us a cosmetical issue at Restore Portal where it show several restore points.

[Gostev]

Wow, Manuel... am I not developer enough for you any longer?

[kirderf]

Fredrik, once you switch to V8 you will no longer need to create multiple jobs to work around scalability problems, so this should remove the very issue.

I am already on v8, but I was not aware of this change. For our case it is not possible to combine everything in one job, as we need to split the objects in our tenant due organizational and legal reasons. So, for us at least it would still be very nice if we could stagger jobs more. As I mentioned before, we create all our jobs automatically and dynamically with scripts, so there is no micromanagement. Having an offset greater than 59 minutes would just mean that the automatic job creation can spread the start times more evenly. We can, and will, of course still benefit a lot from the ability to having larger jobs. I won't speculate on how many jobs we can remove, but I guess at least 50% can be consolidated into larger jobs. So don't get me wrong, this is a very welcome development.

You should probably update the best practices guide to v8 then. I see that current one is only applicable to v7, but that's a small detail many people may not notice. The currently available information online still shows the old limits of 5000 users per job, 20000 objects per proxy etc.

Gostev, is the recommendation of splitting object types (Mail, Sites, OneDrive, Teams) into separate repositories still valid, or is this also history with v7?

[Gostev]

Also history! The optimal split ito separate tasks is automatic within the engine itself (and fully transparent).

@edh just heard back from QA on your query, they tested max 40K objects per job .

[Gostev]

For those who didn't upgrade to V8 yet, be aware we're planning to ship V8.1 before the end of this month.

[karsten123]

Thank you Anton for the hint. May you spoil/ tease some expected features?
Is it worth to wait?

[Gostev]

AFAIK it's mostly polishing around large environments support based on our learnings after deploying V8 into Veeam Data Cloud. And particular focus on upgrading V7 installs with huge backup repositories, fixing some timeout issues.

Anyway, the hidden meaning of my comment was "if you did not upgrade to V8 already, wait for V8.1 so you don't have to upgrade twice in a short period of time". From that perspective it is worth the wait.

[pr_osit]

I have 3 jobs per client, split to mailbox, sharepoint and one drive, some of these are over 10k objects, and I use a custom powershell script to get all jobs (we have over 600 jobs) and start them in batches of 10 at a time, and start a new batch every 5 or 6 mins, we run 3 backups a day (every 8 hours) - just be sure to disable the schedule on the jobs so Veeam doesn't run them directly. So far this has proven very reliable and works well for such a large number of jobs, rather than trying to start them all at once.