Feature Request: More flexible schedules

[TheOnlyWizard17]

Hi,

Right now, Veeam's scheduling options are quite limited. While you can choose between different ways of scheduling, there's no way of combining these. Also, there's no way of scheduling the maintenance things, like HealthChecks and/or Defrag-runs. Right now, they just run on the existing schedules. Suppose you would want to backup 4 times a day, every weekday. But in the weekends you want to leverage the increased production idle-time to backup only once a day, and have the Job do a healthcheck after that. Right now, there's no way to schedule it like that. We've got a 5-node cluster, 5 jobs (one per node), and the only option I'm left with right now is to spread the HC-schedules over "days". So, 5 jobs, means, 5 days of HC's, which is effectively the almost the whole week, and ofcourse sits in the way of the "regular weekday" 4-times schedules to do backups. To get around it, I'm now scheduling HCs 2 jobs a day, which still means one job is left out during the weekend and runs on mondays. Another problem with this is, because you can't schedule the HC at all, except which days it will run, you can't tell it when during the day to do that. The result is that, backing up 4 times a day, at 8:00,12:00, 16:00, 20:00, the HC will start at the first run 8:00, which ofcourse is not really what I was looking for. I have no way of scheduling it on the last run at 20:00, so it could take all night to complete it's HC.

[Gostev]

Suppose you would want to backup 4 times a day, every weekday. But in the weekends you want to leverage the increased production idle-time to backup only once a day, and have the Job do a healthcheck after that. Right now, there's no way to schedule it like that.

There's definitely a way to set this up using the backup window control. You just block out the entire weekend, except for a slot where you want your one backup to be taken. So, Veeam has a much more flexible scheduling than you think

[TheOnlyWizard17]

Doesn't work, because it's supposed to do HCs in the weekend after that first "weekend-backup", which don't run if you only "enable" say the slot of 8:00 on saturday, and have the rest "disabled"... We (support and I) tried that...

[Gostev]

Specifically what I quoted and said should work just fine, and there are currently no known issues with this functionality. So if that does not work for you, then your support engineer should escalate the case into R&D for further troubleshooting - as skipping a health check after the [first and only] weekend job run would be a bug in case a health check is scheduled to run on the given day.

[TheOnlyWizard17]

It's not a bug. Veeam applies logic (as it does with for example a compact aswell) whether it should actually do the scheduled HC/compact/Copy/whatever or not, based on a number of factors. Hence, you, the user, don't have absolute control over the set schedule if Veeam "decides it's not necessary" leaving you, the user, wandering why it didn't happen (it doesn't state the reason for skipping it in the last backup-report for example). The reason for my request is not this perse, but purely to point out that being able to schedule a certain action absolutely would give the user more control over when what will happen (especially in the light of lighter used hardware and thus longer runtimes this could be quite handy). Simply have a complete schedule-option, like the backup-schedule-option, for each of those different actions. If Veeam then starts a HC/compact/whatever, and decides "it's not necessary", simply report that and action done. If not, it will then do exactly what was scheduled (and as such anticipated to run at the schedule). With a HC in particular, it will only run post-backup, and it will only check the last restorepoint (which was just created pre-HC backup).

[Gostev]

This would not be possible with the current architecture, where HC/compact is an integral part of the job, as opposed to some standalone process that can be scheduled and executed at will. But again, you should be able to achieve the exact behavior outlined in my quote above with the current product version.

[rlspeeks]

I second what @TheOnlyWizard17 is saying. I've wanted better scheduling in Veeam for the past year. It is one of my biggest complaints about Veeam. Quest Rapid Recovery had better options as far as scheduling maintenance and scheduling. Veeam is better for backups though. : ) (Just wish you could improve the scheduling)

[TheOnlyWizard17]

Gostev, here you can see what I mean...
5 of 6 jobs are still running over the weekend...
Job 1 Hyper-V Backup 19 Health Check (59% Done)
Job 2 Hyper-V Backup 20 Health Check (83% Done)
Job 3 Hyper-V Backup 17 Health Check (75% Done)
Job 4 Hyper-V Backup 22 Health Check (97% Done)
Job 5 Hyper-V Backup 18 Health Check (99% Done)
Job 6 Hyper-V Backup 1 Stopped
Jobs 1,2 are set to do HC on Fridays
Jobs 3,4 are set to do HC on Saturdays
Jobs 5,6 are set to do HC on Sundays
Also, this weekend, these Jobs failed (artificially, I admit, I canceled them)
Nevertheless, Veeam decided to just run them next schedule, which is today 8:00. So, instead of having the HC's spread over the weekend as the repository can't really handle this total of 6 Jobs, 35+TB all at once, I now have them running all at once on the wrong day and performance has almost come to a complete stop because this way it's overloading the repository.
Aside from the fact that I "artificially created" this situation, it's veeam's logic which is causing this "trouble" now. If I set HC to be done on a Friday, and it got missed (canceled, whatever), I'm not in control anymore of what runs when, if Veeam sololy decides to then "just run it on next available schedule". If you have one job it probably is not such a bad thing, but if you have multiple jobs which you explicitly set up to be spread like above, Veeam's logic messes up unintended !
I now do not have a way of getting out of this, unless I just manually disable HC for all jobs, re-enabling them manually again right before they should run. Otherwise, if I cancel them now, they will simply run again all at once on next schedule. This is why I would want better control of what you tell Veeam to do (or not). Same applies to compact-runs.

[Gostev]

You should look with support to see why HC started on the day it was not scheduled to run, once you cancelled the previous job run. This does not look expected, and would be a bug that needs fixing.

[TheOnlyWizard17]

Support says that's "normal". It does because it was missed previous time. Reality-case was that the previous run merges were taking to long, crossing the day-boundary, so Veeam missed the HC and thus decided to just run it anyhow on next schedule. I now just artificially re-created that situation manually to show the fact Veeam can run HCs/Compacts the wrong day, be it because merges took too long, the job failed for whatever reason, etc. Simply put: Veeam otherwise perfect logic is now working against itself because it deviates from the set settings where it sees fit. Having seperate schedules for seperate tasks would simply overcome these situations...

[Gostev]

I don't agree with support that it's normal for a job to run HC on an unscheduled day, whatever the reason is (previous run skipped or failed). It is too resource intensive activity to run on a random day. They need to review it with R&D.

[TheOnlyWizard17]

Well, please review Case # 04121675
What should I do ? I have no way of re-opening this case...

[Gostev]

I think the case is all we need to ask R&D to review this potential bug. @Egor Yakovlev please discuss this issue with R&D.

[Egor Yakovlev]

On it. Will update this thread as soon as I get more details.
/Cheers!

[Egor Yakovlev]

Hi Bart.

- QA has confirmed that Health-check can run on a day where it was not scheduled. Following a Health-check failure run we will initiate another attempt to run a health-check with next job run, until we get a successful health-check. Another example, if backup job is not scheduled on a day where Health-check is scheduled, it will be triggered with following backup job run. We will tune that behavior in future updates.
- Health-check scheduled on Day Z shall run after first job run of the Day Z no matter of backup window settings, so if it doesn't trigger for some reason in your environment, it shall be investigated with support.

/Thanks!

[TheOnlyWizard17]

You see, although understandable from a developer's point of view, this type of "logic" makes Veeam actually less reliable from a admins' or users' point of view. You set it to run some time, only to find out it didn't or is running at some point you didn't expect, or worse, like in my scenario-case, you DON'T want it to run. Splitting up functionality and giving each it's own robust scheduler and making sure VBR honors that schedule, would make that better, hence the request. Although in all fairness I have to add that that's much easier said then done. If these functionalities are now "embedded" in the Job-logic it would take considerable effort to change that, I understand. Nevertheless, I think it's worthwhile to seriously consider, given the idea that the way things are now gives the users less confidence he's/she's controlling a programs behavior versus not controlling it at all. And, such as in my case, you can't get the behavior you'd want, more specifically, avoid the behavior you DON'T want (like 5 jobs doing all HC at same time, even though scheduled spread out)....

[rlspeeks]

Again, I second what @TheOnlyWizard17 is saying. @Gostev even said " It is too resource intensive activity to run on a random day. They need to review it with R&D."
I'll say it again, another backup software I was using had it so that you could schedule items like HC separately/independently of the backup jobs. Therefore, controlling when you ran them and how they behaved. It is a sincere request that if there is any way possible, this be looked at as a future option. Thanks!

[Egor Yakovlev]

Thanks for feedback, Rebecca.

We do track ideas to improve that behavior in future versions and your voice is heard.

/Thanks!

[rlspeeks]

@TheOnlyWizard17 - on a side note, I was running into the same problem. What I have done to somewhat alleviate the issue is to only run the HC's on my COPY jobs, that way I'm not "tying" up my backup jobs. The COPY job is the same files as the backup job so essentially if the HC is good on the COPY job, the backup job "should be" ok as well. It was suggested to me to do that from support since I was having an issue of not getting backups b/c the HC was taking too long.
FYI

[Moopere]

@rlspeeks - I didn't think that was how HC's work. My understanding was that a HC checks for corruption in the backup and 'auto heals' any problems it finds by drawing from the original source if required. If thats right then you could end up with a corrupted primary backup but a good copy backup - I haven't tried this, does the HC present email/logs which make it obvious something was wrong therefore alerting you to manually do something about the primary backup?

[rlspeeks]

@Moopere - I imagine there would be some logs, however, I haven't necessarily had any issues yet. If this is NOT a good way to do this, I need to know. Again, I only went with this option b/c it was suggest by support and it does help my problem b/c some HC were taking a whole day and when that happens, then of course you're not getting a backup for that day and that is not acceptable. Which is why it would be nice to be able to tell a HC WHEN to run.
If running a HC on a COPY job is not a good "solution", I need to know so I can change it.

[TheOnlyWizard17]

@rlspeeks - For HC it's quasi okay to do, a copy job copies the existing backups, so if you then HC the copies and they're okay, it's very likely the "originals" that were copied are also okay. However, @Moopere is also right, in that HC will only have "auto-heal" functionality when done on the source-backups. This functionality doesn't "work backwards" from Copies back to originals so to speak. So yeah, you loose that part this way. btw., if you're source-backup gets corrupted, then copied, the copy-backup HC-job would notice and alert about it. But this is only true for backup-file curruption, ofcourse it does nothing for the data inside the source-backupfiles itself. If there's corruption in there ("logical corruption"), it wil happily hum along copying it unnoticed. Nevertheless, same is true for the source-HC-job, won't detect "logical" corruption either. Only way to reliably prevent that is to use Sure-backup which includes automated "restore-check" capabilities.

[rlspeeks]

Thank you for this information, then I am back to my original problem. I cannot have a HC running a FULL day on the actual backup job. Then I get no backups for that day. If someone was to come to me and ask me for a restore for that day - and I don't have it - I would be in trouble. What is the solution for this then? SureBackup?

[TheOnlyWizard17]

Well, my work-around (not solution) has been to split up the work in a few more jobs, so each of those would need less time to do HC on. Then configure each job to do HC on a different day, once a week in the weekends. It's still not completely satisfactory, mainly because of all the above I said. Veeam sometimes takes the HC scheduling "into it's own hands", depending on various (failure) conditions, which then messes up my set schedules anyhow. But for the most part, it works...
SureBackup would be the best way, yes, but it requires extra infrastructure, like a Hypervisor to "restore" to, extra networking requirements for the Virtual Lab and knowledge of scripting applicationgroups to do the actual "test and verify" each VM boots and comes up and running, although Veeam does provide some scripts for the most common like Active Directory, Exchange... You can have SureBackup do both: first check the entire VM storage (HC), then boot the VM and use scripts to verify it came up okay, so that would be the most sure way of having valid backups (hence the name SureBackup I guess)

[rlspeeks]

Exactly @TheOnlyWizard17 (in regards to a work-around). It would be nice to have something better.
On another note, it was bugging me where/who told me to do the HC on the backup copy job. I searched my tickets and see 2 tickets I submitted for HC taking all day, but nothing about that. So I searched my email and back in May I was assigned a Customer Success Manager b/c we were having so many issues. She assigned me a technician. It was this technician that suggested it. It's in my emails.....anyway. I guess back to the drawing board. Thanks for your post and bringing this issue up. Because it is an issue.

[TheOnlyWizard17]

Well, the technician was not entirely wrong, but perhaps he should have explicitly mentioned to you that, while it works for checking health, you loose Veeams ability to automatically resolve the detected problems...
In regards to the work-around, IMHO we should be able to tell veeam when to do what, because there simply will always be situations where the "embedded" logic fails or gets it wrong. It should not prevent us from telling Veeam how to what and when, even while for most situations Veeams' built-in logic is more than sufficient.
Hence the request