Backup copy jobs fail after "no copy" window, start at wrong time

[CofIAlan]

I recently redesigned our backup architecture to take faster backups (18 hours down to six), allow faster restores, keep our "long-term" backups where they were, and introduce tape-based air-gapped backups (thanks to Gostev's regular horror stories). The problem is that I did all this to an extremely stable and reliable system, so naturally I wrecked everything and had to learn how to make this all happen despite my attempted attention to the docs. I worked out most of the bugs but I still have one very obnoxious, though not damaging, problem left.

Every Monday morning I get email alerts from each one of our backup copy jobs that for every server "Incremental copy was not processed during the copy interval." These messages, and the corresponding logs, arrive just after 6:00 AM local time even though the backup copy interval starts at midnight. Every other day of the week the jobs are successful and they show they started at midnight as programmed. I think this is a quirk of backup copy jobs in our specific configuration but I'm hoping there's a way around this. (I've read what I could find and the general answer to similar problems seems to be "disable the copy jobs with PowerShell.") For these reasons it seemed more appropriate to ask the community for input rather than thinking it's a defect and needs to go to Support.

My Setup
Primary backup jobs: Monday through Friday I take overnight incremental backups of production servers. Two backup jobs for mission-critical systems take full backups nightly, as these are replicated off-site through a special independent process. On Saturday we run full backups of everything, including our test systems and a couple other little things that almost never change but would be inconvenient to lose.

Backup copy jobs: Configured with daily backup intervals that start at midnight. Monday through Friday there is a no-copy window from midnight to 6:00 AM to let incremental backups run. Saturday has a 24-hour no-copy window to block all backups from transferring while full backups run. Sunday has a 24-hour copy-permitted window to ship those full backups. The copy jobs are configured to read full weekly backups from Saturday rather than synthesizing them. We don't use full GFS because we don't have those RPOs.

On Sunday the copies start running and seem to transfer the correct full backups taken on Saturday. The jobs all complete without error well within the one-day copy interval. Then, on Monday, the jobs don't restart at midnight but instead all say they started at about 6:00 AM when the usual no-copy window expires. They all fail, email me, and then the real interval starts and Monday's incremental backups get copied as expected. From Tuesday forward the intervals start at midnight, the jobs say "Waiting for permitted network usage period," and then data starts moving at 6:00 AM as expected.

I'm trying to understand what's going on and whether there's a way to keep things "vanilla" (that is, not have PowerShell controlling things) but not have this error. I'm guessing it's related to not handling Saturday quite right, starting immediately into Sunday, and then not understanding there are no incrementals on Sunday and only fulls from the day before. When Monday tries to run the interval is messed up from Sunday and only recovers when allowed to copy data for some reason, instead of resetting at midnight on Monday after success on Sunday.

Have I missed something in my setup? Is there something I can tune to capture the different kinds of backups I need in the backup windows I have? Some of this schedule is driven by limited bandwidth to the copy target repository, but that's not getting fixed for about a year.

Or, do I just need PowerShell to disable the copy jobs on Saturday and then resume them all on Sunday? That's kind of my Plan B if it's not built in to Veeam itself.

Thanks!
Alan

[Gostev]

My first reaction is that you're still on the pre-release Update 4 build. Can you verify you have the GA version?

[CofIAlan]

You know, I probably should have said that I'm actually on VBR 9.5 Update 3. I haven't rolled out Update 4 yet, as I didn't want to accidentally create new problems for myself between U3 and U4 while I was ironing out this major change to our process.

[Gostev]

Oh, I see. Then please open a support case, as debug logs review will be required to see what makes the job stop at 6am. I assume you already checked all obvious stuff like allowed data transfer window in their settings, or tape jobs stopping them at this time?

[CofIAlan]

Thanks Gostev. I opened case #3438117 and will report back when I hear more, for anyone else who finds this in the future.