Backup Copy scheduling needs improvement

[Meyercord]

The scheduling features for Backup Copy Jobs have serious shortfalls that are causing some major headaches. This has been documented in case 00745075

Mainly, the problem revolves around the fact that Backup Copy Jobs can only be scheduled in terms of a "Copy Interval" of minutes, hours or days. While this probably works well to keep two repositories in sync on a continuous cycle, that's not the only use for a Backup Copy.

Take a straightforward example: A user wants to perform a Backup Copy to a remote repository once per week every Saturday.
The first problem the user will find is that there's no way to schedule the job to run on a specific day of the week. The copy interval starts on the day that the job starts. So in order to run on Saturdays, the job must be started on a Saturday. If the server is ever rebooted or the job is modified (which restarts the job), then someone has to log back in on Saturday to reset the schedule. If the copy job starts running on the wrong day and a Tape job is scheduled to copy those restore points, it will skip any that are on the wrong days.

Secondly, the destination file is locked for the duration of the copy interval, even after the data has been copied. So, in the example above, if a Tape job is scheduled to copy the duplicated restore points, it will not be able to copy the most recent restore point until 7 days after it has been written when the copy interval finally expires.

I'm aware that these issues can be worked around in some fashion by specifying a copy window to constrain the copy job to the desired day of week or by attaching a script to the tape job that stops and starts the copy job to remove the file lock, but these hacks add complexity and will likely create other issues and maintenance headaches down the road.

All of these limitations are caused by the unnecessarily constrained scheduling rules. Here's one way to improve it:
In the copy interval setting form, add a selector for the start *day*. This should allow selection of a simple weekday (monday, wednesday), a day of month (1st, 2nd, last), an ordinal and weekday (1st monday, last saturday, etc) or a specific date (Thursday, Feb 19, 2015). Also, users should be able to specify *how many* restore points for each VM should be copied into the backup file, and once this limit is reached, the job should close out the file wait until the next interval.

[foggy]

AJ, thank you for the feedback.

Secondly, the destination file is locked for the duration of the copy interval, even after the data has been copied. So, in the example above, if a Tape job is scheduled to copy the duplicated restore points, it will not be able to copy the most recent restore point until 7 days after it has been written when the copy interval finally expires.

This should not be the case. Actually, the files are released right after all the data has been copied and transform has completed. I would ask your engineer to take a closer look at this behavior, if this is indeed the case.

[Meyercord]

I can ask them to reopen the case, but in multiple tests I have attempted to duplicate a backup copy destination repository to tape, and have confirmed that the tape job will not back up files from the repository until after the copy interval has expired. After doing his own research, the engineer on my case told me unequivocally that this is by design and that the files in the backup copy destination repo cannot be copied by another job until the original backup copy interval has expired. This seems to be confirmed by the description in the job setup dialog, which states "Backup Copy job creates a new backup file for each copy interval, and starts copying the most recent restore point of each processed VM into this backup file immediately, or as soon as the new restore point appears in the source backup repository."

Presumably, the job keeps this new backup file open until the end of the interval. I don't know why it would do this (maybe in case more data shows up?), but its a real pain. If you have any sway to get a second opinion, I would sure appreciate it.

[foggy]

Not until the backup copy interval expires, but until all the required data is copied. There can be a situation, where, for example, two backup jobs are selected as source for the backup copy job. If one of the backup jobs completes first and backup copy job copies all the required data for it, it will still wait for the second source backup job to complete to copy its data. At that time the corresponding backup copy job restore point is still locked. But it is released as soon as data for all the source jobs is copied, no need for it to wait until the end of the interval.

[Meyercord]

I have 8 Backup Copy Jobs. Each copy job uses a single Backup Job as its source and all 8 point to the same destination repository. That repository is the source for the subsequent Tape Job. When the Tape Job runs the following day, all Backup Jobs are in Stopped state and the Copy Jobs show a status of Idle, but the tape job ignores the most recent restore point for all 8 copy jobs and writes only the restore points created during the prior week and earlier.

[foggy]

After digging into this more with QA, I can say that the behavior you're seeing is indeed expected. Though backup files are released on the file system and are not locked (for example, restore jobs and other backup copy jobs can process restore points prior the end of the interval), tape job uses another (internal) flag to define restore points to copy. And this flag is dropped only once the interval expires. We will research the ability to change this behavior in future.

[Meyercord]

Thank you, Alexander

[albertwt]

I have 8 Backup Copy Jobs. Each copy job uses a single Backup Job as its source and all 8 point to the same destination repository. That repository is the source for the subsequent Tape Job. When the Tape Job runs the following day, all Backup Jobs are in Stopped state and the Copy Jobs show a status of Idle, but the tape job ignores the most recent restore point for all 8 copy jobs and writes only the restore points created during the prior week and earlier.

Hi Meyer,

what software that you use to backup to tape ?

I'm using Backup Exec to write the Veeam Backup copy job from secondary repository but somehow I've always end up with .VIB files cannot be backed up due to the "lock" put in place by Veeam.

My goal here is to be able to perform Veam backup to Disk(1st repo in DC) --> to Disk (2nd Repo in HQ) --> to tape (from the same 2nd repo server to be sent offsite)

[foggy]

I believe AJ is using Veeam B&R tape functionality. Are you saying that no incremental files can be copied to tape, not just the latest one?

[albertwt]

I believe AJ is using Veeam B&R tape functionality. Are you saying that no incremental files can be copied to tape, not just the latest one?

Yes, that's my problem at the moment. Some of the .VIB cannot be backed up to the tape
Since I upgraded to VBR v8 this problem started to happening.

[foggy]

I've replied in your original thread about the issue, let's keep discussion there for consistency.