Continuous Tape Job Locks Backup file before backup merge

[IHeartCats]

Hi,

I had a quick search for this issue/behaviour but I didn't find anything.


Backups are forever forward incremental and run nightly and the continuous tape jobs are configured to do a synthetic full to tape once a week and to copy the incremental daily(or as often as they manage to run).

Recently I have observed the backup to disk job finishing copying/reading data and then issuing a delete snapshot to our storage. While the delete snapshot happens the backup file is temporarily unlocked and the continuous tape job jumps in and locks the backup file then starts copying data to tape.
Once the snapshot delete has processed on the backup job the next step would be to do the merge, however it now can't do that because the tape job now has the backup file locked. So the backup job sits there at 99% complete until the tape job finishes.

Has anyone got any ideas about how I could get the tape job to wait until the backup and merge is finished before it tries backing up to tape?

I'm concerned as if the merge is outstanding the job can't complete and then may not run that night or at the very least, delay the start time. However this wouldn't be the case if the tape job waited until the merge was done and then copied the incremental.

Please correct my understanding if I have got it wrong. Has anyone else come across this and worked around it?
Many Thanks

[lyapkost]

Hi Helen.

Normally tape job processes only the last incremental storage and doesn't lock the whole backup chain so the merge shouldn't be affected unless your backup job has less than 3 retention points which is strongly unrecommended. By the way, do you have option 'Prevent this job from being interrupted by source backup jobs' in the tape job selected or not? I would suggest to disable this option in your case.

As an alternative you may schedule the tape job to run after the source job finishes (in case you have several jobs processed by one tape job set it to run after the job which is the last to finish).

[IHeartCats]

Hi,

Thanks for your response.
So our backup chains have 8 points in them. (Forever Forward Inc Chains)

So the tape job does seem to be delaying the merge from completing by 1/2 a day. As I mentioned before the tape job seems to jump in and lock the file in between when the backup has finished transferring data and is waiting for our storage snapshot to be deleted. Then the backup job goes on to try and do the merge but the tapejob has locked required files. I wondered if it was perhaps the backup metadata file that the both the tape job and the merge need to access.

21/01/2018 21:31:31 :: Merging oldest incremental backup into full backup file
21/01/2018 21:31:32 :: Waiting for the required backup files to be released by another job: vbr2-Tape-2w-DFS

I had been using 1 continuous copy job to write out all of the data from 5 backup jobs. I wanted to use continuous so that when the first jobs finish (with the smaller VM's in) it can get on with transferring the data to tape and get that done in time for when the bigger jobs complete so it can get going on that (time is quite tight as some of these are large VM's). It had been working reasonably well until before xmas.

But for some reason now the tape job has started locking the backup file before the merge can start.

I currently do have the "Prevent this job from being interrupted by source backup jobs" enabled on the continuous tape job in question. Do you think it might be this option allowing the tape job to jump on the backup files before the job merge is done?? (I really need an option on the backup job that prevents the reverse (The backup job being delayed by the tape)

We do seem to be getting a backups to disk ok currently and mostly within the time window. Getting them to tape it a timely way is proving tricky.

I'll have a think and see if I can make it behave better by splitting out the jobs/schedules a bit more.

Do you think this is worth logging with veeam? I'm trying to work out if its a bug or just an unintended consequence of the options I have selected.

[lyapkost]

I wondered if it was perhaps the backup metadata file that the both the tape job and the merge need to access.

The .vbm file is not required for tape job since it's not copied to tape.

Do you think it might be this option allowing the tape job to jump on the backup files before the job merge is done??

No, this option makes source job wait until tape job finishes and delays merge. Try to disable it and see if any errors in tape jobs such as 'stopped by [source job name]' appear.

The situation when tape job starts copying new backups before the source job finishes is unexpected. But is is not an issue itself because, as I said, tape job shouldn't lock the whole chain but only the new files to copy to tape and merge should go without a hitch. So in your case I suggest you open a support ticket.