Stopped by job?

[namiko78]

Hi guys, maybe i'm not doing this correctly, but I want to create a monthly tape full backup of everything for offsite storage. I created the job, and targeted one of my backup jobs. Started the job, and while it was running i manually started the backup job to see what would happen (expecting it to wait for the tape job to finish) instead it killed the tape job and said "stopped by job"

Is there another way to do this? I'd rather not pull it from the offsite backup copy location, but would that have different results?

Thanks

[Dima P.]

Hello Steven,
Since the backup files become locked by the backup to disk job – backup to tape was suspended. The best approach so far is to properly configure the schedule for tape job. You can use either ‘after this job’ or ‘as a new backup files appear’ triggers with the enabled ‘If some linked backup jobs are still running, wait for up to … minutes’ option. Here is a detailed explanation from the Help Center: Define the Job Schedule. Does it work for your case?

[veremin]

Hi guys, maybe i'm not doing this correctly, but I want to create a monthly tape full backup of everything for offsite storage

Do you want to copy a backup chain as a whole or just the latest full backup? In the latter case, synthesized full backup should be your way to go. Thanks.

[namiko78]

The linked option is only helpful if my backup job is running first, i suppose i could set it to run after the main job has started, and hopefully it finishes before the next scheduled run kills it. There should be an option in the main job to wait for the tape job to finish.

Regarding the synthesized backup, I'm not really clear on what this does, even after reading the help. I want a monthly full on tape, but from a forever incremental chain that runs daily. what option should i use?

Thanks

[veremin]

There should be an option in the main job to wait for the tape job to finish.

You can use pre-job scripts for that.

Regarding the synthesized backup, I'm not really clear on what this does, even after reading the help. I want a monthly full on tape, but from a forever incremental chain that runs daily. what option should i use?

This functionality synthesizes a full backup file out of existing backup chain directly on tapes. Using this functionality, you can create full backup directly on tapes on monthly basis. That's what I'd recommend for you.

Thanks.

[namiko78]

Any examples of these scripts?

So for the synth, if i want to do my full to tape on the first of the month, should i set the sync to be the day before, repeating once a month?
Thanks

[veremin]

Something like a waiting loop within which a job state is checked should answer your requirements:

Code: Select all

$TapeJob = Get-VBRTapeJob -name "Name of tape Job"
do{
Start-sleep -s 60
$State = $TapeJob.LastState
}while ($State -ne "Stopped")

Thanks.

[namiko78]

Thanks!

[veremin]

Since it's just a rough example, feel free to test and/or modify it before implementing. Thanks,

[chefkoch]

Hello,

i'm currently evaluating Veeam Backup & Recovery and i'm having the problem,
that my disk 2 tape jobs don't finish because they get stopped by the regular backup jobs.

The job log of the tape job just states

17.02.2015 01:34:56 :: Error: Stopped by job 'Backup Job 2' (Backup)

In the Job Log of Backup Job 2 there is no mention of the tape job getting cancelled.

Is there a setting to prevent this?

[chefkoch]

My google fu failed me before, but now i found this

http://forums.veeam.com/post132945.html ... ed#p132945

[veremin]

Yep, you can use pre-job script to avoid this behaviour. Otherwise, schedule jobs more carefully, so that they don't overlap.

[Steen]

But if You are have a Tape-job that is backing up a repository (or several repositories)? Then You suspend all backup job that is in that repository?
We are backing up about 40 TB which takes several days and use incremental backup-jobs and when tape-job is processing the .vbk the backup-job is starting to backup and then it will merge the .vbk -file and the whole tape-job stop for the .vbk and all others thats in the repository and the rest of the repositories!

[veremin]

But what would be the desired behaviour in your opinion? Tape job can't copy a .vbk file that is being merged, because it's not the same as it has been at the time when the tape job took place.

In order to avoid that you can add a waiting loop to backup jobs, so that, they wait till tape jobs finish processing. Another option is to schedule jobs more carefully to avoid interference. Finally, you can switch backup jobs to forward forever incremental mode and let backup jobs create virtual synthesized full backup.

Thanks.

[mkreitzer]

Synthesized fulls do not seem to make a difference. Scheduling to avoid overlap is difficult when trying to copy a very large full from a backup job that otherwise runs daily. Ideally Veeam would "snapshot" the full for the duration of the tape job and merge all of the changes at the end so that the daily schedule can continue on the primary job while the tape jobs continues to run. Dynamically updating the pointers in the synthesized full could also work. In the meantime, unless there are other suggestions, it looks like delaying the primary job with a script is the least worst option.

[Dima P.]

Thanks. In the upcoming v9 version the retry logic and ‘wait for the linked backup job’ mechanism will be enhanced, so whenever the linked job is running tape job awaits (at checks another source in the list). Stay tuned for the release and I hope you will find the new behavior more convenient.

[mkreitzer]

Thank you for the info! v9 seems to have a lot of capabilities I'm looking forward to. Another one that could help with this scenario would be parallel tape processing. However, based on various forum posts and the blogs entries I'm concerned that v9's parallel tape processing might not solve one corner case I have to deal with. For example, one VM I have to back up contains around 20TB of compressed data. We take daily incrementals, and I need to push the entire thing to tape once a month. The backup files are even larger than the VM, and pushing that much through a single tape drive takes quite some time. Can parallel processing work with a single VM backup, or does it require multiple chains to work correctly?

[Dima P.]

I am sorry to say that but parallel processing will be possible ether between simulations tape jobs or between VM chains in one backup to tape jobs.

[mkreitzer]

Crud. Two steps forward and one step back. Thanks for the clarification.

[Dima P.]

Two steps forward and one step back

We wanted to cover the case like yours but while designing the feature we came to the conclusion that spreading every single backup file between multiple media will make restore process a nightmare (imagine increment per tape situation) and cause a huge media overrun, thus we stick to the idea of keeping the backup chains within one media set.

[mykl_74]

I'm having this problem as well. Had an idea... I have 2nd copy backups on-site. Is there any reason why I wouldn't want to use them as my backup source for the tape job? I can disable the 2nd copy jobs while the tape job is running, so the files do not change while the job is in progress.

[Dima P.]

Mike,

Proper scheduling is a better solution. Just make sure tape and source jobs do not overlap (or use 'start after this job' scheduling option for backup tape).

[mykl_74]

Hi Dima,
That is true, but like others have mentioned this is a huge amount of data. I cannot schedule it to not overlap with the daily jobs because it takes more than 2 days for my tape job to finish. So, my question is, is it technically OK to do it this way, since it is really my only option. I would rather take a backup to tape of the original files, but in my situation that is not possible.

[veremin]

It's technically ok to stop source job for the duration of secondary one. Thanks.

[einhirn]

Hello!
Should I add to this thread or rather create a new one?

I'm also having issues with primary jobs causing tape jobs to fails with "stopped by job". We're using Veeam 9.5U4a.

I've configured one tape job that backs up multiple VM backup jobs and my Veeam agent repository. The synthetic full backup job takes up to and sometimes over 24h, so it'd be great to have the requirements of the primary jobs coordinated with those of the secondary job - like "primary needs file locked?" - "Secondary stops working on the task that prevents the lock, marks it for retry and skips to the next task".

Of course it'd also be great to estimate time to task completion so that we don't kill tasks at 95% with only maybe 5 Minutes to go - but instead just let the primary wait for that time.

I've seen a message "source files have been modified, retry required", but I also often see "<Tape job> stopped by Job '<some Agent deciding to push a backup just then>'"...

Thanks for your consideration...

einhirn

[lyapkost]

Hi einhirn.

Please consider enabling 'Prevent from being interrupted' option in your tape job. This option prevents source job from modifying the backup chain being processed by tape job.

[einhirn]

Hello lyapkost,

that option won't do for me, because that would halt my primary backups for up to 24h or more when the synthetic full tape job is running. Again, it seems to me that I'm thinking with "Tasks or Objects inside a job"-granularity locking etc. while Veeam only thinks in Job-granularity locking etc.

If "prevent from being interrupted" would only make the primary job wait until its own backup chain (in case of Agent backups) is transferred to tape, it would be better - but, thinking of a multi-chain VM backup job it would be even better if the granularity was even finer. I don't want my primary vm backup job to wait for the tape job to finish (with every single one of ~400 vm chains and ~80 agent chains).

Veeam tape wouldn't need the "prevent from being interrupted" option, if the tape job would just graciously fail sub-tasks it cannot perform because of changing data without failing the whole job. Retrying either that sub-task (better) after everything else is done or the whole job (not so good but still an improvement over the current situation) would be really great.

Please think of this post as a feature request or even bug report - depending on where one stands, the current behavior could very well be thought of as a bug...

Thanks for your consideration!

bye
einhirn