v9 GFS Job Schedule Questions

[ekisner]

Hi there,

Tape jobs are... unreliable. For example, if the tape job is running and backing up an endpoint backup, and the endpoint backup job starts, it will kill the tape job to unlock the endpoint backup (which to me seems backwards, but is not the purpose of this post).

The GFS job has but one start time. If said GFS job fails halfway through, one must manually start the job again to ensure that all data is on tape. Another example would be an active-full job being run later than usual, say for example because it failed the first go-around and the tape job already started (and went past it).

This was previously mitigated by the continuous option, which is no longer available for the GFS schedule.

[veremin]

Hi, Erik,

Not sure whether I follow you on that, can you shed more light on what doesn't work as expected in your opinion? The fact that tape GFS cycles don't retry itself, if failed or something else?

Also, I'm wondering what schedule option you refer to as continuous - "as new file new backup files appear"?

Thanks.

[ekisner]

Sorry, yes, I do mean "As new backups files appear".

The problem is this:
If a full backup takes place on say... Tuesday, where it was normally supposed to run on Friday (which has definitely happened, especially during times where support tickets are involved) then it will not be in the tapes unless the GFS job is manually triggered.

I'm sure it would retry, at least a few times. But that wouldn't help if the tape job completes successfully and misses a file outright.

[Dima P.]

Thanks for bringing this up! I had a conversation with QA team and we’ve decided that endpoint job’s behavior in this case is not correct. Tape job should not be affected while endpoint job should fail: since Endpoint has 23-hour job retry cycle and will continue the backup once tape job succeeds.

I’d say GFS tape job is ‘almost continuous’: if it detects the incomplete restore point on disk or running source job it should restart itself every hour and check the source files. It should continue the same 'restart' procedure until the next scheduled start of the job (that implies non-endpoint jobs).

[ekisner]

Glad to hear the endpoint/tape functionality will be fixed! I agree that the tape job should be more important than the endpoint job. In an ideal world, it would simply tell the tape job to put the endpoint backup to the end of the list, and try it again before it finishes the job.

As for the "almost continuous" GFS, I've seen nothing of the sort. My active full jobs ran on Friday, the GFS job didn't start until Saturday. I manually stopped the GFS job Saturday night because it was using the wrong tapes (different forum thread on that subject) and it sat doing nothing. I did not disable the GFS tape job, I merely stopped it.

With the previous jobs, if it was set to continuous (which I find faster to write than "as new backup files appear", hopefully this doesn't cause misunderstandings) it would try, and try, and try, and try. On days where there was a persistent issue I'd generally get 40-60 or so emails from the B&R server over the course of the night. Unless it was stopped, then it went straight to disabled. If a full active job ran, as soon as it was finished, the tape job would start putting the VIB/VBK to tape.

[Dima P.]

When you stop the tape GFS job it should sit and wait for the next scheduled date (so it stop restarting itself). Let’s find out the root cause of the wrong tape being used in the weekly media set and then look across the logs once again to check it acts correctly when you stop the job. There is no need in separate case – just add time stamp when the tape job was stopped to the case. Thanks

[nikpolini]

I have just updated to V9, and have created a Tape job targeted at a GFS Media pool, I have set these to run on Tuesdays. However, I cannot set the exact start time. I want it at 8am, after all the night's backups are completed.(Reduce load on Primary Backup storage, as it is doing forever forwards) The job is scheduled to run at 12:00 AM.

Can I change this in powershell?

BTW, Loving the new interface/features so far.

[Shestakov]

Hello,

By design, the tape GFS job is running the whole day waiting for new restore points from the source.
Once it receives the new restore point, it copies the newest backup(creates a full) to tape. When restore points from all sources are copied to tape, the job is finished.
The behavior can not be changed in powershell.

Thanks!

[ekisner]

I will point out that I have not observed this behaviour. But then I've also not had a successful job run yet, so I only know that after it is manually stopped it gives up. I assume it will start looking again as described after the job is started and runs successfully?

For reference, the existing ticket which deals with our job needing to be stopped is 01667096, presently with R&D for hotfix development.

[Shestakov]

If job is stopped manually, it will not retry. If you want to finish writing you can manually start the GFS job and choose the same media set(weekly/monthly/quarterly/yearly), VBR will write to the tape only those backups which are not copied yet.
Hope my explanation is clear.
Thanks!

[nikpolini]

What happens if the tape job starts on Tuesday, writes a full of a VM to tape, and is busy with others, and a new incremental comes on for an already completed machine, dose it do it again?
I am backing up 50TB each week to tape via Dell TL4000 and it takes 2-3days, during which time there are new RP added to those VMs...

[veremin]

Nope, it should not do it again. Thanks.

[Steen]

Hi.
I dont understand why i cant choose start time for monthly.
It always starts at 00:00:00?
/Steen

[Shestakov]

Hi Steen,
You are correct, but design the tape-GFS job is running the whole day waiting new restore points and once it has them it performs a write.

[westCPS]

Hello!

We just update to v9.0.0.902 and found the new GFS Tape Pool Setting After we recreate our Backup Copy 2 Tape Job we saw that we only can Change the day but not the running start-time. Is there any info about when we can set up this, 'cause midnight isn't such a good time for that

Best regards Werner

[Shestakov]

Hello Werner,
We do receive the similar requests from time to time and thinking to implement backup window in one of the next releases.
Thanks!

[PenguinSSH]

Can we get a retry option when the job fails. Seriously, if a job fails in the middle of the night because it was stopped by backup jobs, we would need it to retry automatically. Otherwise, we're loosing 10-12 hours of tape transfer.

[Dima P.]

Hi PenguinSSH,

It should be retried automatically if any source job is locked/failed (or running): it constantly checking the source jobs and if the required job is completed – start placing the files to tape. Moreover, it can keep retiring till the next scheduled GFS date (for the source disk jobs that overlap the tape GFS schedule).

[PenguinSSH]

Hi Dima,

This has not been our experience so far with the GFS Tape jobs, if it fails because it is interrupted by a source backup job, it stays failed until we start it manually again.

[Dima P.]

Thanks for the heads up. We are going to check this behavior in our lab – stay tuned.

[Dima P.]

PenguinSSH,

You were right. If the GFS tape job gets interrupted by source jobs (for example, because of the merging process) – GFS job is going to be cancel. Thanks for bringing this up – we will discuss with the team how this behavior can be corrected.

[PenguinSSH]

Hi Dima

It seems to me that there are a couple of processes within Veeam that could benefit from this. If a process such as tape backup is running and depends on the backup files, the VBKs should not be altered until the dependency is completed. So a kind of queue in this process with priority should be helpful.

In other words, merging process and backup file check would have less priority over something like backup, tape backup or backup copy. Once those are finished, the lower priority process could be triggered.

In my opinion, with the different processes such as merging, backup guard, replication from backup files will make it harder for processes depending on VBKs to complete properly when those are lengthy or depend on the original VBKs.

Thanks for the reply and keeping us in the loop.
PenguinSSH

[Shestakov]

PenguinSSH, thanks for the feedback.
By default backup and replication jobs have higher priority than backup copy and backup to tape since first two take a snapshot and actually preserve VMs state, while second two "just" copy existing backups on disk or tape.
You may switch the source backup job to basic forward incremental with no merges to avoid the situation like you face now.

[F.Henne]

The case of interrupted tape jobs due to source backup jobs running merge is a problen for me too. The way I've solved it so far is to use the Job Scripts option and run powershell scripts for increasing/decreasing the source jobs retention policy by a few days. This puts the merge process on hold for a set number of days and at least makes it possible to have the tape job running over several days.

The scripts I use right now are simple dirty fixes for the problem in that they are are only increasing and decreasing by a fixed integer. A cleaner and better solution would be to read the retention policy that exists as of today in your jobs and then store those in clixml (Export-Clixml) for use in the decrease script. I'll leave the scripts below if anyone is interested.

A proper solution from Veeams side would be nice though for tape jobs and forever incremental. If possible it would be nice if the source backup jobs took into consideration any tape jobs running against the repository and skip or postpone any merge processes for later. Perhaps not the best solution for everyone but those us who's got space enough to hold 2-3 extra days of incrementals every week/month shouldn't have any problems with it. Seeing as it is the merge process that makes the jobs fail.

increaseRetainCycle.ps1

Code: Select all

Add-PSSnapin veeampssnapin

$jobs = Get-VBRJob | ? {$_.IsBackupJob -eq $True}

$jobs | Foreach-Object {
$options = $_.GetOptions()
$options.BackupStorageOptions.RetainCycles += 4
$_.SetOptions($options)
}

decreaseRetainCycle.ps1

Code: Select all

Add-PSSnapin veeampssnapin

$jobs = Get-VBRJob | ? {$_.IsBackupJob -eq $True}

$jobs | Foreach-Object {
$options = $_.GetOptions()
if($options.BackupStorageOptions.RetainCycles -gt 20) {
$options.BackupStorageOptions.RetainCycles -= 4
$_.SetOptions($options)
}
}

[Dima P.]

Fredrik,

Thank you very much for sharing. We definitely looking into the solution to let the tape jobs wait for the source disk job if it was interrupted by whatever reason.

[pkelly_sts]

I wonder if there's any scope for creating some kind of "Priority" flag, where we can choose (within reason) which jobs should have higher priority? Could also help with superceding GFS jobs if required (not sure if that side of things is working ok these days until I get onto v9)

[Dima P.]

Hi Paul,

Setting the source job priority is not currently possible but will be added in the upcoming releases.

[pkelly_sts]

Good to hear

[westCPS]

Thanks for all the informations.

Best regards
Werner

[lowlander]

Hi,

is it possible to schedule a GFS tape job @ a specific time. Now, the job always starts @ 12:00AM, while we want to fire it up @ 14:00PM.

Thanks !