Discussions specific to tape backups
ekisner
Expert
Posts: 194
Liked: 33 times
Joined: Jul 26, 2012 8:04 pm
Full Name: Erik Kisner
Contact:

v9 GFS Job Schedule Questions

Post by ekisner » Jan 16, 2016 9:44 pm

Hi there,

Tape jobs are... unreliable. For example, if the tape job is running and backing up an endpoint backup, and the endpoint backup job starts, it will kill the tape job to unlock the endpoint backup (which to me seems backwards, but is not the purpose of this post).

The GFS job has but one start time. If said GFS job fails halfway through, one must manually start the job again to ensure that all data is on tape. Another example would be an active-full job being run later than usual, say for example because it failed the first go-around and the tape job already started (and went past it).

This was previously mitigated by the continuous option, which is no longer available for the GFS schedule.

veremin
Product Manager
Posts: 16524
Liked: 1374 times
Joined: Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin
Contact:

Re: v9 GFS job - No more continuous?

Post by veremin » Jan 18, 2016 10:46 am

Hi, Erik,

Not sure whether I follow you on that, can you shed more light on what doesn't work as expected in your opinion? The fact that tape GFS cycles don't retry itself, if failed or something else?

Also, I'm wondering what schedule option you refer to as continuous - "as new file new backup files appear"?

Thanks.

ekisner
Expert
Posts: 194
Liked: 33 times
Joined: Jul 26, 2012 8:04 pm
Full Name: Erik Kisner
Contact:

Re: v9 GFS job - No more continuous?

Post by ekisner » Jan 18, 2016 3:36 pm 1 person likes this post

Sorry, yes, I do mean "As new backups files appear".

The problem is this:
If a full backup takes place on say... Tuesday, where it was normally supposed to run on Friday (which has definitely happened, especially during times where support tickets are involved) then it will not be in the tapes unless the GFS job is manually triggered.

I'm sure it would retry, at least a few times. But that wouldn't help if the tape job completes successfully and misses a file outright.

Dima P.
Product Manager
Posts: 10138
Liked: 817 times
Joined: Feb 04, 2013 2:07 pm
Full Name: Dmitry Popov
Location: Prague
Contact:

Re: v9 GFS job - No more continuous?

Post by Dima P. » Jan 18, 2016 5:35 pm

Thanks for bringing this up! I had a conversation with QA team and we’ve decided that endpoint job’s behavior in this case is not correct. Tape job should not be affected while endpoint job should fail: since Endpoint has 23-hour job retry cycle and will continue the backup once tape job succeeds.

I’d say GFS tape job is ‘almost continuous’: if it detects the incomplete restore point on disk or running source job it should restart itself every hour and check the source files. It should continue the same 'restart' procedure until the next scheduled start of the job (that implies non-endpoint jobs).

ekisner
Expert
Posts: 194
Liked: 33 times
Joined: Jul 26, 2012 8:04 pm
Full Name: Erik Kisner
Contact:

Re: v9 GFS job - No more continuous?

Post by ekisner » Jan 18, 2016 5:43 pm 1 person likes this post

Glad to hear the endpoint/tape functionality will be fixed! I agree that the tape job should be more important than the endpoint job. In an ideal world, it would simply tell the tape job to put the endpoint backup to the end of the list, and try it again before it finishes the job.

As for the "almost continuous" GFS, I've seen nothing of the sort. My active full jobs ran on Friday, the GFS job didn't start until Saturday. I manually stopped the GFS job Saturday night because it was using the wrong tapes (different forum thread on that subject) and it sat doing nothing. I did not disable the GFS tape job, I merely stopped it.

With the previous jobs, if it was set to continuous (which I find faster to write than "as new backup files appear", hopefully this doesn't cause misunderstandings) it would try, and try, and try, and try. On days where there was a persistent issue I'd generally get 40-60 or so emails from the B&R server over the course of the night. Unless it was stopped, then it went straight to disabled. If a full active job ran, as soon as it was finished, the tape job would start putting the VIB/VBK to tape.

Dima P.
Product Manager
Posts: 10138
Liked: 817 times
Joined: Feb 04, 2013 2:07 pm
Full Name: Dmitry Popov
Location: Prague
Contact:

Re: v9 GFS job - No more continuous?

Post by Dima P. » Jan 18, 2016 6:36 pm

When you stop the tape GFS job it should sit and wait for the next scheduled date (so it stop restarting itself). Let’s find out the root cause of the wrong tape being used in the weekly media set and then look across the logs once again to check it acts correctly when you stop the job. There is no need in separate case – just add time stamp when the tape job was stopped to the case. Thanks

nikpolini
Enthusiast
Posts: 25
Liked: 2 times
Joined: May 28, 2015 3:23 am
Contact:

[MERGED] GFS Tape job Schedule Question.

Post by nikpolini » Jan 21, 2016 5:24 am

I have just updated to V9, and have created a Tape job targeted at a GFS Media pool, I have set these to run on Tuesdays. However, I cannot set the exact start time. I want it at 8am, after all the night's backups are completed.(Reduce load on Primary Backup storage, as it is doing forever forwards) The job is scheduled to run at 12:00 AM.

Can I change this in powershell?

BTW, Loving the new interface/features so far.

Shestakov
Veeam Software
Posts: 6742
Liked: 681 times
Joined: May 21, 2014 11:03 am
Full Name: Nikita Shestakov
Location: Prague
Contact:

[MERGED] Re: GFS Tape job Schedule Question.

Post by Shestakov » Jan 21, 2016 9:55 am

Hello,

By design, the tape GFS job is running the whole day waiting for new restore points from the source.
Once it receives the new restore point, it copies the newest backup(creates a full) to tape. When restore points from all sources are copied to tape, the job is finished.
The behavior can not be changed in powershell.

Thanks!

ekisner
Expert
Posts: 194
Liked: 33 times
Joined: Jul 26, 2012 8:04 pm
Full Name: Erik Kisner
Contact:

Re: v9 GFS Job Schedule Questions

Post by ekisner » Jan 21, 2016 3:48 pm

I will point out that I have not observed this behaviour. But then I've also not had a successful job run yet, so I only know that after it is manually stopped it gives up. I assume it will start looking again as described after the job is started and runs successfully?

For reference, the existing ticket which deals with our job needing to be stopped is 01667096, presently with R&D for hotfix development.

Shestakov
Veeam Software
Posts: 6742
Liked: 681 times
Joined: May 21, 2014 11:03 am
Full Name: Nikita Shestakov
Location: Prague
Contact:

Re: v9 GFS Job Schedule Questions

Post by Shestakov » Jan 21, 2016 5:21 pm

If job is stopped manually, it will not retry. If you want to finish writing you can manually start the GFS job and choose the same media set(weekly/monthly/quarterly/yearly), VBR will write to the tape only those backups which are not copied yet.
Hope my explanation is clear.
Thanks!

nikpolini
Enthusiast
Posts: 25
Liked: 2 times
Joined: May 28, 2015 3:23 am
Contact:

Re: v9 GFS Job Schedule Questions

Post by nikpolini » Jan 22, 2016 5:49 am

What happens if the tape job starts on Tuesday, writes a full of a VM to tape, and is busy with others, and a new incremental comes on for an already completed machine, dose it do it again?
I am backing up 50TB each week to tape via Dell TL4000 and it takes 2-3days, during which time there are new RP added to those VMs...

veremin
Product Manager
Posts: 16524
Liked: 1374 times
Joined: Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin
Contact:

Re: v9 GFS Job Schedule Questions

Post by veremin » Jan 22, 2016 9:57 am

Nope, it should not do it again. Thanks.

Steen
Enthusiast
Posts: 48
Liked: 3 times
Joined: Mar 13, 2015 1:20 pm
Full Name: Steen
Contact:

[MERGED] GFS tape and montly time

Post by Steen » Jan 27, 2016 8:49 am

Hi.
I dont understand why i cant choose start time for monthly.
It always starts at 00:00:00?
/Steen
Regards Steen

Shestakov
Veeam Software
Posts: 6742
Liked: 681 times
Joined: May 21, 2014 11:03 am
Full Name: Nikita Shestakov
Location: Prague
Contact:

Re: v9 GFS Job Schedule Questions

Post by Shestakov » Jan 27, 2016 8:53 am

Hi Steen,
You are correct, but design the tape-GFS job is running the whole day waiting new restore points and once it has them it performs a write.

westCPS
Service Provider
Posts: 14
Liked: never
Joined: Aug 28, 2013 10:44 am
Full Name: Werner Steinegger
Contact:

[MERGED] Feature Request: Tape GFS Job Start-Time-Setting

Post by westCPS » Feb 11, 2016 4:13 pm

Hello!

We just update to v9.0.0.902 and found the new GFS Tape Pool Setting :D After we recreate our Backup Copy 2 Tape Job we saw that we only can Change the day but not the running start-time. Is there any info about when we can set up this, 'cause midnight isn't such a good time for that :wink:

Best regards Werner

Shestakov
Veeam Software
Posts: 6742
Liked: 681 times
Joined: May 21, 2014 11:03 am
Full Name: Nikita Shestakov
Location: Prague
Contact:

Re: v9 GFS Job Schedule Questions

Post by Shestakov » Feb 11, 2016 4:40 pm

Hello Werner,
We do receive the similar requests from time to time and thinking to implement backup window in one of the next releases.
Thanks!

PenguinSSH
Enthusiast
Posts: 28
Liked: 2 times
Joined: Feb 26, 2015 7:19 pm
Full Name: PenguinSSH
Contact:

Re: v9 GFS Job Schedule Questions

Post by PenguinSSH » Feb 12, 2016 5:22 pm

Can we get a retry option when the job fails. Seriously, if a job fails in the middle of the night because it was stopped by backup jobs, we would need it to retry automatically. Otherwise, we're loosing 10-12 hours of tape transfer.

Dima P.
Product Manager
Posts: 10138
Liked: 817 times
Joined: Feb 04, 2013 2:07 pm
Full Name: Dmitry Popov
Location: Prague
Contact:

Re: v9 GFS Job Schedule Questions

Post by Dima P. » Feb 13, 2016 11:21 pm

Hi PenguinSSH,

It should be retried automatically if any source job is locked/failed (or running): it constantly checking the source jobs and if the required job is completed – start placing the files to tape. Moreover, it can keep retiring till the next scheduled GFS date (for the source disk jobs that overlap the tape GFS schedule).

PenguinSSH
Enthusiast
Posts: 28
Liked: 2 times
Joined: Feb 26, 2015 7:19 pm
Full Name: PenguinSSH
Contact:

Re: v9 GFS Job Schedule Questions

Post by PenguinSSH » Feb 15, 2016 2:05 pm

Hi Dima,

This has not been our experience so far with the GFS Tape jobs, if it fails because it is interrupted by a source backup job, it stays failed until we start it manually again.

Dima P.
Product Manager
Posts: 10138
Liked: 817 times
Joined: Feb 04, 2013 2:07 pm
Full Name: Dmitry Popov
Location: Prague
Contact:

Re: v9 GFS Job Schedule Questions

Post by Dima P. » Feb 16, 2016 6:27 pm

Thanks for the heads up. We are going to check this behavior in our lab – stay tuned.

Dima P.
Product Manager
Posts: 10138
Liked: 817 times
Joined: Feb 04, 2013 2:07 pm
Full Name: Dmitry Popov
Location: Prague
Contact:

Re: v9 GFS Job Schedule Questions

Post by Dima P. » Feb 21, 2016 4:51 pm

PenguinSSH,

You were right. If the GFS tape job gets interrupted by source jobs (for example, because of the merging process) – GFS job is going to be cancel. Thanks for bringing this up – we will discuss with the team how this behavior can be corrected.

PenguinSSH
Enthusiast
Posts: 28
Liked: 2 times
Joined: Feb 26, 2015 7:19 pm
Full Name: PenguinSSH
Contact:

Re: v9 GFS Job Schedule Questions

Post by PenguinSSH » Feb 21, 2016 10:49 pm

Hi Dima

It seems to me that there are a couple of processes within Veeam that could benefit from this. If a process such as tape backup is running and depends on the backup files, the VBKs should not be altered until the dependency is completed. So a kind of queue in this process with priority should be helpful.

In other words, merging process and backup file check would have less priority over something like backup, tape backup or backup copy. Once those are finished, the lower priority process could be triggered.

In my opinion, with the different processes such as merging, backup guard, replication from backup files will make it harder for processes depending on VBKs to complete properly when those are lengthy or depend on the original VBKs.

Thanks for the reply and keeping us in the loop.
PenguinSSH

Shestakov
Veeam Software
Posts: 6742
Liked: 681 times
Joined: May 21, 2014 11:03 am
Full Name: Nikita Shestakov
Location: Prague
Contact:

Re: v9 GFS Job Schedule Questions

Post by Shestakov » Feb 22, 2016 10:33 am

PenguinSSH, thanks for the feedback.
By default backup and replication jobs have higher priority than backup copy and backup to tape since first two take a snapshot and actually preserve VMs state, while second two "just" copy existing backups on disk or tape.
You may switch the source backup job to basic forward incremental with no merges to avoid the situation like you face now.

F.Henne
Lurker
Posts: 1
Liked: 1 time
Joined: Jan 14, 2015 4:03 pm
Full Name: Fredrik Henne
Contact:

Re: v9 GFS Job Schedule Questions

Post by F.Henne » Mar 07, 2016 8:25 am 1 person likes this post

The case of interrupted tape jobs due to source backup jobs running merge is a problen for me too. The way I've solved it so far is to use the Job Scripts option and run powershell scripts for increasing/decreasing the source jobs retention policy by a few days. This puts the merge process on hold for a set number of days and at least makes it possible to have the tape job running over several days.

The scripts I use right now are simple dirty fixes for the problem in that they are are only increasing and decreasing by a fixed integer. A cleaner and better solution would be to read the retention policy that exists as of today in your jobs and then store those in clixml (Export-Clixml) for use in the decrease script. I'll leave the scripts below if anyone is interested.

A proper solution from Veeams side would be nice though for tape jobs and forever incremental. If possible it would be nice if the source backup jobs took into consideration any tape jobs running against the repository and skip or postpone any merge processes for later. Perhaps not the best solution for everyone but those us who's got space enough to hold 2-3 extra days of incrementals every week/month shouldn't have any problems with it. Seeing as it is the merge process that makes the jobs fail.

increaseRetainCycle.ps1

Code: Select all

Add-PSSnapin veeampssnapin

$jobs = Get-VBRJob | ? {$_.IsBackupJob -eq $True}

$jobs | Foreach-Object {
$options = $_.GetOptions()
$options.BackupStorageOptions.RetainCycles += 4
$_.SetOptions($options)
}
decreaseRetainCycle.ps1

Code: Select all

Add-PSSnapin veeampssnapin

$jobs = Get-VBRJob | ? {$_.IsBackupJob -eq $True}

$jobs | Foreach-Object {
$options = $_.GetOptions()
if($options.BackupStorageOptions.RetainCycles -gt 20) {
$options.BackupStorageOptions.RetainCycles -= 4
$_.SetOptions($options)
}
}

Dima P.
Product Manager
Posts: 10138
Liked: 817 times
Joined: Feb 04, 2013 2:07 pm
Full Name: Dmitry Popov
Location: Prague
Contact:

Re: v9 GFS Job Schedule Questions

Post by Dima P. » Mar 07, 2016 10:53 pm

Fredrik,

Thank you very much for sharing. We definitely looking into the solution to let the tape jobs wait for the source disk job if it was interrupted by whatever reason.

pkelly_sts
Expert
Posts: 568
Liked: 62 times
Joined: Jun 13, 2013 10:08 am
Full Name: Paul Kelly
Contact:

Re: v9 GFS Job Schedule Questions

Post by pkelly_sts » Mar 09, 2016 4:48 pm

I wonder if there's any scope for creating some kind of "Priority" flag, where we can choose (within reason) which jobs should have higher priority? Could also help with superceding GFS jobs if required (not sure if that side of things is working ok these days until I get onto v9)

Dima P.
Product Manager
Posts: 10138
Liked: 817 times
Joined: Feb 04, 2013 2:07 pm
Full Name: Dmitry Popov
Location: Prague
Contact:

Re: v9 GFS Job Schedule Questions

Post by Dima P. » Mar 15, 2016 4:59 pm

Hi Paul,

Setting the source job priority is not currently possible but will be added in the upcoming releases.

pkelly_sts
Expert
Posts: 568
Liked: 62 times
Joined: Jun 13, 2013 10:08 am
Full Name: Paul Kelly
Contact:

Re: v9 GFS Job Schedule Questions

Post by pkelly_sts » Mar 16, 2016 9:18 am

Good to hear :)

westCPS
Service Provider
Posts: 14
Liked: never
Joined: Aug 28, 2013 10:44 am
Full Name: Werner Steinegger
Contact:

Re: v9 GFS Job Schedule Questions

Post by westCPS » Mar 21, 2016 8:04 am

Thanks for all the informations.

Best regards
Werner

lowlander
Service Provider
Posts: 306
Liked: 21 times
Joined: Dec 28, 2014 11:48 am
Contact:

[MERGED] GFS tape job scheduling

Post by lowlander » Mar 30, 2016 5:34 am

Hi,

is it possible to schedule a GFS tape job @ a specific time. Now, the job always starts @ 12:00AM, while we want to fire it up @ 14:00PM.

Thanks !

Post Reply

Who is online

Users browsing this forum: No registered users and 1 guest