Feature Request (Tape and Forever Incremental)

[newfirewallman]

When someone has large backups to tape that they would like to run weekly, but also get daily backups to disk jobs fail. It would be great for the incremental backup job to run like normal and merge data when the tape job is complete.

[Dima P.]

Hello Blake,

get daily backups to disk jobs fail. It would be great for the incremental backup job to run like normal and merge data when the tape job is complete

I am not sure I got this. If backup to disk job fails and there is no increment – there is nothing to backup on tape.

However, you might be interested that we have a virtual full backup to tape feature available since version 8. It allows creating the full backup on tape from the incremental restore point on disk according to the specified schedule (by schedule I mean choosing the exact day when you want to have a full backup on tape, created from an increment on disk).

[veremin]

When someone has large backups to tape that they would like to run weekly, but also get daily backups to disk jobs fail.

Can you elaborate on your request? You have two jobs (backup and backup to tape one), sometimes it happens that the backup to tape job takes longer than expected, and the subsequent run of the backup job fails due to files being locked, correct? And you want to have sort of timeout after which the backup job can proceed smoothly?

[newfirewallman]

I can elaborate. There are actually a couple different things going on when you have your source backups set as forever incremental. Lets say i keep my retention at 7 points on disk, I need to run backups nightly and get a "full" backup off to tape on fridays.
If my file server backup job is 10 TB my tape job takes 1-2 days to get a backup off to tape because of the size it can't finish before the next normal backup to disk occurs, which causes the tape job to fail. The other issue (Feature) is how to synthesize the full to tape WITHOUT all the incrementals. In my example i only need the 10TB file to disk on friday. Not the 10TB vbk and 6 vib's which would total 14TB. So my options to work around this issue have been:
1. Try and do incremental with synthetic fulls on Friday (transform so i actually have disk space). If the disk IO can keep up this works and i can get it off to tape before the next friday (not the most efficient solution because of IO).
or
2. Only process the fulls and not process incrementals. This gets the size to tape down to the appropriate size, but i'm always a week behind on tape and can't really control if the VBK is even on a friday as in forever incremental is just goes off the first time it was ran (or new active full) and number of retention points.

* Would be nice sometime to keep 14 retention points and only synthesize latest friday to tape, what i would love to do, except i would use 10-15 LTO6 tapes per week (and have redundant restore points from week to week)

Can you elaborate on your request? You have two jobs (backup and backup to tape one), sometimes it happens that the backup to tape job takes longer than expected, and the subsequent run of the backup job fails due to files being locked, correct? And you want to have sort of timeout after which the backup job can proceed smoothly?

Correct. Except i don't want a timeout, i just want the next incremental to run and wait until the tape job finishes before merging.

In my environment besides the job example about i have another 6 backup jobs ranging from 500GB (full) to 2.5TB(full) so you can see how it becomes a compounding concern.

Please keep me posted as this/these are some of the biggest frustations with an otherwise outstanding product.

[veremin]

Correct. Except i don't want a timeout, i just want the next incremental to run and wait until the tape job finishes before merging.

Speaking about workarounds, you might want to set a certain script as a pre-job command. The script will check whether the tape job is running and proceed to execution only if it is not. Thanks.

[newfirewallman]

This would be really nice if it could be done somehow:
2. Only process the fulls and not process incrementals. This gets the size to tape down to the appropriate size, but i'm always a week behind on tape and can't really control if the VBK is even on a friday as in forever incremental is just goes off the first time it was ran (or new active full) and number of retention points.

* Would be nice sometime to keep 14 retention points and only synthesize latest friday to tape, what i would love to do, except i would use 10-15 LTO6 tapes per week (and have redundant restore points from week to week)

[veremin]

Only process the fulls and not process incrementals.

Hmm, what if you untick "process incremental" option in the settings of a backup to tape job and make process only full backups?

[newfirewallman]

That will only grab the VBK which is a week old. It won't synthesize the full on the latest Friday. So if you had 14 retention points your offsite to tape data would be two weeks old.

[veremin]

Then, I'd suggest waiting for Update 2 and stick to synthesized virtual full backup (that currently requires "process incremental" option being enabled).

[newfirewallman]

Great! I can't WAIT!!!!

[veremin]

It should become generally available by the end of this month; so, stay tuned!

[nathanael.duke]

I recently got a feature request for fuzzy matching when creating VSBs, so that in situations where a backup day is skipped, a VSB is still created with the next point closest to the intended VSB day.

What do you think?

[veremin]

If on a given day specified for VSB creation the backup job doesn't create anything, a tape job should look backwards and create VSB out from the closest restore point; is that what you're asking for? Thanks.

[newfirewallman]

So last thing with the update coming out any day now, will it also continue to run the tape job if a new backup job us running?

[veremin]

Will a tape job keep creating a virtual synthesized full backup, if a source backup job runs at the same time? Is that what you're asking?

[newfirewallman]

Yes. My Tape job I run weekly will use 5 LTO6 tapes, and I would still like the daily backup to disk to run while the tape job is running.

[newfirewallman]

Yes. In my environment I need to do daily backups, but my tape job takes longer than 1 day to send off to tape (only do off to tape weekly)

[veremin]

When a source backup job takes place, it should not prevent a tape job from creating virtual synthesized full backup. Thanks.

[newfirewallman]

What happens when the tape job is running and a source job kicks off?

[veremin]

The tape job should be stopped as it has less priority in comparison with backup job (secondary vs primary destination). Thanks.

[newfirewallman]

That is the dilemma!
I want daily backups (source to run daily)
I also want a weekly or monthly tape job that will be a synthetic full.

My backup job to tape is 11.5TB-14TB (takes longer than 24 hours to complete.

Do you see the issue I have now?

[veremin]

Are we talking about synthesized full backup or about normal tape job cycle? As mentioned, a source backup job running should not prevent a synthesized full backup creation, only normal archival cycles would be stopped in this case.

[newfirewallman]

I'm not following your verbiage possible (what do you mean by normal tape job cycle, or source tape job?)

I'm trying to make it as clear (as mud) as I can.
Daily full backup to disk (my primary backup) weekly I would like to synthesize a full backup to tape on Friday. Currently if the tape job starts Saturday morning 9:00am (synthesized full on Friday) and my daily jobs start on Saturday night, it will break the currently running tape job. I would much rather it ignore the retention policy and merging, and just create an extra incremental until the tape job is complete (or at the next incremental that the tape job isn't running) and then merge and get back on schedule with the retention job.

[veremin]

I'm not following your verbiage possible (what do you mean by normal tape job cycle, or source tape job?)

As to source job, I've made a typo and written source tape job, instead of source backup job; apologize for confusion.

Normal tape job cycle means a cycle during which a backup to tape job is just copying files, and not creating virtual synthesized full backup directly on tapes. So, the next question during Saturday's cycle you're willing to create a virtual synthesized full backup on tape or just make backup to tape job copy files?

Currently if the tape job starts Saturday morning 9:00am (synthesized full on Friday) and my daily jobs start on Saturday night, it will break the currently running tape job.

Thanks.

[newfirewallman]

I'm not really following, but if you would like to discuss in person that would be excellent.

My tape job only runs once a week (starts on Sunday or Monday) and synthesizes a backup from the source backup to disk job as of Friday. So when the backup to Tape job is running it takes in excess of 25 hours and obviously my regularly scheduled daily backup will have to run before it finishes, hence it affecting the vbk/chain that the synthesized backup is using to tape.

Does that make sense or am i not understanding it properly?

[veremin]

According to the QA team, in Update 2 if a backup to tape job is creating the synthesized full backup, and at that moment a backup job (that is specified as the source for the backup to tape job) starts, it should not prevent the backup to tape job from creating virtual synthesized full backup.

Hope it helps.
Thanks.

[newfirewallman]

Sorry i was doing full without snythesized (which wasn't ideal because the VBK's could be a week behind what i really wanted) which when the incremental to disk job ran would change the vbk.
Installed Update 2 Will run a tape job tonight with synthesized as of saturday and see what happens when the tape job is running and the incremental to disk kicks off.

[veremin]

Glad to hear we've clarified out conclusion.

Virtual synthesized full backup is recommended for forward forever incremental chains, indeed. Not only should usage of it allow to have up-to-date .vbk on tapes, but also it should solve you the issue of backup job interrupting the tape one.

Thanks.

[newfirewallman]

So i have V8 Update 2 and running synthetic full as of saturday. Started the job, it created the synthetic backup and starting writing to Tape. Then the normal daily incremental to disk job started while the job to tape was running. And it failed the tape backup with the following error: "Synthetic Full stopped by job (name of source disk backup)"

[Dave338]

And the issue of copying to tape the synthesized backup and also the full vbk in every job has been corrected in anyway?

I can't have 2 full backups transfered to tape and so the forever incremental modifies the vbk every day (merging oldest incremental), there are two vbk files copied to tape, the oldest point in the chain, and the synthesized one (plus the incremental files)
I've using reverse incremental to avoid this problem and copy to tape only the full backup, that is the newest one, but It stresses my backup target too much=slow.

Regards.