GFS Job Retries

[jshiflet]

So I've spoken support about this, but wanted to post here to see if anyone else happens to have this same issue:

We have our backup-to-disk jobs to kick off an incremental on Fridays, but then to immediately create a synthetic full from it (so we are generating "fulls" every week). Because of the sheer size of some of our data, it can take from Friday night until Monday morning for the synthetic full to complete, and we perform Active fulls the last Friday of each month, which suffer the same problem of just taking 48+ hours to complete because of the amount of data. (I'm hoping to move to ReFS repositories in the future, but it's not doable right now because of space limitations where I don't have the space to move repositories around in order to reformat the volumes yet).

So with that said, what we're running into is that we have our GFS Tape jobs configured to run on Sunday, which means they kick off at 12:00am Sunday morning. Well, because a number of our synthetic fulls/active fulls don't complete by 12:00am Sunday, the GFS jobs just sit there for 24 hours until it "retries" at 12:00am on Monday. And even then some jobs may still not be completed, but the GFS job will finally pick up that a full VBK is in the repository and ready to be archived to tape. It seems that after the initial "kick-off" of the job at 12:00am on Sunday, it won't re-check or retry until 24 hours later. The strange behavior that gets me is that when it retries at 12:00am on Monday, if the job it's done, it still sits there, but if the source job completes at say 4:30am on Monday, the GFS job will then pick up that the VBK is ready to be archived within 15 minutes of the source completing.

What I was told by support was that GFS jobs won't retry for 24 hours, by design. That seems a little counter-intuitive. On the same note, before we updated to 9.5 Update 3, the GFS jobs behaved like I would expect: if the source job completed at like 4am on Sunday, then at about 4:15am on Sunday, they'd start getting archived to tape. There seems to be an unnecessary 24-hour waiting period on GFS jobs that require retries, but only for the first 24-hours that has suddenly appeared in Update 3 and is now causing issues with our objectives for getting tapes off-site.

[lyapkost]

Hi Jason. The reason of this behavior is here:

If no backup appears in 24 hours, the GFS job looks for the most recent restore point available. If it is a full backup, the GFS job copies it.

So your Sunday GFS job waits for Sunday's backup and your running backup which started on Friday counts as a Friday's one because of the creation (not modification) date. Only when no backup appears in 24 hours is starts to search back for the previous backups.

To acheive your goal and not wait for the whole Sunday I suggest to set a special registry key which changes backup wait timeout for GFS jobs. You may get the key from your support engineer.

[jshiflet]

Okay, this makes more sense now, but here's a question: if the source backup job is still computing a synthetic full, after the GFS timeout passes, does the GFS job detect that the synthetic full is still being computed by the source job and waits for that to complete, or is it going to create it's own synthetic full (.vrb file, I think) separate from the source job's new VBK?

I think I've seen it waiting for the source synthetic full VBK to complete, but I just want to double-check.

(And because the forum is yelling at me for not posting the support case ID when I try to post this reply, it's Case # 02638807)

[jshiflet]

Okay, I re-read that documentation section and I see that it is supposed to do what I expect after the initial 24-hour waiting period.

Thanks for helping clarify how the GFS jobs operate Konstantin!

[lyapkost]

After waiting for 24 hours a tape job takes the most recent backup from one of the previous days (Friday in your case) and copies it to tape if it is a full (vbk, your case) or creates a virtual full if it is an increment (vib). vrb is a reverse incremental backup file and is not processed by tape jobs at all.

You're welcome Jason.