[Feature Request] Retry tape job

[EmpTee]

Hi,

Can you please include the option to retry a failed task in a tape backup job, in the same way you can retry a failed VM backup (via right-click retry)?

For example I have a GFS job that runs 15 backup jobs to tape. If one of those has an error, there is no easy way to just re-run that single task to the same backup set. I have to either re-run the full job which is a ~25TB backup, or create a new job for the task(s) that failed and run it again. Creating a new job puts it in a different backup set with different expiration settings which is not optimal.

Thanks

[veremin]

May a pre-job command set for a GFS backup to tape job would do the trick for you? The script will analyze whether some source jobs have "failed" as the last state, retry those that do, and proceed to the tape job execution afterwards. Thanks.

[EmpTee]

May a pre-job command set for a GFS backup to tape job would do the trick for you? The script will analyze whether some source jobs have "failed" as the last state, retry those that do, and proceed to the tape job execution afterwards. Thanks.

It might - can you provide an example/link to more information on this?


Thanks

[veremin]

1. Find jobs that have failed during previous cycle. Use job name property to search only for the required jobs:

Code: Select all

$JobName = "Name of your first job","Name of your second job"
$FailedJobs = Get-VBRJob -name $Jobname | where {$_.getlastresult() -eq "Failed"}

2. Retry failed jobs:

Code: Select all

Start-VBRJob -Job $FailedJobs -RetryBackup

I don't have access to the console at the moment, so you'd better check and modify the provided example, if something doesn't work immediately.

For more information regarding our PS snap-in check corresponding section of our Help Center.

Thanks.

[EmpTee]

I don't think this will resolve the issue. Not sure I explained the situation very well. If I have a backup to disk job that contains 5 VM's and one of the 5 VMs fails to backup I can right click that job and select retry. The job will re-run and only that single VM will be backed up. I would like the same functionality for a tape job.

In my situation the source jobs to disk have completed with no issues, so none have a status of failed from the last cycle.

The GFS Tape job I have made is constructed by selecting "Add... Backup Jobs" in the "Backup Files" section of the Tape job creation wizard. I have made a selection of 10 backup jobs (not repositories) to write to tape.

What I need to retry is when one of those 10 backup tasks contained within the single GFS Tape job fails to write to tape.

As an example, I recently had one task out of 10 fail to write to tape with the following error:
--------------
Synthetic full
BackupSynthesizedStorageToTape failed Tape fatal error. Data error (cyclic redundancy check). Tape write error Exception from server: Shared memory connection was closed. Tape fatal error.
---------------
After that failure the other 9 items in the single GFS job completed with no issues, so I had a set of tapes that contained only 9 of the 10 items I needed in the set.

Currently to retry that failed item in the GFS job I need to either create a new backup to tape job with just that single source backup job in the selection list, or re-run the entire GFS tape job which contains 10 source jobs, and is approx. 25TB. 25TB takes a long time to backup.

Creating a new job means the tape used is in a separate media set with a different expiry from the rest of the tapes.

A Retry would resolve this as when the item is successful it would be placed placed in the same media set with the same expiry.

[veremin]

Your explanation has clarified the confusion indeed, since originally I thought that you're talking about failed backup jobs being skipped by tape GFS one from processing.

Getting back to your request, we already have built-in retry mechanism inside GFS backup to tape job. In your case the job should have been stopped and then retried in the nearest "round" hour. Say, it stopped at 18:43 with one source job failed to archive, it would be retried at 19:00, since not all source jobs have been processed successfully. The only case in which the retry logic should not get triggered is a GFS job stopping within 23:00 - 24:00 period.

If the described logic doesn't take place in your situation, kindly, reach our support team directly.

Thanks.

[rreed]

+1 for any kind of tape job (simple, etc.). If something blows up part way through, a simple right-click, retry what you just did the last time around like regular backup jobs would be great please.

[rreed]

Ugh, case in point, we had a bad tape in our library. I don't know why but until I got down to the data center to remove it this morning every time a tape job would run it would pick this tape - usually just failing and giving up right then and there, or on the occasion I managed to load a Free tape into one of the drives then kick off the job - until that tape filled and it needed another one, and picked that bad tape again. And failed, giving up right there after all that time and another wasted tape.

In addition to Retry (what you just did), "If the tape you tried to use is trash and there are available tapes and/or space on existing ones, try using another tape instead of just giving up!!!" Please.

[Dima P.]

Hi,

we had a bad tape in our library

Was it somehow mechanically damaged? When the tape reaches its life cycle it should be automatically moved to expired on the software level (what VBR does).

[rreed]

No, it's throwing different block size errors, won't let me format it (something error 82 something), etc. Encountering a tape w/ different block size than what the drive is currently set might be another conversation. Despite it being a brand new tape. Anyways, regardless of the tape issue, sure would be great if it would say "Darn, can't write to this tape, I'll try another one and continue on."

[Dima P.]

Count it as noted

[alanbolte]

If you are running v9 and are unable to erase a tape because of its block size, please request a hotfix from support.

[rreed]

Thanks Alan, #01821607.

[efd121]

+1 for me also. I just had a tape job that was backing up over 5M files and it was at 60% after 45hrs and it just died with a very generic "Job has failed unexpectedly" Seems my only option to start over? It would be great to just retry instead of losing 45hrs of time.
Dave

[Dima P.]

David,

Thanks for the input. By the way, can you elaborate how you tape library is connected to the source repository?

[efd121]

I have a Spectra Logic T380 library that has a single drive partition connected via fiber channel (8Gb) to a physical W2K16 server. The 5M files being backed up are on a virtual machine

[Dima P.]

The 5M files being backed up are on a virtual machine

Have you tried new v9.5? We have improved engine for file to tape jobs with a huge number of source files which should come handy in case like yours.

[efd121]

Yes I have tried 9.5 that is the version I was running when this job failed
Dave

[Chris.B]

I came across this thread as we had a couple of backups fail on our yearly GFS tape job over the weekend, being able to retry failed tape jobs would be really useful.

My solution was to clone the failed GFS tape job and then edit it to remove the VMs which had backed up successfully so only the VMs which had failed were left. I then ran the cloned job by right-clicking and selecting Active Full > Yearly. This ensured that only the failed VMs were processed and the backups ended up on the correct media set.

Chris

[sasilik]

+1 for a little bit different "retry tape job". Seems that if there is some problems reading source files then tape job fails. Could the tape jobs have similar option as VM jobs - retry N times and wait N minutes before retrying. Maybe there is some temporary problem and maybe there is only problem with reading one file and job could go on and report failing files. For now Veeam fails entire tape job when these kind of errors occur.

[bct44]

Hello,

From my POV and current production problematic, it lacks a retry button on backup/files to tape jobs as it possible on backup jobs or backup copy jobs. Even V12 is more stable for backup to tape jobs on large scale infra, we're lacking options like backup jobs or backup copy jobs to enable automatic retry for errors specially on mission critical jobs.

Could you consider my request?

Thanks!

[HannesK]

Hello,
yes, I also got that request recently at an event and we count your request +1

Best regards,
Hannes

[JGM2023]

Hello,

Since the Improvements In 12 & 12.1 with Increasing the objects In Backup Jobs, It has now caused a knock on effect with Tape Jobs, For example If we put over 100 machines in a Backup Job, This could be 30TB+ that we need to write to Tape. We run full backups on the 1st of the Month At times this does not complete before the backup has to start again.

Is there a reason that the function to retry machines Is not Implemented In Tape Jobs? If this was the case and It did not have to start a whole full again for the whole Job, then this wouldn't be so much of an issue.

[Mildur]

Hi JGM

I moved your request to a similar topic. There is no retry button today.
We are aware about the request to have a retry option for tapes to only process "not yet processed data".
Your vote is counted as +1.

Best,
Fabian

[johannesk]

@Mildur,

+1 vote from me to have a retry option on tape jobs.

[Samba222]

+1

[Felix4002]

+ 1 Vote from me for the Retry Button

[RaHa]

+ 1 vote

[KoflerTx]

+1

[reni3008]

+1