[Feature Request] Retry tape job

Everything about backing up to tape

[Feature Request] Retry tape job

Veeam Logoby EmpTee » Mon May 02, 2016 1:05 am

Hi,

Can you please include the option to retry a failed task in a tape backup job, in the same way you can retry a failed VM backup (via right-click retry)?

For example I have a GFS job that runs 15 backup jobs to tape. If one of those has an error, there is no easy way to just re-run that single task to the same backup set. I have to either re-run the full job which is a ~25TB backup, or create a new job for the task(s) that failed and run it again. Creating a new job puts it in a different backup set with different expiration settings which is not optimal.

Thanks
EmpTee
Novice
 
Posts: 7
Liked: never
Joined: Thu Feb 04, 2016 2:14 am

Re: [Feature Request] Retry tape job

Veeam Logoby v.Eremin » Mon May 02, 2016 11:41 am

May a pre-job command set for a GFS backup to tape job would do the trick for you? The script will analyze whether some source jobs have "failed" as the last state, retry those that do, and proceed to the tape job execution afterwards. Thanks.
v.Eremin
Veeam Software
 
Posts: 13558
Liked: 1006 times
Joined: Fri Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin

Re: [Feature Request] Retry tape job

Veeam Logoby EmpTee » Mon May 02, 2016 11:00 pm

v.Eremin wrote:May a pre-job command set for a GFS backup to tape job would do the trick for you? The script will analyze whether some source jobs have "failed" as the last state, retry those that do, and proceed to the tape job execution afterwards. Thanks.


It might - can you provide an example/link to more information on this?


Thanks
EmpTee
Novice
 
Posts: 7
Liked: never
Joined: Thu Feb 04, 2016 2:14 am

Re: [Feature Request] Retry tape job

Veeam Logoby v.Eremin » Tue May 03, 2016 2:40 pm

1. Find jobs that have failed during previous cycle. Use job name property to search only for the required jobs:

Code: Select all
$JobName = "Name of your first job","Name of your second job"
$FailedJobs = Get-VBRJob -name $Jobname | where {$_.getlastresult() -eq "Failed"}


2. Retry failed jobs:

Code: Select all
Start-VBRJob -Job $FailedJobs -RetryBackup


I don't have access to the console at the moment, so you'd better check and modify the provided example, if something doesn't work immediately.

For more information regarding our PS snap-in check corresponding section of our Help Center.

Thanks.
v.Eremin
Veeam Software
 
Posts: 13558
Liked: 1006 times
Joined: Fri Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin

Re: [Feature Request] Retry tape job

Veeam Logoby EmpTee » Wed May 04, 2016 4:43 am

I don't think this will resolve the issue. Not sure I explained the situation very well. If I have a backup to disk job that contains 5 VM's and one of the 5 VMs fails to backup I can right click that job and select retry. The job will re-run and only that single VM will be backed up. I would like the same functionality for a tape job.

In my situation the source jobs to disk have completed with no issues, so none have a status of failed from the last cycle.

The GFS Tape job I have made is constructed by selecting "Add... Backup Jobs" in the "Backup Files" section of the Tape job creation wizard. I have made a selection of 10 backup jobs (not repositories) to write to tape.

What I need to retry is when one of those 10 backup tasks contained within the single GFS Tape job fails to write to tape.

As an example, I recently had one task out of 10 fail to write to tape with the following error:
--------------
Synthetic full
BackupSynthesizedStorageToTape failed Tape fatal error. Data error (cyclic redundancy check). Tape write error Exception from server: Shared memory connection was closed. Tape fatal error.
---------------
After that failure the other 9 items in the single GFS job completed with no issues, so I had a set of tapes that contained only 9 of the 10 items I needed in the set.

Currently to retry that failed item in the GFS job I need to either create a new backup to tape job with just that single source backup job in the selection list, or re-run the entire GFS tape job which contains 10 source jobs, and is approx. 25TB. 25TB takes a long time to backup.

Creating a new job means the tape used is in a separate media set with a different expiry from the rest of the tapes.

A Retry would resolve this as when the item is successful it would be placed placed in the same media set with the same expiry.
EmpTee
Novice
 
Posts: 7
Liked: never
Joined: Thu Feb 04, 2016 2:14 am

Re: [Feature Request] Retry tape job

Veeam Logoby v.Eremin » Wed May 04, 2016 2:04 pm

Your explanation has clarified the confusion indeed, since originally I thought that you're talking about failed backup jobs being skipped by tape GFS one from processing.

Getting back to your request, we already have built-in retry mechanism inside GFS backup to tape job. In your case the job should have been stopped and then retried in the nearest "round" hour. Say, it stopped at 18:43 with one source job failed to archive, it would be retried at 19:00, since not all source jobs have been processed successfully. The only case in which the retry logic should not get triggered is a GFS job stopping within 23:00 - 24:00 period.

If the described logic doesn't take place in your situation, kindly, reach our support team directly.

Thanks.
v.Eremin
Veeam Software
 
Posts: 13558
Liked: 1006 times
Joined: Fri Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin

Re: [Feature Request] Retry tape job

Veeam Logoby rreed » Thu May 05, 2016 6:55 pm

+1 for any kind of tape job (simple, etc.). If something blows up part way through, a simple right-click, retry what you just did the last time around like regular backup jobs would be great please.
VMware 6
Veeam B&R v9
Dell DR4100's
EMC DD2200's
EMC DD620's
Dell TL2000 via PE430 (SAS)
rreed
Expert
 
Posts: 354
Liked: 72 times
Joined: Tue Jun 30, 2015 6:06 pm

Re: [Feature Request] Retry tape job

Veeam Logoby rreed » Mon Jun 06, 2016 8:50 pm

Ugh, case in point, we had a bad tape in our library. I don't know why but until I got down to the data center to remove it this morning every time a tape job would run it would pick this tape - usually just failing and giving up right then and there, or on the occasion I managed to load a Free tape into one of the drives then kick off the job - until that tape filled and it needed another one, and picked that bad tape again. And failed, giving up right there after all that time and another wasted tape.

In addition to Retry (what you just did), "If the tape you tried to use is trash and there are available tapes and/or space on existing ones, try using another tape instead of just giving up!!!" Please.
VMware 6
Veeam B&R v9
Dell DR4100's
EMC DD2200's
EMC DD620's
Dell TL2000 via PE430 (SAS)
rreed
Expert
 
Posts: 354
Liked: 72 times
Joined: Tue Jun 30, 2015 6:06 pm

Re: [Feature Request] Retry tape job

Veeam Logoby Dima P. » Mon Jun 06, 2016 11:46 pm

Hi,

we had a bad tape in our library

Was it somehow mechanically damaged? When the tape reaches its life cycle it should be automatically moved to expired on the software level (what VBR does).
Dima P.
Veeam Software
 
Posts: 6517
Liked: 454 times
Joined: Mon Feb 04, 2013 2:07 pm
Location: SPb
Full Name: Dmitry Popov

Re: [Feature Request] Retry tape job

Veeam Logoby rreed » Tue Jun 07, 2016 2:08 pm 1 person likes this post

No, it's throwing different block size errors, won't let me format it (something error 82 something), etc. Encountering a tape w/ different block size than what the drive is currently set might be another conversation. Despite it being a brand new tape. Anyways, regardless of the tape issue, sure would be great if it would say "Darn, can't write to this tape, I'll try another one and continue on." :wink:
VMware 6
Veeam B&R v9
Dell DR4100's
EMC DD2200's
EMC DD620's
Dell TL2000 via PE430 (SAS)
rreed
Expert
 
Posts: 354
Liked: 72 times
Joined: Tue Jun 30, 2015 6:06 pm

Re: [Feature Request] Retry tape job

Veeam Logoby Dima P. » Tue Jun 07, 2016 11:58 pm

Count it as noted :wink:
Dima P.
Veeam Software
 
Posts: 6517
Liked: 454 times
Joined: Mon Feb 04, 2013 2:07 pm
Location: SPb
Full Name: Dmitry Popov

Re: [Feature Request] Retry tape job

Veeam Logoby alanbolte » Wed Jun 08, 2016 12:39 am

If you are running v9 and are unable to erase a tape because of its block size, please request a hotfix from support.
alanbolte
Expert
 
Posts: 635
Liked: 172 times
Joined: Mon Jun 18, 2012 8:58 pm
Full Name: Alan Bolte

Re: [Feature Request] Retry tape job

Veeam Logoby rreed » Wed Jun 08, 2016 2:16 pm

Thanks Alan, #01821607.
VMware 6
Veeam B&R v9
Dell DR4100's
EMC DD2200's
EMC DD620's
Dell TL2000 via PE430 (SAS)
rreed
Expert
 
Posts: 354
Liked: 72 times
Joined: Tue Jun 30, 2015 6:06 pm

Re: [Feature Request] Retry tape job

Veeam Logoby efd121 » Fri Dec 09, 2016 2:47 am

+1 for me also. I just had a tape job that was backing up over 5M files and it was at 60% after 45hrs and it just died with a very generic "Job has failed unexpectedly" Seems my only option to start over? It would be great to just retry instead of losing 45hrs of time.
Dave
efd121
Enthusiast
 
Posts: 25
Liked: never
Joined: Fri Aug 07, 2015 8:45 pm
Full Name: David Engler

Re: [Feature Request] Retry tape job

Veeam Logoby Dima P. » Fri Dec 09, 2016 10:52 am

David,

Thanks for the input. By the way, can you elaborate how you tape library is connected to the source repository?
Dima P.
Veeam Software
 
Posts: 6517
Liked: 454 times
Joined: Mon Feb 04, 2013 2:07 pm
Location: SPb
Full Name: Dmitry Popov

Next

Return to Tape



Who is online

Users browsing this forum: No registered users and 5 guests