Discussions related to exporting backups to tape and backing up directly to tape.
Post Reply
EmpTee
Novice
Posts: 9
Liked: never
Joined: Feb 04, 2016 2:14 am
Contact:

[Feature Request] Retry tape job

Post by EmpTee »

Hi,

Can you please include the option to retry a failed task in a tape backup job, in the same way you can retry a failed VM backup (via right-click retry)?

For example I have a GFS job that runs 15 backup jobs to tape. If one of those has an error, there is no easy way to just re-run that single task to the same backup set. I have to either re-run the full job which is a ~25TB backup, or create a new job for the task(s) that failed and run it again. Creating a new job puts it in a different backup set with different expiration settings which is not optimal.

Thanks
veremin
Product Manager
Posts: 20270
Liked: 2252 times
Joined: Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin
Contact:

Re: [Feature Request] Retry tape job

Post by veremin »

May a pre-job command set for a GFS backup to tape job would do the trick for you? The script will analyze whether some source jobs have "failed" as the last state, retry those that do, and proceed to the tape job execution afterwards. Thanks.
EmpTee
Novice
Posts: 9
Liked: never
Joined: Feb 04, 2016 2:14 am
Contact:

Re: [Feature Request] Retry tape job

Post by EmpTee »

v.Eremin wrote:May a pre-job command set for a GFS backup to tape job would do the trick for you? The script will analyze whether some source jobs have "failed" as the last state, retry those that do, and proceed to the tape job execution afterwards. Thanks.
It might - can you provide an example/link to more information on this?


Thanks
veremin
Product Manager
Posts: 20270
Liked: 2252 times
Joined: Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin
Contact:

Re: [Feature Request] Retry tape job

Post by veremin »

1. Find jobs that have failed during previous cycle. Use job name property to search only for the required jobs:

Code: Select all

$JobName = "Name of your first job","Name of your second job"
$FailedJobs = Get-VBRJob -name $Jobname | where {$_.getlastresult() -eq "Failed"}
2. Retry failed jobs:

Code: Select all

Start-VBRJob -Job $FailedJobs -RetryBackup
I don't have access to the console at the moment, so you'd better check and modify the provided example, if something doesn't work immediately.

For more information regarding our PS snap-in check corresponding section of our Help Center.

Thanks.
EmpTee
Novice
Posts: 9
Liked: never
Joined: Feb 04, 2016 2:14 am
Contact:

Re: [Feature Request] Retry tape job

Post by EmpTee »

I don't think this will resolve the issue. Not sure I explained the situation very well. If I have a backup to disk job that contains 5 VM's and one of the 5 VMs fails to backup I can right click that job and select retry. The job will re-run and only that single VM will be backed up. I would like the same functionality for a tape job.

In my situation the source jobs to disk have completed with no issues, so none have a status of failed from the last cycle.

The GFS Tape job I have made is constructed by selecting "Add... Backup Jobs" in the "Backup Files" section of the Tape job creation wizard. I have made a selection of 10 backup jobs (not repositories) to write to tape.

What I need to retry is when one of those 10 backup tasks contained within the single GFS Tape job fails to write to tape.

As an example, I recently had one task out of 10 fail to write to tape with the following error:
--------------
Synthetic full
BackupSynthesizedStorageToTape failed Tape fatal error. Data error (cyclic redundancy check). Tape write error Exception from server: Shared memory connection was closed. Tape fatal error.
---------------
After that failure the other 9 items in the single GFS job completed with no issues, so I had a set of tapes that contained only 9 of the 10 items I needed in the set.

Currently to retry that failed item in the GFS job I need to either create a new backup to tape job with just that single source backup job in the selection list, or re-run the entire GFS tape job which contains 10 source jobs, and is approx. 25TB. 25TB takes a long time to backup.

Creating a new job means the tape used is in a separate media set with a different expiry from the rest of the tapes.

A Retry would resolve this as when the item is successful it would be placed placed in the same media set with the same expiry.
veremin
Product Manager
Posts: 20270
Liked: 2252 times
Joined: Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin
Contact:

Re: [Feature Request] Retry tape job

Post by veremin »

Your explanation has clarified the confusion indeed, since originally I thought that you're talking about failed backup jobs being skipped by tape GFS one from processing.

Getting back to your request, we already have built-in retry mechanism inside GFS backup to tape job. In your case the job should have been stopped and then retried in the nearest "round" hour. Say, it stopped at 18:43 with one source job failed to archive, it would be retried at 19:00, since not all source jobs have been processed successfully. The only case in which the retry logic should not get triggered is a GFS job stopping within 23:00 - 24:00 period.

If the described logic doesn't take place in your situation, kindly, reach our support team directly.

Thanks.
rreed
Veteran
Posts: 354
Liked: 72 times
Joined: Jun 30, 2015 6:06 pm
Contact:

Re: [Feature Request] Retry tape job

Post by rreed »

+1 for any kind of tape job (simple, etc.). If something blows up part way through, a simple right-click, retry what you just did the last time around like regular backup jobs would be great please.
VMware 6
Veeam B&R v9
Dell DR4100's
EMC DD2200's
EMC DD620's
Dell TL2000 via PE430 (SAS)
rreed
Veteran
Posts: 354
Liked: 72 times
Joined: Jun 30, 2015 6:06 pm
Contact:

Re: [Feature Request] Retry tape job

Post by rreed »

Ugh, case in point, we had a bad tape in our library. I don't know why but until I got down to the data center to remove it this morning every time a tape job would run it would pick this tape - usually just failing and giving up right then and there, or on the occasion I managed to load a Free tape into one of the drives then kick off the job - until that tape filled and it needed another one, and picked that bad tape again. And failed, giving up right there after all that time and another wasted tape.

In addition to Retry (what you just did), "If the tape you tried to use is trash and there are available tapes and/or space on existing ones, try using another tape instead of just giving up!!!" Please.
VMware 6
Veeam B&R v9
Dell DR4100's
EMC DD2200's
EMC DD620's
Dell TL2000 via PE430 (SAS)
Dima P.
Product Manager
Posts: 14396
Liked: 1568 times
Joined: Feb 04, 2013 2:07 pm
Full Name: Dmitry Popov
Location: Prague
Contact:

Re: [Feature Request] Retry tape job

Post by Dima P. »

Hi,
we had a bad tape in our library
Was it somehow mechanically damaged? When the tape reaches its life cycle it should be automatically moved to expired on the software level (what VBR does).
rreed
Veteran
Posts: 354
Liked: 72 times
Joined: Jun 30, 2015 6:06 pm
Contact:

Re: [Feature Request] Retry tape job

Post by rreed » 1 person likes this post

No, it's throwing different block size errors, won't let me format it (something error 82 something), etc. Encountering a tape w/ different block size than what the drive is currently set might be another conversation. Despite it being a brand new tape. Anyways, regardless of the tape issue, sure would be great if it would say "Darn, can't write to this tape, I'll try another one and continue on." :wink:
VMware 6
Veeam B&R v9
Dell DR4100's
EMC DD2200's
EMC DD620's
Dell TL2000 via PE430 (SAS)
Dima P.
Product Manager
Posts: 14396
Liked: 1568 times
Joined: Feb 04, 2013 2:07 pm
Full Name: Dmitry Popov
Location: Prague
Contact:

Re: [Feature Request] Retry tape job

Post by Dima P. »

Count it as noted :wink:
alanbolte
Veteran
Posts: 635
Liked: 174 times
Joined: Jun 18, 2012 8:58 pm
Full Name: Alan Bolte
Contact:

Re: [Feature Request] Retry tape job

Post by alanbolte »

If you are running v9 and are unable to erase a tape because of its block size, please request a hotfix from support.
rreed
Veteran
Posts: 354
Liked: 72 times
Joined: Jun 30, 2015 6:06 pm
Contact:

Re: [Feature Request] Retry tape job

Post by rreed »

Thanks Alan, #01821607.
VMware 6
Veeam B&R v9
Dell DR4100's
EMC DD2200's
EMC DD620's
Dell TL2000 via PE430 (SAS)
efd121
Enthusiast
Posts: 46
Liked: 2 times
Joined: Aug 07, 2015 8:45 pm
Full Name: David Engler
Contact:

Re: [Feature Request] Retry tape job

Post by efd121 »

+1 for me also. I just had a tape job that was backing up over 5M files and it was at 60% after 45hrs and it just died with a very generic "Job has failed unexpectedly" Seems my only option to start over? It would be great to just retry instead of losing 45hrs of time.
Dave
Dima P.
Product Manager
Posts: 14396
Liked: 1568 times
Joined: Feb 04, 2013 2:07 pm
Full Name: Dmitry Popov
Location: Prague
Contact:

Re: [Feature Request] Retry tape job

Post by Dima P. »

David,

Thanks for the input. By the way, can you elaborate how you tape library is connected to the source repository?
efd121
Enthusiast
Posts: 46
Liked: 2 times
Joined: Aug 07, 2015 8:45 pm
Full Name: David Engler
Contact:

Re: [Feature Request] Retry tape job

Post by efd121 »

I have a Spectra Logic T380 library that has a single drive partition connected via fiber channel (8Gb) to a physical W2K16 server. The 5M files being backed up are on a virtual machine
Dima P.
Product Manager
Posts: 14396
Liked: 1568 times
Joined: Feb 04, 2013 2:07 pm
Full Name: Dmitry Popov
Location: Prague
Contact:

Re: [Feature Request] Retry tape job

Post by Dima P. »

The 5M files being backed up are on a virtual machine
Have you tried new v9.5? We have improved engine for file to tape jobs with a huge number of source files which should come handy in case like yours.
efd121
Enthusiast
Posts: 46
Liked: 2 times
Joined: Aug 07, 2015 8:45 pm
Full Name: David Engler
Contact:

Re: [Feature Request] Retry tape job

Post by efd121 »

Yes I have tried 9.5 that is the version I was running when this job failed
Dave
Chris.B
Lurker
Posts: 2
Liked: 1 time
Joined: Jan 09, 2017 10:20 am
Full Name: Chris B
Location: UK
Contact:

Re: [Feature Request] Retry tape job

Post by Chris.B » 1 person likes this post

I came across this thread as we had a couple of backups fail on our yearly GFS tape job over the weekend, being able to retry failed tape jobs would be really useful.

My solution was to clone the failed GFS tape job and then edit it to remove the VMs which had backed up successfully so only the VMs which had failed were left. I then ran the cloned job by right-clicking and selecting Active Full > Yearly. This ensured that only the failed VMs were processed and the backups ended up on the correct media set.

Chris
sasilik
Expert
Posts: 104
Liked: 13 times
Joined: Jun 12, 2014 11:01 am
Full Name: Markko Meriniit
Contact:

Re: [Feature Request] Retry tape job

Post by sasilik »

+1 for a little bit different "retry tape job". Seems that if there is some problems reading source files then tape job fails. Could the tape jobs have similar option as VM jobs - retry N times and wait N minutes before retrying. Maybe there is some temporary problem and maybe there is only problem with reading one file and job could go on and report failing files. For now Veeam fails entire tape job when these kind of errors occur.
bct44
Veeam Software
Posts: 110
Liked: 27 times
Joined: Jul 28, 2022 12:57 pm
Contact:

[MERGED] Features Request Retry button and retry x times options

Post by bct44 »

Hello,

From my POV and current production problematic, it lacks a retry button on backup/files to tape jobs as it possible on backup jobs or backup copy jobs. Even V12 is more stable for backup to tape jobs on large scale infra, we're lacking options like backup jobs or backup copy jobs to enable automatic retry for errors specially on mission critical jobs.

Could you consider my request?

Thanks!
HannesK
Product Manager
Posts: 14287
Liked: 2877 times
Joined: Sep 01, 2014 11:46 am
Full Name: Hannes Kasparick
Location: Austria
Contact:

Re: [Feature Request] Retry tape job

Post by HannesK » 1 person likes this post

Hello,
yes, I also got that request recently at an event and we count your request +1 👍

Best regards,
Hannes
JGM2023
Influencer
Posts: 14
Liked: 4 times
Joined: Jun 09, 2023 12:47 pm
Full Name: JGM
Contact:

[MERGED] Feature Request - Retry Certain Machines In a Tape Job

Post by JGM2023 »

Hello,

Since the Improvements In 12 & 12.1 with Increasing the objects In Backup Jobs, It has now caused a knock on effect with Tape Jobs, For example If we put over 100 machines in a Backup Job, This could be 30TB+ that we need to write to Tape. We run full backups on the 1st of the Month At times this does not complete before the backup has to start again.

Is there a reason that the function to retry machines Is not Implemented In Tape Jobs? If this was the case and It did not have to start a whole full again for the whole Job, then this wouldn't be so much of an issue.
Mildur
Product Manager
Posts: 8549
Liked: 2223 times
Joined: May 13, 2017 4:51 pm
Full Name: Fabian K.
Location: Switzerland
Contact:

Re: [Feature Request] Retry tape job

Post by Mildur » 1 person likes this post

Hi JGM

I moved your request to a similar topic. There is no retry button today.
We are aware about the request to have a retry option for tapes to only process "not yet processed data".
Your vote is counted as +1.

Best,
Fabian
Product Management Analyst @ Veeam Software
johannesk
Expert
Posts: 146
Liked: 31 times
Joined: Jan 19, 2016 1:28 pm
Full Name: Jóhannes Karl Karlsson
Contact:

Re: [Feature Request] Retry tape job

Post by johannesk » 1 person likes this post

@Mildur,

+1 vote from me to have a retry option on tape jobs.
Samba222
Service Provider
Posts: 79
Liked: 24 times
Joined: Jan 18, 2017 11:54 am
Full Name: Ronald
Contact:

Re: [Feature Request] Retry tape job

Post by Samba222 »

+1
Felix4002
Novice
Posts: 3
Liked: 3 times
Joined: Aug 12, 2017 9:10 am
Full Name: Roland
Contact:

Re: [Feature Request] Retry tape job

Post by Felix4002 »

+ 1 Vote from me for the Retry Button
Post Reply

Who is online

Users browsing this forum: No registered users and 23 guests