tape backups broken after 9.5 upgrade

Everything about backing up to tape

tape backups broken after 9.5 upgrade

Veeam Logoby psychodad » Mon Dec 12, 2016 4:34 pm

Support ticket 01999446

Not getting much help from support so I thought I would post here.

First the infrastructure layout: We have two dedicated proxies direct connected to two HP tape libraries. All equipment is new and dedicated. Everything worked on the updated version of 9.0.

After upgrading to 9.5 all tape jobs failed with various errors, the new tapes in the drive were retired by VEEAM and set as "corrupted". The first errors were:
Code: Select all
12/5/2016 9:08:39 PM :: All tape drives are disabled. 
12/5/2016 9:48:22 PM :: Library HP 1x8 G2 AUTOLDR 4.50 does not contain any enabled tape drives 
12/7/2016 5:37:48 PM :: No tape libraries are online 

I figured a good firm reboot and kick in the pants for all equipment might be in order as is sometimes the case after upgrades. Two full cycles down on all related equipment, new tapes, new catalogs, and then this error:

Code: Select all
12/8/2016 9:29:09 AM :: TapeDrive alert: Media statistics have been lost at some time in the past.   
12/8/2016 9:59:06 AM :: Failed to prepare to start new tape session: Tape positioning error: Tape error: '23' (Data error (cyclic redundancy check).)
Tape positioning error: Tape error: '23' (Data error (cyclic redundancy check).)   
12/8/2016 9:59:18 AM :: TapeDrive alert: The tape drive is having problems reading data. No data has been lost, but there has been a reduction in the performance of the tape..   
12/8/2016 9:59:18 AM :: TapeDrive alert: The operation has stopped because an error has occurred while reading or writing data that the drive cannot correct..   
12/8/2016 9:59:18 AM :: TapeDrive alert: Your data is at risk:.  Copy any data you require from this tape. Do not use this tape again. Restart the operation with a different tape. 
12/8/2016 9:59:18 AM :: TapeDrive alert: The tape is damaged or the drive is faulty. Call the tape drive supplier helpline..   

Now, to have 2 physically different tape drives and server proxies go bad at the exact same time? Immediately following a software upgrade? Doubtful. So I started trying the jobs manually one at a time. Success! Tape backups work.

Two jobs active? Fail

So what seems to happen is the jobs will complete at 99% and wait for the success message, but before that job is complete at 100% - the next job kicks in, takes over the tape drive and that's when the errors start to kick in. I'm not sure, but it seems you need to wait for one job to complete successfully before firing off the tape drive again.

Support ticket has been entered last week, but no progress to report. Any help is greatly appreciated.
psychodad
Lurker
 
Posts: 2
Liked: never
Joined: Mon Jul 25, 2016 7:12 pm
Full Name: Mike Amb

Re: tape backups broken after 9.5 upgrade

Veeam Logoby Dima P. » Wed Dec 14, 2016 9:04 am

Hi psychodad,

The behavior does not look like expected. Any updates from support side?
Dima P.
Veeam Software
 
Posts: 6242
Liked: 440 times
Joined: Mon Feb 04, 2013 2:07 pm
Location: SPb
Full Name: Dmitry Popov

Re: tape backups broken after 9.5 upgrade

Veeam Logoby Andi0r » Mon Feb 06, 2017 6:39 am

Hi,
we have the same problem with veeam 9.5, but we have only one proxy and one HP Tape Drive LTO5. Bevore 9.5 we never had this problem.
is there any solution?

BR
Andreas
Andi0r
Novice
 
Posts: 3
Liked: never
Joined: Fri Feb 13, 2015 6:56 am
Full Name: Andreas Lechner

Re: tape backups broken after 9.5 upgrade

Veeam Logoby Dima P. » Tue Feb 07, 2017 9:43 pm

Hello Andreas,

Please report an issue to our support team and share the case ID. Thanks in advance.
Dima P.
Veeam Software
 
Posts: 6242
Liked: 440 times
Joined: Mon Feb 04, 2013 2:07 pm
Location: SPb
Full Name: Dmitry Popov

Re: tape backups broken after 9.5 upgrade

Veeam Logoby Andi0r » Fri Feb 10, 2017 1:56 pm

Hi,

in the meantime we have replaced the SAS Controller and also the Library with another Tape drive.
the case is: 02059980

With the new Lib and SAS Card we have also a problem with the performance, before with LTO5 we always had 130mb/s to tape, now it is between 30-130 mb/s and the backup takes 7 hours more than before.

BR
Andreas
Andi0r
Novice
 
Posts: 3
Liked: never
Joined: Fri Feb 13, 2015 6:56 am
Full Name: Andreas Lechner

Re: tape backups broken after 9.5 upgrade

Veeam Logoby Andi0r » Wed Mar 15, 2017 11:06 am

The Problem was solved temporary with uninstalling VEEAM One, but now the problem appears again after some weeks.

Does anybody have the same problem?

Code: Select all
Full backup: Failed to continue tape backup session on the next tape medium ContinueBackupSrvFileToTape failed Tape fatal error. The specified network resource or device is no longer available. Tape write error Exception from server: Shared memory connection was closed. Tape fatal error.


Unloading tape A00048 from Drive 1 (Server: W8ATBACKUP02, Library: EXABYTE MAGNUM 224 C306, Drive ID: Tape0) to Slot 9 Error: Failed to call RPC function 'StorageMediaRemoval': The system cannot find the file specified. Problem with CreateFile in '\\.\Tape0'.
[W8ATBACKUP02.at.chiesigroup.lan] [TapeService] StorageMediaRemoval Failed Failed to call RPC function 'StorageMediaRemoval': The system cannot find the file specified. Problem with CreateFile in '\\.\Tape0'. The system cannot find the file specified.
Andi0r
Novice
 
Posts: 3
Liked: never
Joined: Fri Feb 13, 2015 6:56 am
Full Name: Andreas Lechner

Re: tape backups broken after 9.5 upgrade

Veeam Logoby v.Eremin » Wed Mar 15, 2017 6:27 pm

The Problem was solved temporary with uninstalling VEEAM One, but now the problem appears again after some weeks.

Not sure how Veeam ONE could cause an original issue. Neither am I aware of how deleting it could fix the problem.

Anyway, kindly, open a separate ticket for the new issue and let support team investigate it directly.

Thanks.
v.Eremin
Veeam Software
 
Posts: 13266
Liked: 968 times
Joined: Fri Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin


Return to Tape



Who is online

Users browsing this forum: No registered users and 2 guests