tape backups broken after 9.5 upgrade

[psychodad]

Support ticket 01999446

Not getting much help from support so I thought I would post here.

First the infrastructure layout: We have two dedicated proxies direct connected to two HP tape libraries. All equipment is new and dedicated. Everything worked on the updated version of 9.0.

After upgrading to 9.5 all tape jobs failed with various errors, the new tapes in the drive were retired by VEEAM and set as "corrupted". The first errors were:

Code: Select all

12/5/2016 9:08:39 PM :: All tape drives are disabled.  
12/5/2016 9:48:22 PM :: Library HP 1x8 G2 AUTOLDR 4.50 does not contain any enabled tape drives  
12/7/2016 5:37:48 PM :: No tape libraries are online  

I figured a good firm reboot and kick in the pants for all equipment might be in order as is sometimes the case after upgrades. Two full cycles down on all related equipment, new tapes, new catalogs, and then this error:

Code: Select all

12/8/2016 9:29:09 AM :: TapeDrive alert: Media statistics have been lost at some time in the past.   
12/8/2016 9:59:06 AM :: Failed to prepare to start new tape session: Tape positioning error: Tape error: '23' (Data error (cyclic redundancy check).) 
Tape positioning error: Tape error: '23' (Data error (cyclic redundancy check).)   
12/8/2016 9:59:18 AM :: TapeDrive alert: The tape drive is having problems reading data. No data has been lost, but there has been a reduction in the performance of the tape..   
12/8/2016 9:59:18 AM :: TapeDrive alert: The operation has stopped because an error has occurred while reading or writing data that the drive cannot correct..   
12/8/2016 9:59:18 AM :: TapeDrive alert: Your data is at risk:.  Copy any data you require from this tape. Do not use this tape again. Restart the operation with a different tape.  
12/8/2016 9:59:18 AM :: TapeDrive alert: The tape is damaged or the drive is faulty. Call the tape drive supplier helpline..   

Now, to have 2 physically different tape drives and server proxies go bad at the exact same time? Immediately following a software upgrade? Doubtful. So I started trying the jobs manually one at a time. Success! Tape backups work.

Two jobs active? Fail

So what seems to happen is the jobs will complete at 99% and wait for the success message, but before that job is complete at 100% - the next job kicks in, takes over the tape drive and that's when the errors start to kick in. I'm not sure, but it seems you need to wait for one job to complete successfully before firing off the tape drive again.

Support ticket has been entered last week, but no progress to report. Any help is greatly appreciated.

[Dima P.]

Hi psychodad,

The behavior does not look like expected. Any updates from support side?

[Andi0r]

Hi,
we have the same problem with veeam 9.5, but we have only one proxy and one HP Tape Drive LTO5. Bevore 9.5 we never had this problem.
is there any solution?

BR
Andreas

[Dima P.]

Hello Andreas,

Please report an issue to our support team and share the case ID. Thanks in advance.

[Andi0r]

Hi,

in the meantime we have replaced the SAS Controller and also the Library with another Tape drive.
the case is: 02059980

With the new Lib and SAS Card we have also a problem with the performance, before with LTO5 we always had 130mb/s to tape, now it is between 30-130 mb/s and the backup takes 7 hours more than before.

BR
Andreas

[Andi0r]

The Problem was solved temporary with uninstalling VEEAM One, but now the problem appears again after some weeks.

Does anybody have the same problem?

Code: Select all

Full backup: Failed to continue tape backup session on the next tape medium ContinueBackupSrvFileToTape failed Tape fatal error. The specified network resource or device is no longer available. Tape write error Exception from server: Shared memory connection was closed. Tape fatal error.


Unloading tape A00048 from Drive 1 (Server: W8ATBACKUP02, Library: EXABYTE MAGNUM 224 C306, Drive ID: Tape0) to Slot 9 Error: Failed to call RPC function 'StorageMediaRemoval': The system cannot find the file specified. Problem with CreateFile in '\\.\Tape0'.
[W8ATBACKUP02.at.chiesigroup.lan] [TapeService] StorageMediaRemoval Failed Failed to call RPC function 'StorageMediaRemoval': The system cannot find the file specified. Problem with CreateFile in '\\.\Tape0'. The system cannot find the file specified.

[veremin]

The Problem was solved temporary with uninstalling VEEAM One, but now the problem appears again after some weeks.

Not sure how Veeam ONE could cause an original issue. Neither am I aware of how deleting it could fix the problem.

Anyway, kindly, open a separate ticket for the new issue and let support team investigate it directly.

Thanks.

[migeauxy]

Hi Guys

Exactly the same issue as you, but I don't have Veeam One installed

Any updates on this issue. Running VBR 9.5 update 3a

Thanks

[veremin]

Can you tell us your support ticket ID? Thanks.

[migeauxy]

Hi folks

After days of testing and troubleshooting, I managed to find a workaround to avoid the "Full backup: Failed to continue tape backup session on the next tape medium ContinueBackupSrvFileToTape failed Tape fatal error. The specified network resource or device is no longer available. Tape write error Exception from server: Shared memory connection was closed. Tape fatal error." error

it's not about versionning, uninstalling/reinstalling, patching or updating firmwares and drives (which I did).

On the Veeam proxy, just disable the "portable device enumerator service" service. As simple as that ! (service is by default in manual startup mode (Triggered)).

What I can't understand, is the "process" that triggers this service startup, because I didn't fin the source issue.

Nevertheless; We have 2 proxies, I disabled it on one of them, and tape jobs ran to success, when on the second one where the service is not disabled, tape jobs continued to fail.

@Veeam support : would you have an explanation ?

Thanks

[veremin]

In order for us to provide an explanation we need to check debug logs as well as your environment. Our support engineers will be able to do it once you open a support ticket with us. Thanks!

[Keyn28]

Hello
We also have the same error: "TapeDrive alert: Your data is at risk: Do not use this tape again."
Version 9.5 Update 3a

[Dima P.]

Keyn28,

I am sorry for the delay, but we need to ask you to open support case: this error code might be generated by the device itself, so debug log analysis is required to identify the root cause. Please do not forget to share the case ID in this thread. Thank you.