Tape fail (shared memory connection closed) Case # 02154519

[A.Lamsdell]

I am currently unable to reliably backup to tape due to a read error that occurs under certain circumstances, just curious if anyone else has bumped into this before and may have a suggestion.

These issues began after updating the drive system to LTO7 and adding an H830 for bolt on storage, although that could be circumstantial.

A rundown of the Veeam environment:
Windows Server 2016 Datacenter running on a R730XD
Connected to dual IBM ULT3580-HH7 running in a TL4000 via SCSI on a Dell 12Gbps SAS card

Also attached to the device is bolt on storage running through a PERC H830

It is possible to achieve successful backup results in testing but some backup sets seem to be causing CRC failures:

Synthetic full
Full backup: BackupSrvFileToTape failed Tape fatal error. Data error (cyclic redundancy check). Tape write error Exception from server: Shared memory connection was closed. Tape fatal error.

I looked into the logs and found more indepth error reports for these issues:

From the agent TapeVMBackup logs:

Code: Select all

[12.05.2017 04:31:26] < 10784> cli| >>  |Tape fatal error.
[12.05.2017 04:31:26] < 10784> cli| >>  |Data error (cyclic redundancy check).
[12.05.2017 04:31:26] < 10784> cli| >>  |Tape write error
[12.05.2017 04:31:26] < 10784> cli| >>  |--tr:Error code: 0x00000017
[12.05.2017 04:31:26] < 10784> cli| >>  |--tr:TapeFileWriteStream failed to write 2097152 bytes.
[12.05.2017 04:31:26] < 10784> cli| >>  |--tr:Cannot append file block to the end of file. File: [\\.\Tape0, pos: 2535870, blockOffset: 778240, size: 2525844227584]. Write position: [1948740747264].
[12.05.2017 04:31:26] < 10784> cli| >>  |--tr:Unable to asynchronously write data block. Block identity: [Data block. Start offset: [1948740747264], Length: [2097152], Area ID: [929232].].
[12.05.2017 04:31:26] < 10784> cli| >>  |An exception was thrown from thread [10784].
[12.05.2017 04:31:26] < 12756> alg|   Stopping processing threads....
[12.05.2017 04:31:26] < 10280> cli| Thread finished. Role: 'area reader : Receiver channel for \\.\Tape0, pos: 2535870, blockOffset: 778240, size: 2525844227584'.
[12.05.2017 04:31:26] < 10784> mt | Mark file as corrupted: E:\Backup\Eton Virtual Machines\Low Priority Backup Job To UKVEEAM01D2017-05-05T190047_432094dc-b78a-493f-a8c9-29684dccf601.vsb
[12.05.2017 04:31:26] < 10784> mt | Mark file as corrupted: E:\Backup\Eton Virtual Machines\Low Priority Backup Job To UKVEEAM01D2017-05-05T190047_432094dc-b78a-493f-a8c9-29684dccf601.vsb ok.
[12.05.2017 04:31:26] < 10784> mt | WARN|[Low Priority Backup Job To UKVEEAM01D2017-05-05T190047.vbk] finalized as corrupted.

And from last nights run (agent TapeBmBackup.log):

Code: Select all

[15.05.2017 17:45:56] < 9404> cli| An error generated by server side will be rethrown. 
[15.05.2017 17:45:56] < 9404> cli| An error occurred on client side. 
[15.05.2017 17:45:56] < 9404> cli| Removing stdin/stdout redirector: '1334262889840'. 
[15.05.2017 17:45:56] < 9404> ERR |Tape fatal error. 
[15.05.2017 17:45:56] < 9404> >> |Data error (cyclic redundancy check). 
[15.05.2017 17:45:56] < 9404> >> |Tape write error 
[15.05.2017 17:45:56] < 9404> >> |--tr:Error code: 0x00000017 
[15.05.2017 17:45:56] < 9404> >> |--tr:TapeWriter failed to end session 
[15.05.2017 17:45:56] < 9404> >> |--tr:TapeWriteSession failed to finish session 'Backup set 15/05/2017 12:16:08'. 
[15.05.2017 17:45:56] < 9404> >> |--tr:TapeWriteSession failed to finish session 'Backup set 15/05/2017 12:16:08', metadata: . 
[15.05.2017 17:45:56] < 9404> >> |Exception from server: Shared memory connection was closed. 
[15.05.2017 17:45:56] < 9404> >> |--tr:Failed to read data from shared memory IO device. 
[15.05.2017 17:45:56] < 9404> cli| Starting to send error string [Tape fatal error. 
[15.05.2017 17:45:56] < 9404> cli| Data error (cyclic redundancy check). 
[15.05.2017 17:45:56] < 9404> cli| Tape write error 
[15.05.2017 17:45:56] < 9404> cli| --tr:Error code: 0x00000017 
[15.05.2017 17:45:56] < 9404> cli| --tr:TapeWriter failed to end session 
[15.05.2017 17:45:56] < 9404> cli| --tr:TapeWriteSession failed to finish session 'Backup set 15/05/2017 12:16:08'. 
[15.05.2017 17:45:56] < 9404> cli| --tr:TapeWriteSession failed to finish session 'Backup set 15/05/2017 12:16:08', metadata: . 
[15.05.2017 17:45:56] < 9404> cli| Exception from server: Shared memory connection was closed. 
[15.05.2017 17:45:56] < 9404> cli| --tr:Failed to read data from shared memory IO device. 
[15.05.2017 17:45:56] < 9404> cli| --tr:event:3: 
[15.05.2017 17:45:56] < 9404> cli| ] to manager. 
[15.05.2017 17:45:56] < 9404> cli| Sending error string completed. Tape fatal error. Data error (cyclic redundancy check). Tape write error Exception from server: Shared memory connection was closed. Tape fatal error. 

They are from two separate backup sets, but I am just trying to get some clarity on what is going wrong and where as I am not hugely experienced with these error logs and google hasn't helped much.

Thanks in advance,

[lyapkost]

Hi Antony.

The CRC-error indicates to a hardware problem. Further investigation should be done by both Veeam and hardware vendor support. Additional question is whether the failed jobs used old tapes or new LTO-7 ones?

[A.Lamsdell]

They utilise the new tapes exclusively, I guess now the problem is trying to identify exactly where the hardware faults are occurring.

I've seen good and bad runs happen on both LTO7's with various tapes, the SAS card didn't change during the swap-over to LTO7 and functioned absolutely fine on LTO6.

The next stage for my testing will be to revert to LTO6 and see if the issue occurs on that.

Thanks for your feedback and patience,

[lyapkost]

Thank you for sharing. Please keep us updated on the case.