Tape Back Fails - Exception from server: Shared memory connection was closed.

[sosowin]

VEEAM B&R 9.5.4.2866
AWS Storage Gateway Virtual Tape Backup

Does anyone have any idea what causes the tape backup to fail? Having looked at the logs, I haven't been able to determine the cause. The backup jobs do run successfully everyday.

Code: Select all

Error    [CTapeChainBackupUnit] [Incremental:vm:app01, task:PROD] failed
Error    Tape fatal error. (Veeam.Backup.Common.CCppComponentException)
Error    Data error (cyclic redundancy check). (Veeam.Backup.Common.CCppComponentException)
Error    Tape write error (Veeam.Backup.Common.CCppComponentException)
Error       in c++: Error code: 0x00000017
Error       in c++: TapeFileWriteStream failed to write 2097152 bytes.
Error       in c++: Cannot append file block to the end of file. File: [\\.\Tape3, pos: 6358907, blockOffset: 2048, size: 9808379904]. Write position: [1451229184].
Error       in c++: Unable to asynchronously write data block. Block identity: [Data block. Start offset: [1451229184], Length: [2097152], Area ID: [692].].
Error       in c++: Processing of asynchronous write requests has failed. Output file: [\\.\Tape3, pos: 6358907, blockOffset: 2048, size: 9808379904].
Error       in c++: Failed to process conveyored task.
Error       in c++: CTapeRemoteFileBackupClient failed to process client
Error       in c++: TapeRemoteFileTransport failed to backup file.
Error       in c++: TapeFileWriter backup file data to tape failed.
Error    Exception from server: Shared memory connection was closed. (Veeam.Backup.Common.CCppComponentException)
Error       in c++: Failed to read data from shared memory IO device.
Error       in c++ event: ClientErrorEvt
Error       at Veeam.Backup.AgentProvider.CClientAgentProtocol.WaitUntilWorking(Int32 timeout)
Error       at Veeam.Backup.AgentProvider.CClientAgentProtocol.DoAgentCommand(String command, String logCommand, Boolean doWait, Nullable`1 outputLogLevel)
Error       at Veeam.Backup.AgentProvider.CClientAgentProtocol.BackupSrvFileToTape(String srvFilePath, String tapePath, Int32 errHandlingType)
Error       at Veeam.Backup.AgentProvider.CBackupClient.BackupSrvFileToTape(String srvFilePath, String tapePathLink, Int32 errHandlingType)
Error    --- End of stack trace from previous location where exception was thrown ---
Error       at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
Error       at Veeam.Backup.Common.ExceptionFactory.ThrowNecessaryAggregateException(IEnumerable`1 exceptionsCollection)
Error       at Veeam.Backup.AgentProvider.CBackupClientImpl.ConstructException(Exception exception, String error)
Error       at Veeam.Backup.AgentProvider.CBackupClient.ConstructException(Exception ex, String error, Object[] args)
Error       at Veeam.Backup.AgentProvider.CBackupClient.BackupSrvFileToTape(String srvFilePath, String tapePathLink, Int32 errHandlingType)
Error       at Veeam.Tape.Core.CTapeVmBackupObjectAlgorithmSpecific.RunBackupOnAgent(EAgentErrorHandlingType errorHandling, ITapeAgentsSession agentsSession, IXmlLogger logger)
Error       at Veeam.Tape.Core.CTapeBackupObjectAlgorithm.Backup()
Error       at Veeam.Tape.Core.Jobs.Tasks.CTapeStorageBackupPerformer.Perform(CTapeVmBackupOibBuilder oibBuilder)
Error       at Veeam.Tape.Core.CTapeChainedVibsBackupAlg.RunBackup()
Error       at Veeam.Tape.Core.ResourceOwners.CTapeChainBackupUnitPerformer.PerformVibsBackup(ITapeBackupResource leasedResources)
Error       at Veeam.Tape.Core.ResourceOwners.CTapeChainBackupUnitPerformer.PerformChainBackup(Object resources)
Warning  [CTapeChainBackupUnit] Set completed with error for chain: Incremental:vm:app01, task:PROD
Warning  [CTapeChainBackupUnit] Tape fatal error.
Warning  Data error (cyclic redundancy check).
Warning  Tape write error
Warning  Exception from server: Shared memory connection was closed.
Warning  Tape fatal error.

[wishr]

Hi Solomon,

When you encounter a technical error it's always recommended to raise a support case immediately to get a solution asap, as our ability to troubleshoot through the forum is limited. Please, let us know your case ID once you have it for reference purposes. Also note, that according to our forum rules any topic not having a case ID will be eventually removed by forum moderators.

Thank you!

[sosowin]

My bad.

Here it is - Case# 03907137.

[soncscy]

Hey Solomon,

Check your local cache settings for the VTL. A few of my clients had awful issues with the local cache just not offloading fast enough -- basically, Amazon writes to the local cache, then offloads to AWS on its own time, so that you're not 100% gated by your network speed. What we were seeing was that the local cache was dramatically undersized, and the end result was that the jobs were topping-off the cache and then just returned random CRC errors.

What's your up-speed to AWS and also how is your local cache looking?

[Dima P.]

Hi folks,

Can confirm that cache is not being uploaded 'on the go' and might cause issues, so it's strongly recommended to allocate cache according to amount of data it should store at the specific point in time (basically the size of all emulated tapes required for a job run).

There is another implementation of the virtual tape library, unlike the native Amazon's, it can start the upload the data immediately when it lands the cache. In case you are interested -StarWind Virtual Tape Library. Cheers!

[sosowin]

The Cache is 3tb and the buffer is 2tb on AWS-Storage-Gateway VTL Server. Apparently, the VEEAM Server was running out of memory when TAPE Jobs were running.

Code: Select all

[11.12.2019 05:46:15] < 2932> cli| ERR |Send thread (channel) has failed.
[11.12.2019 05:46:15] < 2932> cli| >> |Tape fatal error.
[11.12.2019 05:46:15] < 2932> cli| >> |Data error (cyclic redundancy check).
[11.12.2019 05:46:15] < 2932> cli| >> |Tape write error
[11.12.2019 05:46:15] < 2932> cli| >> |--tr:Error code: 0x00000017
[11.12.2019 05:46:15] < 2932> cli| >> |--tr:TapeFileWriteStream failed to write 2097152 bytes.
[11.12.2019 05:46:15] < 2932> cli| >> |--tr:Cannot append file block to the end of file. File: [\\.\Tape3, pos: 6373874, blockOffset: 2048, size: 24185405440]. Write position: [13878951936].

[11.12.2019 05:47:25] < 10176> Adjusting system file cache limit (25 percent)
[11.12.2019 05:47:25] < 10176> Total physical memory installed: [8191 MB], available: [1861 MB].
[11.12.2019 05:47:25] < 10176> Maximum file cache size is already [2047 MB].

After increasing the memory, I no longer have the tape backup failure.

[Dima P.]

sosowin,

To clarify, was that a Tape Server component or Veeam B&R server (or you have both within the same machine)? Thanks!

[sosowin]

The memory was increased on VEEAM B&R server. Tape Server component and VEEAM B&R server are running on different VMs.

[Dima P.]

Understood, thank you for sharing!