Tape fatal error - Exception from server: Shared memory connection was closed.

[MIC00]

Hello,

we are currently have problems with Veeam V12 (12.1.2.172) and a backup to tape job.
Current CaseID: 07339900 (First CaseID: 07237816, already closed, because Veeam recommended to contact HPE / Quantum Support)

The job keeps failing on different VMs with the following error message. This mainly occurs with long backup jobs, but not only there:

Code: Select all

19.07.2024 13:15:49 :: [VMNAME] Full backup: Tape fatal error.
The system cannot find the specified path
Tape write error.
Exception from server: Shared memory connection was closed.
TapeRemoteFileBackupServer failed to process file E:\Veeam\Repositories\Repo1\Backup vSphere - General\VMNAME.vm-127400D2024-07-18T223017_573F_e21706fb-d31c-4bd8-bcf7-add146b67517.vsb, size: 950656340480.
Tape fatal error.
The system cannot find the specified path
Tape write error.
Exception from server: Shared memory connection was closed.
TapeRemoteFileBackupServer failed to process file E:\Veeam\Repositories\Repo1\Backup vSphere - General\VMNAME.vm-127400D2024-07-18T223017_573F_e21706fb-d31c-4bd8-bcf7-add146b67517.vsb, size: 950656340480.  

The job runs normally before and after this error. There is no specific pattern to why the job fails. It is not tied to a specific time, a specific written size, or a specific VM.
However, we are currently unable to achieve consistent full backups.

We use the following components:

Quantum SuperLoader 3 + IBM ULTRIUM 8 HH TAPE Drive (LTO 8 )
HPE ProLiant DL 380 Gen10 Plus (Windows Server 2022, 21H2) with HPE Smart Array E208e-p SR Gen10

The following has been tried so far:

- Replacement of the SAS controller (SmartArray) by HPE - No improvement
- Full write tests via IBM ITDT with Quantum Support - no problems were detected, the connection interruptions are caused by the server.
- All firmware updates and drivers for Tape Libary, Tape Drive and HP Server are up to date. - No improvement
- Switching off the antivirus program (TrendMicro) - No Improvement
- Veeam support indicated that this could be related to the GPO updates. The server is not part of a domain.
- Test with the Windows driver “Unknown media changer” and native SCSI commands in Veeam - No improvement

We have been unable to write correct backups to tape in this environment for several months now and urgently need help.
The problem has definitely occurred after an update to Veeam 12. Before that, the backup to tape was working fine.

Best regards,
Michael

[david.domask]

Hi Michael,

Thank you for the detailed write up and sorry to hear about the long-running issue. All the steps above (barring ITDT*) are good isolation steps; "Tape fatal error" means just that usually, some error occurred when writing to the tape. While I understand you had contacted Quantum and did not find any results, I still suspect that there is something getting missed as that particular error is rarely misleading, it's just sometimes hard to find.

I've contacted Support regarding the case and it looks like there is a new lead based on the most recent logs, so expect an update soon and let's see where that leads the case. Please remember that if at any time you have concerns about the processing of a case, you can always contact Support Management and express your concerns, and they will review the case.

* ITDT and HPE's Tape Tools, while quite useful, can be misleading when it comes to backup application troubleshooting -- the tools don't send data from the executing server typically without specific parameters and instead just initiate tests on the tape hardware itself, meaning that the tape server and the connections to the tape hardware are not tested thoroughly, so there's a "blind spot" which can result in accurate but easy to misinterpret results. Similarly, given that the issue is intermittent, it's entirely possible that everything was "A-Okay" when the test was done, but there is an intermittent issue.

But, let's wait to see what Support updates, and hopefully it's the lead needed to advance the case.

[Tecbil]

We are facing exactly the same issue with Quantum SuperLoader 3 and are out of ideas after checking everything.
Is there any helpful update on this?
Thank you!

[david.domask]

Hi Tecbil,

Sorry to hear about the challenges. For the case above, the root cause was identified as a hardware issue though we did not get confirmation from the Vendor on the behavior, but event log entries demonstrated storport errors that corresponded to the errors in the tape jobs.

Do you have a Support Case already for the issue? If not, please open a Support case and ensure logs for the failing job are included. (Use the 1st radio button, select one of the affected jobs for review)

Please share your case number here once created. Thanks!

[Tecbil]

Hi David

Yes, the support case is the following:
#07574515

[Moderator: Edited to include correct case number]

[david.domask]

Hi Tecbil,

Thanks for sharing, though I edited the post to remove your Support ID and include the case number (case numbers always start with 07*).

Checked it quick, not sure it's the same issue as the topic creators on first blush, but looks like Support has some additional items to check with you so please continue with Support. If possible, please confirm for them how the tape hardware is presented to the tape server (iscsi, SAS, fibrechannel, etc) and if any other backup applications are installed on the tape server which may lock the tape devices as well.