Restores not completing Veeam Plugin for RMAN v12

[wmaxime75]

Veeam Support Case: 06049546
Since upgrading our entire environment to Veeam v12 none of the restores attempted by the Oracle DBAs are completing. Yes we are using the new procedure which includes running OracleRMANConfigTool --get-backup-for-restore, selecting the source client for restores, etc. DBA runs the scripts, files are being restored but eventually the restore just sits there, never completes. All this was working fine with v11. I did notice some errors in the plugin logs:

WARN|Failed to send data transfer status: code [%1] message [%2]
[11.05.2023 12:47:10.306] < 1> plugin_co| >> |No such device or address
[11.05.2023 12:47:10.306] < 1> plugin_co| >> |Failed to open pipe: /tmp/pipe{09f15d50-2839-4c6e-9a9d-de928973dee2}
[11.05.2023 12:47:10.306] < 1> plugin_co| >> |--tr:Failed to serialize data transfer state.

DBAs are seeing this in their logs:
ORA-27192: skgfcls: sbtclose2 returned error - failed to close file
ORA-19511: non RMAN, but media manager or vendor specific failure, error text:
   Transmission pipeline hanged, aborting process.
--tr:Backup client has failed to execute command.
--tr:Failed to restore data from the backup.

Been dealing with this issue for days now. DBAs unable to refresh their DBs. Application owners waiting. I have a webex session with Veeam scheduled for tomorrow...

[PetrM]

Hi Worrel,

It's not a known issue, for an unknown reason Data Movers just do not start data transmission. Let's see for what our support can figure out tomorrow. I'll ask our support team to prioritize your request.

Thanks!

[wmaxime75]

Case was eventually escalated. The first recommendation was to downgrade the plugin version to 11. Disappointing. Ironically, most of the DBAs after many retries, changing restore dates, changing number of allocated channels etc got some of the restores to complete. Some restores are still failing. Very weird. The restore starts, files are moving, but eventually transmission just stops. I check my Cisco 3260 storage servers, no connections to the target, everything goes idle until it eventually fails.

[PetrM]

Hello,

I think our engineer wanted to use such a test to narrow down the scope of possible root causes, as far as I know the downgrade helped once in another support case. Anyway, if it's not an option for you, I'm absolutely sure that our support will propose a suitable action plan soon, moreover, the case is already escalated.

Thanks!

[wmaxime75]

Ended up downgrading the plugin version to v11(11.0.101.1264). Restores are completing successfully. I would hate to have to downgrade 60+ clients. Not sure if this issue is only related to restoring to another server. In our testing we restored to original location.

[PetrM]

Let me discuss it with the support team. I fully understand the idea to downgrade a plug-in as a temporary workaround to quickly fix an urgent issue but I'd try to avoid downgrading 60+ clients.

Thanks!

[bct44]

interesting issue, we have not encountered any problems like that even v12 update. I'm just curious what is the targeted repo? OS of Oracle nodes? Oracle version?

[wmaxime75]

Restoring from Cisco 3260 storage servers. OS on Oracle nodes Solaris 11.4. Oracle version 19c.

[wmaxime75]

Update. Issue happened again: Case # 06164978 - Restore stuck on last data file. This time the DBA team got Oracle support involved. It appears the plug-in does in fact restore the data but never issues a finished signal to the client. Oracle support confirmed the data files were restored. Before proceeding, we had to engage the unix team to kill any veeam processes running on the client. DBA then proceeded to restore archive logs which were skipped. Oracle support had the DBA issue a SQL command reset the archive log(or something like that, I'm not a DBA). The database was successfully restored.

[PetrM]

Hello,

I asked our support team to prioritize a new case and to carry out the root cause analysis. However, it's too early to conclude that a new issue is related to the old one, maybe it's just a coincidence.

Thanks!

[wmaxime75]

Are there any known issues with NESSUS SECURITY VULNERABILITY SCAN software and Veeam v12 Rman plugin?

[PetrM]

Hello,

I'm not aware of any but I guess that our engineers may ask you to disable AV/security software temporarily just for testing purposes to isolate the issue.

Thanks!

[mark1117]

Hi,
In my environment,i have a similar problem,use v12.0.3.1423 plugin backup oracle rac 11.2.0.3.0 ,also report 'Failed to write data to pipe: /tmp/pipe{####}',downgrading the plugin version to v11.0.101.1264,everything is ok.Is it a known issue?

[PetrM]

Hello,

I'm not aware of this issue but I suggest you clarify this behavior with our support team. Also, please provide a support case ID so that we can keep an eye on it and make sure that it's on the right track.

Thanks!

[UweLandsmann]

Hi,
in our environment (Solaris 11, Oracle 19c, Veeam Plugin 12.0.0.1420) we have a similar problem.
Example 1: A Database is running on Server A. Backup/Restore with rman Scripts and Veeam Plugin runs fine.

Example 2: A Database in running on Server B. Backup with rman Scripts and Veeam Plugin runs fine. Restore the Database to Server A fails.
Agent.Source.Log shows: There was no finalized server sessions during the last 60 minutes
RmanPluginManager.log shows: plugin_co| Lease keeper: sending keep-alive request for lease [{c3394f55-b334-4f45-92db-5a94ed09812a}]

After downgrading to VeeamPluginforOracleRMAN-11.0.103.1265-1.SPARC all works fine.