Veeam and StoreOnce - Error: Failed to call RPC function 'PluginsHost.StoreOnceReadFile':

[mike_stewart]

Good morning,
After upgrade to Veeam 12 all of my StoreOnce repository Jobs start to fail.
I Have a StoreOnce connected through FC to a Gateway Veeam Proxy.

VEEAM CASE: #05886962 (Awaiting RnD response)

Errors:
"Failed to call RPC function 'PluginsHost.StoreOnceReadFile': The remote procedure call was cancelled. RPC function call failed. Function name: [DoRpcWithBinary]. Target machine: [XX.XX.XX.XX:6160]."
"Failed to call RPC function 'PluginsHost.StoreOnceTruncateFile': The remote procedure call was cancelled. RPC function call failed. Function name: [DoRpcWithBinary]. Target machine: [XX.XX.XX.XX:6160]."
"Failed to call RPC function 'PluginsHost.StoreOnceWriteFile': The remote procedure call was cancelled. RPC function call failed. Function name: [DoRpcWithBinary]. Target machine: [XX.XX.XX.XX:6160]."
After that, ALL the Jobs that come after are failing too with timeouts.

I noticed that the Veeam are locking files and not releasing them in the StoreOnce. After the error, I have multiple files with 0bytes in StoreOnce.

I don't have any issues in the event viewer from the Gateway/Proxy Server. I don't have Firewall and AntiVirus enabled in V&B server and Gateway/Proxy.
Before the Upgrade all was working as expected.

Does anyone with the same situation? Or some kind of workaround?
I have urgency in this resolution, because it's affecting all of my production virtual machines and databases!

Thanks

[PetrM]

Hi Sergio,

Yes, looks like an urgent issue but we cannot perform efficient troubleshooting over the forum posts. The best way is to let our engineers and RnD team work on it. Anyway, I contacted our support team leaders and asked them to look for options to accelerate the research of the root cause and to look for some temporary workarounds.

Thanks!

[mike_stewart]

Thanks for the quick reply.
I can say that the only way for the Backups to work again is to restart the Veeam gateway server and delete the files with 0bytes in the StoreOnce.
After this procedure the Backups run fine and after 1/2 days they fail again with the same error.

I'm waiting for news about the case and thank you for reinforcing this situation with Veeam engineers and developers.

Regards.

[foggy]

Hi Sergio, this is an already known issue - we had a few similar cases after the v12 release. This comes down to an issue with the storage FC connections limit, the fix is in the works.

[foggy]

The fix for this issue will be included in the next v12 patch and the private fix for the latest version is also available.

[mike_stewart]

Good afternoon,
I have already applied the private patch and in the next few days I will check that everything is fully operational.
Thank you very much.

Regards.

[mike_stewart]

Good morning,
I can confirm that the private patch fixed the issue.
I have several days in a row without any problem with StoreOnce.
Thanks!

Regards.

[foggy]

Hi Sergio, appreciate the feedback.

[GabGoe]

Hi,
we are also affected by the bug and applied the mentioned private patch.

Is the fix included in the latest official patch P20230412? The release notes are not clear about that and I don't want to break things again.

Thanks!

[mike_stewart]

Good morning,
The release notes are not clear on this point.
I don't know if Patch P20230412 includes this private fix. Can Veeam confirm it?
Thanks.

[EShutov]

Hi,
The fix #483431 is included in v12 P20230412.