Backup Failed to call RPC function

[serhan]

Hello,
when i start a backup, backup job failed.
Veeam version is 9.5.4 and my repository is Data Domain.
I opened a case, case number is 03887536.
Could you please help me to solve this error.

Code: Select all

Error;
[03.12.2019 10:10:10] <01> Error    Failed to call RPC function 'DDBoostGetStorageInfo': illegal operation. Err: 5037. Failed to connect to storage. Storage: datadomain, User: OSTUSER.. InParams: '<InputArguments><Storage value="datadomain" /></InputArguments>'. 
[03.12.2019 10:10:10] <01> Error    illegal operation. Err: 5037
[03.12.2019 10:10:10] <01> Error    Failed to connect to storage. Storage: datadomain, User: OSTUSER
[03.12.2019 10:10:10] <01> Error    --tr:Failed to complete RPC request. Method: [DDBoostGetStorageInfo], In XML: [<InputArguments><Storage value="datadomain" /></InputArguments>]
[03.12.2019 10:10:10] <01> Error    --tr:Failed to invoke RPC request
[03.12.2019 10:10:10] <01> Error    --tr:Failed to call DoRpc. CmdName: [DoSerialRpc] inParam: [<InputArguments/>].
[03.12.2019 10:10:10] <01> Error    --tr:Failed to call DoRpcWithBinary. Method: [DoSerialRpc], In xml: [<InputArguments/>]
[03.12.2019 10:10:10] <01> Error    --tr:Failed to call DoRpcWithBinary
[03.12.2019 10:10:10] <01> Error    --tr:Failed to do RPC call with serialized data, Strategy type: DefaultStrategy
[03.12.2019 10:10:10] <01> Error    --tr:Failed to call [DoSerialRpc], session id: {d9774d88-e6f2-4036-90cf-42c31689a381}
[03.12.2019 10:10:10] <01> Error    --tr:Failed to call [DDBoostGetStorageInfo] (System.Runtime.InteropServices.COMException)
[03.12.2019 10:10:10] <01> Error       at VeeamProxyClient.GenerateComException(basic_string<char\,std::char_traits<char>\,std::allocator<char> >* excMsg, Int32 errHrCode)
[03.12.2019 10:10:10] <01> Error       at VeeamProxyClient.RegenerateCLRException(exception* excBase)
[03.12.2019 10:10:10] <01> Error       at VeeamProxyClient.CCliProxyRpcInvoker.DoRpc(String cmdName, String inXml, String& outXml)
[03.12.2019 10:10:10] <01> Error       at Veeam.Backup.ProxyProvider.CProxyRpcInvoker.Call(String methodName, CProxyInvokeInputArg inputArgs, Boolean secureCall)
[03.12.2019 10:10:10] <01> Error    Failed to determine virtual synthetic DDBoost options
[03.12.2019 10:10:10] <01> Error    Failed to obtain information about storage datadomain (System.Exception)
[03.12.2019 10:10:10] <01> Error       at Veeam.Backup.ProxyProvider.CProxyDDBoostFileCommander.GetStorageInfo(String storageName)
[03.12.2019 10:10:10] <01> Error       at Veeam.Backup.Core.CDDBoostOptionsDeterminer.IsVirtualSyntheticsEnabled()
[03.12.2019 10:10:10] <01> Error       at Veeam.Backup.Core.CDDBoostOptionsDeterminer.IsVirtualSyntheticsEnabledSafe()
[03.12.2019 10:10:10] <01> Error    illegal operation. Err: 5037 (Veeam.Backup.Common.CCppCOMException)
[03.12.2019 10:10:10] <01> Error    Failed to connect to storage. Storage: cevvaldd.tcloud.local, User: OSTUSER (Veeam.Backup.Common.CCppComponentException)
[03.12.2019 10:10:10] <01> Error       in c++: Failed to complete RPC request. Method: [DDBoostGetStorageInfo], In XML: [<InputArguments><Storage value="datadomain" /></InputArguments>]
[03.12.2019 10:10:10] <01> Error       in c++: Failed to invoke RPC request
[03.12.2019 10:10:10] <01> Error       in c++: Failed to call DoRpc. CmdName: [DoSerialRpc] inParam: [<InputArguments/>].
[03.12.2019 10:10:10] <01> Error       in c++: Failed to call DoRpcWithBinary. Method: [DoSerialRpc], In xml: [<InputArguments/>]
[03.12.2019 10:10:10] <01> Error       in c++: Failed to call DoRpcWithBinary
[03.12.2019 10:10:10] <01> Error       in c++: Failed to do RPC call with serialized data, Strategy type: DefaultStrategy
[03.12.2019 10:10:10] <01> Error       in c++: Failed to call [DoSerialRpc], session id: {d9774d88-e6f2-4036-90cf-42c31689a381}
[03.12.2019 10:10:10] <01> Error       in c++: Failed to call [DDBoostGetStorageInfo]
[03.12.2019 10:10:10] <01> Error       at Veeam.Backup.ProxyProvider.CProxyRpcInvoker.Call(String methodName, CProxyInvokeInputArg inputArgs, Boolean secureCall)
[03.12.2019 10:10:10] <01> Error       at Veeam.Backup.ProxyProvider.CProxyDDBoostFileCommander.GetStorageInfo(String storageName)

[Dima P.]

Hello serhan,

Based on logs it looks like a connectivity/authentication error, but I bet you've already checked all the credentials and connection to DD. Please keep working with our support team as they can check the remaining portion of debug logs to find the root cause. Cheers!

[serhan]

Hello,
My problem fixed. credentials or connection didnot cause this error.
I use gateway server and gateway server has 50 CPU so i can back up 50 vmdk or file concurrently. at this conjuction, limit maximum concurrent task is 900 in DD, therefore this situation cause bottleneck between proxies and gateway server. because of concurrent task limit and gateway server CPU count, i took RPC error.
Concurrent task limit in DD has set to 50 and everything is fine for a while.