Veeam 9.5 Update 3 RPC Error with Data Domain Boost

[Wtrippler]

Since upgrading to 9.5 update 3, we are getting this error when trying to access our data domain devices:
Failed to call RPC function DDBoostSetCreds

Support fixed this once by restarting Veeam services and running the VeeamPluginsHostX64 application from C:\Program Files (x86)\Veeam\Backup Transport\x64

It was fine for 2 days then all of our jobs failed last night with the same error. Any ideas?

Thanks!
Wendy

[veremin]

Kindly, re-open ticket with our support team and let them investigate it further. Thanks.

[javis]

We also are experiencing this exact same error with out data domain after installing 9.5 update 3. We have a ticket opened with Veeam support.

[rntguy]

We have the exact same error. Any luck with this?

[rntguy]

Since upgrading to 9.5 update 3, we are getting this error when trying to access our data domain devices:
Failed to call RPC function DDBoostSetCreds

Support fixed this once by restarting Veeam services and running the VeeamPluginsHostX64 application from C:\Program Files (x86)\Veeam\Backup Transport\x64

How did they 'run' that? I just double clicked it but not sure what to look for.

[javis]

This issue had to do with our remote proxy server which is on a different subnet. We've rebooted our Data Domain, proxy server, Veeam server and still didn't solve the issue. I noticed our Veeam server could not establish a connection to our remote proxy on port 6160. I restarted the Veeam Installer Server (our version is 9.0.0.1715) on our remote proxy server and connection was established on port 6160 and this has been working great ever since.

[tkeith]

We are having similar issues and have missed backups the last two nights!
All our issues started after we updated to Update 3 last tuesday...

Getting different errors:
Error: Failed to call RPC function 'DDBoostFcIsExist': The remote procedure call was cancelled. RPC function call failed. Function name: [DoRpc].
Error: Failed to call RPC function 'DDBoostSetCreds': The remote procedure call was cancelled. RPC function call failed. Function name: [DoRpc].
Error: The RPC server is unavailable. function call failed. Function name: [GetSvcVersion].

These error happens against the data domain, mount servers, and now the backup server itself. We seem to have the installer service crash

We back up thousands of VMs and this can't be happening!

Currently under Veeam support case: 02464083

[omegagx]

Did you ever get this resolved?

[UFE]

Hi there,
we had a nearly same problem.
Error: Failed to call RPC function 'TestCompatible': Error code: 0x80070583. Cannot initialize COM runtime.
here is the solution from Veeam support

"It is much pretty common with Update 3 when 3rd party monitoring tools proadcastrsVeeam with PowerShell queries and it affects our backups.
I would recommend the following steps:

1) Disable zabbix service
2) Restart Veeam server
3) Make sure that service is disabled
4) Open Veeam and re-try the job

You can use SNMP instead of Powershell as a workaround or refer to 3rd party software in order to have latest updates

Please keep me updated. I will be happy to help or answer your questions. Have a great day!
Best regards,
Mikhail Zolotov
Veeam Support"

After disabling Zabbix agent, backups works normally. Do you have some monitoring tools there?

[Sindhuja]

Hello All,

We observed a weird issue at one of our customers, possibly related to a combination of Veeam backup & replication 9.5 Update 3, Data domain and Powershell scripts.
All backups (which were successful so far) failed after the execution of our utility that queries Veeam Server using Powershell Cmdlets (as exposed by VeeamPSSnapIn).

Code: Select all

"Error: Failed to call RPC function 'DDBoostSetCreds': The remote procedure call was cancelled. RPC function call failed. Function name: [DoRpc]."

After some lookout, we came across the below thread,
https://forums.veeam.com/veeam-backup-r ... ml#p273664

This thread mentions, "With Veeam Backup & Replication 9.5 Update 3, the use of Powershell Cmdlets queries may result in a backup failure".
This is very unfortunate!

We would like to know the following:

• Is this a known issue and has it been communicated to the customers?
• Is there a patch available to fix this issue? If not available, any ETA on the patch?
• Is this related to any specfic Powershell Cmdlet which has an alternative approach and can be avoided?

Regards,
Sindhuja

[veremin]

Few questions to start with:

* Have you already reached our support team?
* Have you tried the workaround mentioned above?

Thanks.

[omegagx]

This is exactly the type of issue that should have a knowledge base article written up about. It would allow the customer to find a solution (disable monitoring such as zabbix) on the Veeam website instead of wasting time calling support. If this is a known issue that was already experienced by other customers, why not document it instead of forcing everyone to call in?

[foggy]

Support has a hotfix for this issue, so please contact them to get it.

[Sindhuja]

Thank you for the response!
We will reach out to Veeam support to help us with the hot fix.

[Gostev]

This is exactly the type of issue that should have a knowledge base article written up about. It would allow the customer to find a solution (disable monitoring such as zabbix) on the Veeam website instead of wasting time calling support. If this is a known issue that was already experienced by other customers, why not document it instead of forcing everyone to call in?

Simply because it did not pass the threshold on the number of support cases opened. With up to 1000 support cases open daily, obviously they have to approach KB article creation selectively. Thanks!

[oncominaspa]

We have the same problem here.

After upgrade, only two jobs executed well, and then: Error: Failed to call RPC function 'TestCompatible': Error code: 0x80070583. Cannot initialize COM runtime.
If we restart the Veeam Backup Service, more two jobs execute well, and the third fail again.

I will try to disable the zabbix agent.

[foggy]

I'd contact support to verify the issue and get the hotfix.

[omegagx]

What is that threshold? I think that *ANY* issue that requires an actual hotfix because Update 3 broke something in terms of functionality ABSOLUTELY requires a corresponding KB article. Unless you already released over a 1000 hotfixes for 9.5 Update 3 but I sincerely hope that's not the case.

Simply because it did not pass the threshold on the number of support cases opened. With up to 1000 support cases open daily, obviously they have to approach KB article creation selectively. Thanks!

[oncominaspa]

We have the same problem here.

After upgrade, only two jobs executed well, and then: Error: Failed to call RPC function 'TestCompatible': Error code: 0x80070583. Cannot initialize COM runtime.
If we restart the Veeam Backup Service, more two jobs execute well, and the third fail again.

I will try to disable the zabbix agent.

After disable the Zabbix agent service, the VBR works ok!

PS: I think that is a issue with the update 3 and Zabbix agent. I hope Veeam team can resolve soon this issue! Until that, our VBR server monitoring with zabbix is disabled.

[jah2323]

awesome! as i receive the error at random times and on differing veeam servers. it is usually related to windows updates, so i thought, and uninstalling the last one would usually fix it. weird that this time, the server waited weeks to have the problem, and it only showed up as windows 2016 has been telling me it has updates to install. I will get with support here shortly.

[zag]

I just got this error as well, after a good call with support we found the DLL in C:\Windows\Veeam\Backup and Replication on the host was corrupted or had reverted to an older version.

Re-installing the agent on the host fixed this issue.