Case # 01012365 - The RPC server is unavailable RPC function

[jgq85]

Looking to get some community help on this issue.
Server OS is 2012 R2.
Veeam Backup & Replication 8.0.0.2030

Backups job were failing due to some RPC server errors:
The RPC server is unavailable RPC function
The RPC server is unavailable
Failed to open management RPC connection ro proxy service Port 6162

etc....

but I think I've narrowed it down to the reason being the "Veeam Data Mover Service" not running since it's not running, and the backups appear to start correctly after I start the service.
The server is set to Automatic, so I'm not sure what's going on.

Oddly, Google search results show any others experiencing issues with these service. At least from what I could find on the front page with my browser...

I started the service, but it seems hours later or so I think it stops running, and I have to manually re-start it for backup jobs to work.
At this point I've disabled Symantec Endpoint Protection, even though it's not causing issues to my knowledge or any alerts, but just trying to see if that helps.

[foggy]

John, have you looked into Event Viewer for any events related to that service?

[jgq85]

There's nothing I can find in the event logs about the service.

This isn't even an isolated issue with backups, it's actually a problem with adding Veeam backup proxy servers as well.

I have 8 VM's deployed running 2012 R2 Core. Same template, same domain, same subnet, same firewall group policies.
I can add about half as New VMWare Proxy, but the rest of them come back with "Failed to connect to transport service on host '[proxy name here]'. Failed to connect to Veeam Data Mover Service port 6162. Failed to open management RPC connection to proxy service port 6162.
When I check on those servers that I can add, I find VeeamTransportSvc is indeed running.
When I check on those servers that I cannot add, I find that VeeamTransportSvc is stopped.

tell me how this error would come up on some, but not others, if they are all fresh identical provisioned servers?

seems strange.

No anti-virus is installed on any of these, either.
With this brand new deployment I've had to add a looping powershell script to check for and turn on the VeeamTransportSvc service about every 5 mins (if it isn't in a running state). I may have to deploy this scheduled tasks to all proxy backup servers as well if there's no fix I can find.

[foggy]

Let's wait for what our engineers could tell after reviewing the logs they've requested.

[jgq85]

Ok thanks I have uploaded per request

[Tobias_Elfstrom]

Just adding a lame "me too" comment.

I had two cases open on this almost a year ago I think with no success. Lots of logs was sent but nothing was found, my workaround was like yours, scheduled checks and restarts of services. I still do this on two implementations now on version 8 so... I guess it's just how it is. :/

[jgq85]

Just adding a lame "me too" comment.

I had two cases open on this almost a year ago I think with no success. Lots of logs was sent but nothing was found, my workaround was like yours, scheduled checks and restarts of services. I still do this on two implementations now on version 8 so... I guess it's just how it is. :/

I looked around and it looks like Veeam services issue is related to SCCM.

Per this article http://www.veeam.com/kb1721 last modified 7/8/2015:

Challenge
Veeam services stop at the same time every night.
Cause
Microsoft SCCM has a Health Agent Task scheduled to run over night. The task does not support WMI 3.0. WMI 3.0 is incorporated in Veeam Backup & Replication as a dependency. When the Health Agent clears the WMI repository, it kills Veeam services in the process. You might see other WMI errors in the event logs, or a message about an SMS Agent starting shortly before.


I’ve confirmed the “Configuration Manager Health Evaluation” service exists on the Veeam servers and runs every day at 12:05am.
I’ve confirmed the Veeam Installer Service and Veeam Data Mover Service’s both stop at 12:05am as described in the article.

will try deleting/disabling those,may have to config sccm to not deploy it, don't know much about it yet, and will see if that helps.

[jgq85]

The root fix would require SCCM be updated to SP1 or later.
That corrects the scheduled task and WMI issues, and of course many other things according to each collection of release notes.
I located the particular issue we experienced that SP1 fixes here: https://support.microsoft.com/en-us/kb/2796086