Backup Copy Jobs failing - RPC Server Unavailable - 00570956

[Dave-Departed]

Hi All,

Got this raised with support, but just wondering if anyone else has seen this - It's been plaguing us for a while now.

We keep seeing Backup Copy Jobs fail with a message stating that the RPC server is unavailable. It seems to be somehow related to WMI, as the only way we can get the jobs to run successfully again each time is to run the "winmgmt.exe /resetrepository" command on our VBR server. If we reboot the VBR server for updates or anything, it causes this behaviour. Or it can just happen intermittently and jobs will start failing after working fine for days. I don't really think this is a Veeam issue. Would you advise building a new VBR server from scratch and importing our backups?

Cheers

Dave

[veremin]

Quick search suggests that you're the first one to come across this issue. So, kindly, keep working with our support team. Most likely, thorough log analysis will shed a light on existing problem. Thanks.

[Dave-Departed]

Hi Vladimir,

Thanks for that, yeah I couldn't find anything about it really. I've raised the issue a few times with Veeam support, thinking that it was a Veeam error, but as I said, it points more towards WMI. Will wait and see what support says!

[Battlestorm]

typical have you tried any hotfixes post

as even if the hotfix doesn't list the issue you are having it will bring the WMI system up to date using the latest builds
what version of windows are you using?

Hot fix list for 2008 / R2
http://support.microsoft.com/kb/2591403

[Dave-Departed]

Thanks Battlestorm :p

Server is fully updated, and I've applied those hotfixes now. Once the current BCJ's are finished, I will reboot the server and see what happens!

[Dave-Departed]

Looking good so far since the WMI hotfixes, rebooted twice with seemingly no issues! Will update the thread after BCJ's have run tonight!

[Dave-Departed]

BCJ's failed last night, but after a server reboot (and no WMI reset), they worked fine. So the OPPOSITE behaviour since before the hotfixes were installed! Have asked Veeam support to hold the case until Monday, when I hope to have done some further testing.

[Battlestorm]

Just to confirm, your running W2k8r2?
Assuming you are fully up to date from windows update? As there is a hotfix dealing with RPC dealing http://support.microsoft.com/kb/2637692 but it's been superseded by a security fix http://support.microsoft.com/kb/2849470

Long shot is the enterprise rollup for Win7/2k8r2

http://catalog.update.microsoft.com/v7/ ... ?q=2775511
Then 3 fixes for the rollup
http://support.microsoft.com/kb/2775511

[Dave-Departed]

Yup, W2k8r2, fully patched and updated from Windows Update.

Will check these ones out and see what happens, thanks.

[GarionB]

Quick search suggests that you're the first one to come across this issue. So, kindly, keep working with our support team. Most likely, thorough log analysis will shed a light on existing problem. Thanks.

He's not the only one facing this issue...we are also having this issue on v7 patch 3, running Windows 2012 R2 on multiple servers. We also have a case with support and this is very frustrating.

[Dave-Departed]

I have just learned that the RPC issues, which have resulted in missed BCJ cycles, have rendered my offsite backups useless. I can't restore from any incremental point in the backup chain of the offsite backups, only from the full offsite backups, as it complains that there is an increment missing (and I can't see anywhere in the logs that there should be an increment, the logs just show that the jobs failed on that day).

I'm now having to start my BCJ's all over again from scratch.

[Dave-Departed]

Ok, so all of our VBR and Proxy/WAN Accelerator servers have been patched with all of the above mentioned WMI hotfixes from Microsoft, but the issue persists (it now appears to move between VBR/Proxy servers, even though they have all been patched). The only thing these servers run is Veeam, so nothing else should be getting in the way.

I will reopen this job and submit logs for all servers.

But I am sorely tempted to kill my main VBR server and start over again. We didn't have this issue on previous versions, and it's only an issue with Backup Copy Jobs (which is a shame, because that's the whole reason we upgraded to v7 in the first place).

[foggy]

Please kindly keep us posted on how investigation is going. Thanks.

[Dave-Departed]

Hi All,

As with previous support cases we've opened on the issue:
[ID# 00466465] Backup Copy jobs failing
[ID# 00555092] RPC server unavailable - Function name [InvokerTestConnection]
[ID# 00450161] RPC server unavailable - Function name [InvokerTestConnection]
[ID# 00570956] Backup Copy jobs fail - RPC unavailable error - WMI related

Our issue with Backup Copy Jobs has returned, out of nowhere! Backup Copy Jobs began failing last night, for no apparent reason, with the old RPC server unavailable message. Support job is raised [ID# 00606749]. I see on the forums that others do have this issue - Is it recognised by Veeam? The only workaround I've found thus far is to run the "winmgmt.exe /resetrepository" command on whichever VBR server is being affected at the time (I've seen it happen on the local VBR server that hosts our onsite backups, the remote VBR server which is the target for our Backup Copy Jobs, and servers that are purely VBR proxies/WAN accelerators). Would you advise building a new VBR server from scratch and importing our backups? We can't continue like this, with it happening randomly.

For reference, here is my previous thread on the subject:

http://forums.veeam.com/veeam-backup-re ... 21854.html

Many thanks in advance.

Dave

[veremin]

As far as I'm aware, the said issues are related typically to network problems and no specific for backup copy jobs, since InvokerTestConnection is a generic method used by all types of job (replica, backup, backup copy, etc.). Thanks.

[bernardw]

I have been running into this issue for the past week or so as well. From one of the other threads on a similar topic I just tried rebuilding/updating the Proxy, Repository and WAN Accelerator (just going into Properties and walking through the steps, not changing anything). That seems to have fixed the issue for me (as my copy job is now running again).

I am running Win 2012 R2 on both the main server and the proxy server server is running plane 2012
Running Veeam 7.0.0.839

[Dave-Departed]

Hi All,

Resurrecting this thread as I finally had enough of the RPC errors, spent a day rebuilding our VBR server, and getting everything set up again, and... Guess what? RPC errors on Backup Copy Jobs AGAIN.

What I noticed is that it seems to be specifically related to WAN accelerators... Most of our BCJ's failed yesterday, yet as soon as I changed them all from using WAN accelerators to direct connection, they all worked instantly. Also, I noticed that the ones that didn't fail were the ones using WAN accelerators located on VBR Proxies, and the ones that DID fail were using the WAN accelerator on the main VBR server.

As it happens, it appears our offsite bandwidth is currently capable of supporting direct transfers, so it's not a huge issue for now, but I'll most likely reopen the support case and pick it up again - It would be good to get to the bottom of what's causing this!

NB - I had been told before that RPC errors are usually because of network connectivity in Windows, which would perhaps suggest that there is an issue with our SAN network design, yet regular Backup jobs live on the same repository which the BCJ's pull from, and they have never suffered the RPC error, so I'm unsure how that could be the case?

[Dave-Departed]

A bit more detail relating to my last post above:

The WAN accelerator that lives on the main VBR server has its cache located on the same iSCSI volume that contains our main onsite VBR repository. It appears to mainly be this WAN accelerator that is associated with BCJ's failing.

The WAN accelerator that lives on our onsite VBR proxy server has its cache located on its own local disk. More often than not, BCJ's associated with this WAN accelerator do not get the RPC error.

Both the VBR server and VBR proxy are VMs that live on our SAN, which is on the same network and has the same connection capacity as the iSCSI volume that contains our main onsite VBR repository, yet there are no bandwidth/performance issues there.

As previously stated, the main VBR server uses the repository on the iSCSI volume as the location for all of our main onsite backup jobs, and none of these have ever failed with an RPC error (yet they have a much higher throughput than any of our BCJ's).

Anyone have any thoughts please?

[VladV]

I am seeing this error since v7 in a simple backup copy setup.

- Main VBR server with backup proxy and wan accelerator services installed.
- Target backup respository with backup proxy and wan accelerator services installed.

Every once in a while I see:

Code: Select all

10/23/2014 12:46:07 AM :: Processing 'xxxxx' Error: The remote procedure call failed RPC function call failed. Function name: [InvokerTestConnection]. Target machine: [192.168.xx.xx].

but only once per job and usually between processing disks. For us it is not a big issue because it retries after a few seconds and continues processing the next disks.

The connection between the two servers is through a transparent VPN 100Mbit link. Both servers are 2012R2.

It's curios why only the RPC procedure fails and not transferring the actual disk (the disk is transferred successfully), so the connection between the two points is fine.

[Dave-Departed]

Hi VladV,

Yes, I see exactly that error, yet mine retries 5 times and then causes the job to fail. But as I said, changing the jobs to direct connection removes the error, therefore logically, it must be related to the WAN accelerator(s) somehow?

[VladV]

Yes, the error appears only on jobs with wan acceleration in our case also.

[chrisdearden]

Subscribing to this thread , I think I have seen a similar issue with an evaluating customer.

[Dave-Departed]

Subscribing to this thread , I think I have seen a similar issue with an evaluating customer.

Hi Chris,

Do you have any further info at all, please?

[chrisdearden]

nothing other than the errors above that I worked around at the time by going direct rather than via the Wan Accelerators.

[veremin]

Hi VladV,

Yes, I see exactly that error, yet mine retries 5 times and then causes the job to fail. But as I said, changing the jobs to direct connection removes the error, therefore logically, it must be related to the WAN accelerator(s) somehow?

I think both number of attempts and timeout between them are configurable via registry. May be by increasing these values you can achieve the situation when the connection can be re-established and ,thus, the job doesn't fail?

Thanks.

[Dave-Departed]

I think both number of attempts and timeout between them are configurable via registry. May be by increasing these values you can achieve the situation when the connection can be re-established and ,thus, the job doesn't fail?

Do you know where I can find those reg values, please?

But, saying that... It still puzzles me why this would only be triggered by a reboot of the main VBR server? As I mentioned before, the jobs would run consistently for weeks at a time without issue, and as soon as the server was rebooted for any reason, bam! BCJ failures due to RPC error. And as I said, the only workaround I could find was to reset the WMI repository using winmgmt /resetrepository, but even that stopped working recently.

I have a theory, and I am going to test it this week - I am going to build a brand new proxy/WAN accelerator using local storage and import the WAN cache into the new proxy from the main VBR server WAN cache, and see what happens.

[foggy]

Here's the registry key that controls the number of retries.

[rdelc]

I have had similar problem also. I did not contact Veeam support as after doing all the usual suggestions, it really boiled down to two problems: One my iSCSI target was having issues. I had to clean that up and fix it. On another occasion, I had a service that had not started properly.

So look at your source and target were you are sending your backups. Make sure you are having solid connections and there are no hiccups.

One thing that I do now on both ends is this:
Before I shut the Veeam server down:
1. I stop all the services and put the on manual.
2. After I reboot, I make sure that all the storage appears as it should.
3. I restart the Veeam Services.
4. I double check jobs.

thanks, hope that helps.

[Dave-Departed]

Thanks rdelc, much appreciated.

As stated previously, the target of our regular onsite backup jobs and the source WAN accelerator of some of our offsite backup copy jobs is one large iSCSI repository, and the regular backup jobs have never, ever had a connection issue, RPC-related or otherwise, so I can't see how it can be related to the iSCSI target?

But thanks for the other pointers, I will make sure I try following that process next time I want to reboot our main VBR server.