The RPC server is unavailable

[m.novelli]

Hi guys, would like to share a recent issue with VSS-aware processing

Environment: Veeam 7.0 running on Windows 2012 R2 and backing up a couple of vSphere 5.1 U1 hosts

All Windows 2008 and Windows 2008 R2 VM where saved successfully with VSS-aware processing, all Windows XP and Windows 2003 VM failed with error

Failed to inventory guest system: Veeam Guest Agent is not started
Failed to prepare guest for hot backup. Error: The RPC server is unavailable. (Exception from HRESULT: 0x800706BA)
Error: The RPC server is unavailable. (Exception from HRESULT: 0x800706BA)

and sometimes error

Failed to inventory guest system: Veeam Guest Agent is not started
Failed to prepare guest for hot backup. Error: Access is denied. (Exception from HRESULT: 0x80070005 (E_ACCESSDENIED))
Error: Access is denied. (Exception from HRESULT: 0x80070005 (E_ACCESSDENIED))

I was using the Administrator user of the AD domain to perform VSS-aware backup

I spent a full day trying to restrict RPC ports (http://www.veeam.com/kb1174), disable firewall, disable antivirus, check DNS name resolution, etc etc but no way. Network configuration (subnet, gateway) was the same on Windows 2008 / 2008 R2 VM and Windows XP / 2003 VM, but only the latter failed (maybe a screwed up patch from Microsoft on those systems?)

At the end I found this workaround to be success: http://www.veeam.com/KB1671

Added new registry value that reverses the sequence of application-aware processing, making jobs try network-less processing mode before network one.
HKEY_LOCAL_MACHINE\SOFTWARE\VeeaM\Veeam Backup and Replication
DWORD: InverseVssProtocolOrder
Value = 1
To disable (default behavior), value is 0 (false)

After changing this registry key all VM where saved successfully with VSS-aware processing

I'm wondering why Veeam didn't try automatically the VIX api method to perform VSS-aware processing when seeing the the network mode failed

Cheers

Marco

[foggy]

Marco, Veeam B&R definitely should automatically fail over from network to VIX mode, probably some environmental issues affected the ability to fail over, you could open a case to investigate this.

[Gostev]

Either environmental issue, or some changes in v7 affected the failover logic. We would really appreciate if you could send us the logs through the support case, so that we could investigate this and fix in the future update. Thanks!

[mrt]

I have been battling the same issue in the same environment. Thanks for posting a workaround! i look forward to returning to green jobs!

[m.novelli]

Either environmental issue, or some changes in v7 affected the failover logic. We would really appreciate if you could send us the logs through the support case, so that we could investigate this and fix in the future update. Thanks!

Hi Gostev, I've opened the case 00452781

Marco

[Slick25]

You're a genius! this workaround worked for me too! thanks...now how about a possible hotfix Veeam? I have the exact same thing - 2012 vbr server running v7 backing up 2003 servers off a esxi 5.1 host and every backup of a 2k3 guest would fail. Mario, how you ever decided to change the logic of the job is beyond me...but I'm glad you did, I can backup my 2k3 servers now!

[veremin]

According to the support case, this issue is still under investigation. So, it might be worth opening your own ticket with our support team and let them collect as much data regarding this problem as possible.

Thanks.

[Slick25]

Well it turns out I jumped the gun because although we could backup our 2k3 servers, the scheduled runs of our 2k8 servers failed...must've been this change. Anyways, I opened up case #00454845, I'll post back any findings.

[veremin]

Just out of curiosity – 2008 jobs failed with the exact error as 2003 ones did? Or you’re experiencing different issue? I’m asking since regardless of guest operating system, you should be able to backup VMs in networksless mode.

Anyway, thank you for providing the ticket number; much appreciated.

Thanks.

[jmarlhioud]

I had the exact issue on a similar setup, installed patch 1 for Veeam B&R 7 recreated the job and no longer getting the RPC error.

Client Setup

Hosts
ESXi 5.1 U1

VMs in backup job
Windows 2003 SP2 VMs
vCenter Linux Appliance VM

Veeam Server
Veeam (All-in-One) VM - Windows Server 2012 patched up

Maybe this will help.

Regards

[jmarlhioud]

Issue came back, forgot I had disable application-aware processing on that VM for the backup job

We found a fix with support, for reference case 00457700.

First Veeam 7 tries to exclude the vCenter Database from application-aware processing, it recreates a Database Exclusion entry (Hidden in the UI) with the name of VM (using the full FQDN) and the name of the vCenter Database.
The Full FQDN name of the VM is the problem so to fix the issues we need to change it to short name.

To enable Database Exclusion in Veeam B&R 7 UI

Open Regedit and add this DWORD key in

• [HKEY_LOCAL_MACHINE\SOFTWARE\Veeam\Veeam Backup and Replication]
  EnableDBExclusions = 1

Open Veeam Backup & Replication Console

• Go in the Top left open the context menu (same place were you manage Veeam options)
  You should see the Database Exclusions in the menu after the registry edit

Add or Edit Database (example)

• DNS name or IP address of the SQL Server :
  your-sql-server-name (not your-sql-server-name.domain.local)
  
  Database name:
  Your_VMware_vCenter

Restart / Retry your job

Hope this help

[foggy]

Janick, thanks for sharing. The requirement to have short name in the database exclusions while backing up vCenter's remote SQL Server with VSS also was mentioned in this thread.

[m.novelli]

I suppose it's an glitch between Veeam 7.0 and Windows 2012 R2 that will probably corrected in Patch 2

Actually I've installed 4 or 5 servers with this configuration (Veeam 7.0 + Windows 2012 R2) and I've faced 100% this issue. One time happened that also the Windows 2008 R2 VMs failed with RPC error, adding the reg key solved 100% the issue

Marco

[VladV]

Has a fix been confirmed to be included in R2 release of v7 or are we stuck using VIX when using server 2012R2 as a host for VBR?

[kte]

Case # 00476863
the register key doesn't fix all the issues , some server s doesn't understand VIX, or W2003 isn't working thatn or vise versa

Koen

[foggy]

For everyone experiencing problems with installing Veeam B&R components on Windows Server 2012 R2: it is recommended to upgrade to Veeam B&R v7 R2, which provides full Windows Server 2012 R2 support, and verify whether it addresses the said issues. Thanks.

[kte]

Any updates is R2 fixing everything, because I still have some servers with this isuues , can't find RPC, can't truncate logs , or veeamagent couldn't start
maybe I need to reboot some VM's or restrat the vss service to get ride of those errors

[foggy]

Koen, I recommend you just upgrade and see it for yourself. Remember that every environment is different.

[kte]

i upgraded , and the first test gave still this issue

[Gostev]

Then, this likely means the issue has nothing to deal with Windows Server 2012 R2 to start with, and something else is causing this.

[kte]

i ran all the backups correctly in w2012 for 2 months, so i only see the upgrade an the veeamagent that crached the vss

[Gostev]

This is certainly a possibility, it is just that the likelihood is low. Our support statistics shows that while vast majority of support cases created when something stops working after working fine are started of with customers linking those issues to product or platform updates, however investigation shows that at least 9 out of 10 cases are caused by some other environmental changes unnoticed by the end user.

Also, note that roughly half of posters in this thread who have this issue do not even have 2012 R2.

[gkennedy]

Hi,

We had exactly the same issues, and I can confirm R2 has resolved all of our issues.

Good luck!

Cheers,
Gavin

[kte]

ok all the backups are running again
k

[LarsWulf]

There is definitely a problem with Server 2012 R2 and network VSS to Server 2003.

We have 3 jobs with 5 different VMs using application aware processing. 4 of these 5 VMs are based on Server 2008 R2. And only on is using Server 2003 R2.
All these jobs have been running fine for month.

After I migrated the Veeam Backup Server from a Server 2008 R2 based system to Server 2012 R2 based system the Server 2003 R2 VM started failing with RPC unavailable error.
This was with Veeam v6.5 at the beginning. Yesterday I updated Veeam to v7 R2 but the error still occured.

So I started to search the forum and found this posts talking about the mode for VSS. So I gave it a try, set the InverseVssProtocolOrder RegKey and now my Server 2003 R2 VM is running fine again.

As this started to occur directly after the migration to Server 2012 R2 there definitely seems to be something different with VSS in network mode when using Server 2012 R2.

Regards
Lars

[foggy]

Lars, kindly please open a case with technical support as this supposed to be fixed in Veeam B&R v7 R2 update and we would like to investigate this closer. Thanks!

[LarsWulf]

Case is open: 481380

[RGlintmeijer]

I also had this problem after migrating the Veeam Backup server to Windows 2012 R2 with Veeam 7 Patch #1. The Windows 2003 Servers had issues with the RPC server.

After upgrading Veeam 7 to R2/Patch #2, the problem was still there. The cause in my case seems to be the VeeamVssSupport service, which was still running on the guests. Stopping and deleting the service seems to solve the problem.

sc stop VeeamVssSupport
sc delete VeeamVssSupport

[Gostev]

Yes, normally it should not exist on guests, as this is a run time process that is leveraged while preparing guest for backup. I can certainly see how this process can cause issues if it persists. Thanks for coming back to post the update.

[LarsWulf]

There is definitely a problem with Server 2012 R2 and network VSS to Server 2003.

We have 3 jobs with 5 different VMs using application aware processing. 4 of these 5 VMs are based on Server 2008 R2. And only on is using Server 2003 R2.
All these jobs have been running fine for month.

After I migrated the Veeam Backup Server from a Server 2008 R2 based system to Server 2012 R2 based system the Server 2003 R2 VM started failing with RPC unavailable error.
This was with Veeam v6.5 at the beginning. Yesterday I updated Veeam to v7 R2 but the error still occured.

So I started to search the forum and found this posts talking about the mode for VSS. So I gave it a try, set the InverseVssProtocolOrder RegKey and now my Server 2003 R2 VM is running fine again.

As this started to occur directly after the migration to Server 2012 R2 there definitely seems to be something different with VSS in network mode when using Server 2012 R2.

Regards
Lars

I just wanted to quickly let you know what I found out with the help of the support about my case.

As stated above I moved my Veeam v6.5 installation to Server 2012 R2. So I ran into the RPC bug. And as I could find out with the support I had the "VeeamVSSSupport" service still running because of the RPC bug. After updating Veeam to v7 R2 I still had the "VeeamVssSupport" Service running. The RPC bug itself is fixed in v7 R2 but in network mode v7 R2 is not capable of recognizing the running VeeamVssSupport Service. So my backup was still failing as the VeeamVssSupport service was running. When forcing VIX mode as first try the VeeamVssSupport service got stopped and the backup worked as expected with no more VeeamVssSupport Service running after the backup. After that I was again able to do backups in network mode again.

Things learned:
Veeam v7 R2 has fixed the RPC bug for connections from Server 2012 R2 to Server 2003. So my above statement

There is definitely a problem with Server 2012 R2 and network VSS to Server 2003.

is not correct
But if you are stuck to an still running VeeamVssSupport service you need to stop and delete the service as Robert did. Or you could switch once to VIX mode.

In short: Veeam v7 R2 has fixed the RPC bug but is not capable of fixing the sympthons (still running VeeamVssSupport service) of the RPC bug that occur in pre v7 R2 versions.

Regards
Lars