RPC client failed to call method. / Win32 error: Network path not found

[andrefaria]

Hello,

I just want to clarify some points, the problem is:
After update Veeam 10 to 11.0.1.1261 P20211211, one VM started to take 6 hours to finish the backup, before was just 10 minutes.

I found that while testing Guest Credentials it fail to connect via RPC.

Code: Select all

27/12/2021 17:44:11 :: Connecting to guest OS via RPC
27/12/2021 17:44:11 :: Testing admin$ share accessibility via RPC
27/12/2021 17:45:17 :: Cannot connect to the admin share. Host:  [ts01.domain.local]. Account: [administrador@domain.local].;Win32 error:Network path not found.; Code: 53;
27/12/2021 17:45:17 :: Cannot connect to the admin share. Host:  [10.1.2.141]. Account: [administrador@domain.local].;Win32 error:Network path not found.; Code: 53;
27/12/2021 17:45:17 :: Connecting to guest OS via Persistent Guest Agent Error: A connection attempt failed because the connected component did not respond
correctly after a period of time or the established connection has failed
because the connected host did not respond
27/12/2021 17:45:17 :: Testing host accessibility via Persistent Guest Agent
27/12/2021 18:00:20 :: Connecting to guest OS via VIX
27/12/2021 18:00:20 :: Testing admin$ share accessibility via VIX
27/12/2021 18:00:57 :: Testing guest OS connectivity via VIX
27/12/2021 18:01:03 :: Credentials and connectivity have been validated successfully for VIX transport

I was analyzing the log before update, and it show that was failing to connect via RPC, but right after failing, it was switching to Web Service (VIX?) and the backup ended in 10 minutes.

Code: Select all

[05.12.2021 22:00:32]   6688  Info             Guest agent control version info
[05.12.2021 22:00:32]   6688  Info             {
[05.12.2021 22:00:32]   6688  Info             Module name: [VeeamGuestHelperCtrl.exe].
[05.12.2021 22:00:32]   6688  Info             Module path: [C:\Program Files (x86)\Veeam\Backup Transport\GuestInteraction\VSS\VeeamGuestHelperCtrl.exe].
[05.12.2021 22:00:32]   6688  Info             Module version: [10.0.0.4461].
[05.12.2021 22:00:32]   6688  Info             }
[05.12.2021 22:00:32] <29> Info     [VssGAConn(TS01)] Connecting to guest via rpc, host 'ts01.domain.local', user 'administrador@domain.local'
[05.12.2021 22:00:32]   6688  Info             Connecting to a remote host. Host: [ts01.domain.local]. Specified account: [administrador@domain.local].
[05.12.2021 22:00:32]   6688  Info             Creating network logon session. User: [administrador]. Domain: [domain.local].
[05.12.2021 22:00:32]   6688  Info             Creating network logon session. User: [administrador]. Domain: [domain.local].. Ok.
[05.12.2021 22:00:32]   6688  Info             Creating network connection [\\ts01.domain.local\ADMIN$]
[05.12.2021 22:00:32]   6688  Info             Impersonating network logon session.
[05.12.2021 22:00:32]   6688  Info             Impersonating network logon session.. Ok.
[05.12.2021 22:00:54]   6688  Info             Connecting to a remote host. Host: [ts01.domain.local]. Specified account: [administrador@domain.local].Failed.
[05.12.2021 22:00:54]   6688  Error             Cannot connect to the host's administrative share. Host:  [ts01.domain.local]. Account: [administrador@domain.local].
[05.12.2021 22:00:54]   6688  Error             Win32 error:O caminho da rede n\E3o foi encontrado.
[05.12.2021 22:00:54]   6688  Error              Code: 53
[05.12.2021 22:01:38] <29> Error    ' (Veeam.Backup.VssProvider.VssRpcException)
[05.12.2021 22:01:38] <29> Error       em Veeam.Backup.VssProvider.CGuestRpcConnection.ConnectInternal(IVeeamGAConnection connection, String vmDisplayName, String vmHostName, String[] ipAddresses, CCredentials vmCreds, Boolean isForceConnectByIp, Boolean isOnPort, Int32 port)
[05.12.2021 22:01:38] <29> Error       em Veeam.Backup.VssProvider.CVssLocalConnection.CreateEx(CVmwareSingleConnectionInfo info, String vmDisplayName, String vmHostName, String[] vmIpAddrs, EGuestOsType osType, Boolean waitUninstallAfterRestore, Boolean useVix, Exception& networkExc)
[05.12.2021 22:01:38] <29> Info     [WinGuestConnectionProxy] Connecting to guest service via web service, hostIp '10.1.2.247', postPort '443', hostUser 'administrator@vsphere.local', vmRef 'vm-31', vmUser 'administrador@domain.local'
[05.12.2021 22:01:38]   6688  Info             Connecting to host [10.1.2.247] over web services. Login: [administrator@vsphere.local].
[05.12.2021 22:01:38]   6688  Info             Connecting to host [https://10.1.2.247:443/sdk]
[05.12.2021 22:01:38]   6688  Info             Retrieving service content for host [https://10.1.2.247:443/sdk]

Back to the present, now when it fails, it start again, and again, and again, for six hours.

Code: Select all

[21.12.2021 22:02:06] <31> Info         [VssGAConn(TS01)] Initializing VSS guest agent connection.
[21.12.2021 22:02:06]  12136  Info             ==================================================================================
[21.12.2021 22:02:06]  12136  Info             Guest agent control version info
[21.12.2021 22:02:06]  12136  Info             {
[21.12.2021 22:02:06]  12136  Info             Module name: [VeeamGuestHelperCtrl.exe].
[21.12.2021 22:02:06]  12136  Info             Module path: [C:\Program Files (x86)\Veeam\Backup Transport\GuestInteraction\VSS\VeeamGuestHelperCtrl.exe].
[21.12.2021 22:02:06]  12136  Info             Build edition: [standard].
[21.12.2021 22:02:06]  12136  Info             Module version: [11.0.1.1261].
[21.12.2021 22:02:06]  12136  Info             }
[21.12.2021 22:02:06] <31> Info         [VssGAConn(TS01)] Connecting to guest via rpc, host 'ts01.domain.local', user 'administrador@domain.local'
[21.12.2021 22:02:06]  12136  Info             Connecting to a remote host via the guest agent. Host: [ts01.domain.local]. Specified account: [administrador@domain.local].
[21.12.2021 22:02:06]  12136  Info             Impersonating network logon session.
[21.12.2021 22:02:06]  12136  Info             Impersonating network logon session.. Ok.
[21.12.2021 22:17:06]  12136  Info             Retrying RPC call. Attempt 0
[21.12.2021 22:17:06]  12136  Error             RPC client failed to call method. Sleep interval: 0 MSec
[21.12.2021 22:17:06]  12136  Error             RPC error:A chamada de procedimento remoto falhou e n\E3o foi executada.
[21.12.2021 22:17:06]  12136  Error              Code: 1727

- The point is, what changed? why is taking to long to it fall back to VIX.
- I can change this behavior? Like change timeout or invert RPC / VIX order?

I'm working to fix the RPC problem, but for sake of curiosity.

[Gostev]

Hello! You should investigate this with support as this is not normal. In theory, the guest communication mode selection logic was expanded in V11 due to the introduction of optional persistent guest agent, so some behavior changes are not completely unexpected. But yes, there should be a registry value to change the order. It was called InverseVssProtocolOrder in previous versions but I believe it may have also changed in V11 due to the addition of the 3rd mode. Thanks!

[Anton.Kisel]

Hello!
Have you already escalated this case to our technical support? If no, please do so, and I would like to ask you to provide logs from the VM, Guest Interaction Proxy and the backup server, and also the firewall settings on the VM for ports mentioned in this article. Thanks!