Down proxy causes job failure rather than failover to another working proxy

[jandrewartha]

As background, we have a VMware datacenter of 4 hosts, with 2 hosts each in separate physical datacentres. The VLANs are stretched but the storage is not. I have a 2 vCPU Veeam proxy VM on each host, and the jobs are configured to only use the proxy VMs in the appropriate datacentre, for both backup (hotadd) and guest interaction.

Recently (and it's not the first time it's happened), one of the backup proxies got stuck after the monthly Windows update. Veeam still tried to use it as a backup proxy, and failed because it couldn't talk to it, but then it didn't try the other one. Further, once it was up and running again, I manually disabled it as a proxy in the Veeam UI and it was still used as a guest interaction proxy (but not a backup proxy). This was quite annoying as I was off sick for two days just as the proxy failed, so then the backups failed and the SQL server logs filled up.

So my feature request is for Veeam to retry another backup proxy if the one it selects is unreachable.

I opened case 03591089 on this, it was closed as functioning as designed with the recommendation to take it to the forums as a feature request.

[foggy]

Chances are another proxy was not capable of hotadd due to one of these limitations and failover to NBD mode was not enabled in the job settings. If this is not the case, I recommend reopening the case for closer investigation.

[jandrewartha]

Actually the selected proxy (which was down) only had NBD:

[30.05.2019 10:53:59] <01> Info [ProxyDetector] Detected mode [nbd] for proxy [vm-pxy-pe2-esx1.ad.example.edu]
[30.05.2019 10:53:59] <01> Info [ProxyDetector] Detected mode [hotadd;nbd] for proxy [vm-pxy-pe2-esx2.ad.example.edu]

Then it choose pe2-esx1 for reasons that aren't entirely clear, and fails:

[30.05.2019 10:54:00] <01> Info Resolved by NTLM strategy ip addresses and host names: 10.86.0.18 vm-pxy-pe2-esx1.ad.example.edu
[30.05.2019 10:54:00] <21> Info [RPC] Loading options.
[30.05.2019 10:54:21] <21> Error [RPC] RPC client failed to call method
[30.05.2019 10:54:21] <21> Error [RPC] The RPC server is unavailable.
[30.05.2019 10:54:21] <21> Error [RPC] An exception was thrown from thread [7868].
[30.05.2019 10:54:21] <21> Info [RPC] Reconnecting...
[30.05.2019 10:54:21] <21> Info [RPC] Retrying RPC call. Attempt: 1
[30.05.2019 10:54:42] <01> Warning [vm-pxy-pe2-esx1.ad.example.edu] Failed to connect to Installer service on 10.86.0.18:6160. Reason: [vm-pxy-pe2-esx1.ad.example.edu] Failed to check whether remote Installer service is available. at Veeam.Backup.ProxyProvider.CProxyRawDeployerService.TestConnection(String srvName, CCliVeeamDeployer veeamDeployer, WindowsIdentity identityForImpersonationOrNull)

Then retries to the same proxy fail and the job fails.

[foggy]

Please continue working with the technical support on this.

[jandrewartha]

Tech support closed the case because it was apparently functioning as designed and to open a feature request in the forums.

[foggy]

The case (#03591089) doesn't contain a detailed explanation of the behavior, so I'm not sure if it was handled properly. The fact is that if at least one of the proxies capable of performing backup is available, the job shouldn't fail. Logs should clearly tell what proxies can and cannot be used and why. If you see the same behavior again and feel it is wrong, please open a new case and escalate it to get a proper understanding of the issue (or design peculiarities).