Kerberos Only + standalone VBR + Veeam Agents upgrade issue

[emachabert]

Hi,

After hardening a domain by disabling NTLM we face an issue regarding Agent upgrade, it does not work anymore because of authentication failure.

The setup:
- Active Directory domain, Kerberos Only, Workstations with Veeam Agent
- Standalone VBR 12.2
- A service account for VBR to interact with the Active Directory domain and workstation
- Standalone VBR can reach the active directory domain and workstations on required ports, domain name resolution and time sync.
- Kerberos SMB sessions successfuly tested with admin shares to workstations (C$, ADMIN$ etc) from VBR server.

The issue:
- Rescan or upgrade of Agents now fail with : The network path was not found. The network path was not found. (ERROR_BAD_NETPATH). (System.Exception)
- Network capture shows:
- VBR tries RPC call and get the information to use kerb5
- VBR requests Kerberos Ticket for RPC Calls (TGS-REQ+TGS-REP)
- VBR successfuly autenticate with Agent over RPC (GSSAPI/SPNEGO KRB5)
- VBR tries new RPC calls on a new context but using NTLMSSP --> fails
- VBR then tries SMB connection, workstation reply with capability (must sign and kerb)
- VBR thus request new ticket from KDC
- VBR successfuly connect to the workstation using SMB for IPC$ tree and GSSAPI Kerb5 as Security blob, tries got get network interface info through winreg (but object not found)
- VBR disconnect the SMB session (clean disconnect)
- VBR then tries to establish a new SMB session but using NTLMSSP --> Fails
--> rescan job fails

Log from the rescan jobs show that line: [SNetworkAddressResolver] Host not joined to domain. Using NTLM only strategy.


It is strange because VBR is successfuly using kerberos to connect to DCERPC and IPC$ but then decides to go NTLM, looks like you have a logic which says if the VBR is workgroup, then use NTLM only.

Any idea ?

[Dima P.]

Hello Eric,

It looks like the issue with domain trusts as both the client and the server must belong to the same domain, or a trust relationship must exist between domains, otherwise Kerberos authentication wont work.

All Requirements and Limitations are here: Veeam Backup & Replication 12 > Kerberos Authentication. Thanks!

[emachabert]

OMG, that's not the answer I was expecting.

Why do you impose such a constraint ?

You could make it work without being in the same domain nor having a trust. As an example, the standalone windows is able to connect to a domaine computer using SMB3, Kerberos and a signed session as far as the standalone Windows know how to find the KDC (DNS), has a valid account (service account), and share the same time synchronization.

[Dima P.]

Hi Eric,

Thank you for sharing more details! We've already started to investigate this issue with RnD folks, I'll keep you posted!

[emachabert]

Some follow-up.

We did re-enter the server in the domain to unlock Agent updates.
Some of the agents did update and other kept failing, while the logs were saying everything looks ok (Kerberos, domain joined, DNS resolve etc).

So we did dig another time and made more network capture while looking what was doing the server.

We did see that when the server wanted to talk to the Deployment Service it was the moment it failed to find the SPN for the service on the workstation account and then it was trying to fall back to NTLM.
So we did manually add the SPN VeeamDeploySvc/FQDN on the computer account and it solves the issue.

So we started to understand why some machines were missing the VeeamDeploySvc SPN.

We saw that the SPN registration takes place at the start of the Veeam service and in our case the root issue is these computers are connected to the enterprise network throught VPN AFTER the service is started so the registration fails. If you restart the services while connected to the VPN, it adds the SPN

Dima, you could ask the RnD to have a periodic check of the presence of the SPN and add them in case they are missing. Thus, if your computers are connected to the domain after establishing the VPN connection, it could solve the issue very easily.

[Dima P.]

Hello Eric,

Thank you for sharing the updates! Sure we will discuss your idea with RnD folks!

[aabdelhakim]

Hello ,
is there any news on this , it would be great if Standalone VBR servers not fall back to NTLM strategy

[thebdur]

I am also interested in a solution for this. A non-domain computer is more than capable of connecting to domain resources using Kerberos. For environments where NTLM is blocked, running VEEAM B&R in a workgroup becomes a bit of a conundrum.