Guest interaction proxy generating failed login events for computer account on guest VMs

[kgabriel]

Hi all,

I have a case open for this issue but haven't resolved the issue so thought I would try posting here to see if anyone might have any additional input. Case # for reference: 04969957

Environment details:
We have a Veeam B&R v11 server running in Azure with a guest interaction proxy VM running on a Hyper-V cluster at our datacenter (dedicated Veeam proxy with no other roles or software installed). Both servers are running Server 2019 and are joined to the same internal AD domain. We have backup jobs configured for multiple client VMs that are running in our datacenter (mostly Hyper-V w/ some VMware) and each group of client VMs is using it's own separate AD domain. The local Veeam proxy server is used for all backups of the client guest VMs, and jobs have a domain service account specified for guest processing that is unique to each client domain (CLIENT1\veeambackup, CLIENT2\veeambackup, etc). All backup jobs are running and completing successfully without any issues.

The problem:
We recently deployed a security event monitoring solution for one of our clients, and after doing so we began receiving notifications for multiple failed login attempts on that client's guest VMs. These are logged as Event ID 4625 in the Windows security event logs and the event details show failed authentication attempts coming from the Veeam proxy IP address and using the proxy computer account (BACKUPDOMAIN\VEEAMPROXY1$). We do also see the successful logins using the service account that we have configured for the backup job (CLIENTDOMAIN\veeambackup), and the jobs complete successfully with no warnings or errors on the Veeam side. However, every time the job runs (currently every 2 hours) we also see the 4625 events for the proxy computer account login failures logged roughly 30-40 times or so on the guest VMs. I have checked some of our other client VMs and confirmed that they all show the same 4625 events for the VEEAMPROXY1$ login failures whenever their respective jobs run, so this issue is not specific to one client/AD domain and appears to be happening for every job that is using the Veeam proxy.

Current status:
Veeam support has stated that the Veeam proxy does not use its computer account for authentication to the guest VMs in any way, and that it will only use the configured credentials for guest processing (which are working normally). Because of this, they say that this is not a Veeam issue and is a problem with something else in our AD environment (GPOs, etc) that is causing the proxy to attempt to authenticate with the guest VMs using the computer account. This seems to be highly unlikely and we are not aware of any policies or configurations for our internal backup domain that would be causing this to happen, and since the failed authentication events only occur when the Veeam backup jobs run (via the proxy) it seems like it has to somehow be related to that process. As stated previously, the proxy is a dedicated VM that is not doing anything else other than local guest interaction for Veeam, so it does not ever communicate with guest VMs in any way other than when the backups run.

Has anyone out there seen this issue before or have any ideas what might be causing these failed login attempts from the Veeam proxy computer account? I will be happy to provide more info or details as needed, just let me know.

Thanks!

[HannesK]

EDIT: removed some unnecessary information that could cause confusion

Hello,
what I can say is, that I don't have these repeated logins.

As you say, that you have them with every job run, I have no idea. By using the "test credentials" button I could see that event.

Best regards,
Hannes

[kgabriel]

Hello Hannes,

I can use the "test credentials" button and the tests all complete successfully using the client domain account credentials that are specified for guest processing. However, running the test does also generate more of the proxy computer account authentication failures as well. So the same results happen whether it's a backup job running or just using the "test credentials" option from Veeam. In the security logs on the guest VMs we see the successful logins for our CLIENTDOMAIN\veeambackup service account as expected (event ID 4624), but then we also get the failed logins for the BACKUPDOMAIN\VEEAMPROXY1$ computer account as well (event ID 4625).

I have attached copies of the exact event log details to the support case already, but just to provide additional info for the forum post here is a sanitized version of the failure event:

Log Name: Security
Source: Microsoft-Windows-Security-Auditing
Date: 9/10/2021 4:49:13 PM
Event ID: 4625
Task Category: Logon
Level: Information
Keywords: Audit Failure
User: N/A
Computer: [Guest VM FQDN]
Description:
An account failed to log on.

Subject:
Security ID: NULL SID
Account Name: -
Account Domain: -
Logon ID: 0x0

Logon Type: 3

Account For Which Logon Failed:
Security ID: NULL SID
Account Name: VEEAMPROXY1$
Account Domain: BACKUPDOMAIN

Failure Information:
Failure Reason: Unknown user name or bad password.
Status: 0xC000006D
Sub Status: 0xC0000064

Process Information:
Caller Process ID: 0x0
Caller Process Name: -

Network Information:
Workstation Name: VEEAMPROXY1
Source Network Address: [VEEAMPROXY1 IP address]
Source Port: 52785

Detailed Authentication Information:
Logon Process: NtLmSsp
Authentication Package: NTLM
Transited Services: -
Package Name (NTLM only): -
Key Length: 0

This event is generated when a logon request fails. It is generated on the computer where access was attempted.

The Subject fields indicate the account on the local system which requested the logon. This is most commonly a service such as the Server service, or a local process such as Winlogon.exe or Services.exe.

The Logon Type field indicates the kind of logon that was requested. The most common types are 2 (interactive) and 3 (network).

The Process Information fields indicate which account and process on the system requested the logon.

The Network Information fields indicate where a remote logon request originated. Workstation name is not always available and may be left blank in some cases.

The authentication information fields provide detailed information about this specific logon request.
- Transited services indicate which intermediate services have participated in this logon request.
- Package name indicates which sub-protocol was used among the NTLM protocols.
- Key length indicates the length of the generated session key. This will be 0 if no session key was requested.

[HannesK]

Hello,
EDIT: I did some more tests and I can see it now for backups and "test credentials" for a machine that is in a workgroup (probably similar scenario likes yours with different domains). That might be expected behavior for the Windows API we use. I asked support to double check. But probably it's expected.

Best regards,
Hannes

[RobMiller86]

That's really unfortunate as it makes it impossible to monitor for failed logins, using basically any SIEM on any server protected by Veeam with a guest interaction proxy. This isn't an issue, unless you are using the guest interaction proxy. This should be configured to only attempt to auth using the specified creds. If we host client domains in our cloud, we have to back them up using Veeam, and Veeam will never be in the same domain. If using Veeam means we can't monitor for failed logins, that's a huge issue.

[RobMiller86]

Would this be resolved if the guest interaction proxy VM was in the same domain as the Hyper-V cluster? We could make that change. What we can't do is put it in the same domain as the guests. We have a separate dedicated AD domain for all Veeam services. They Hyper-V cluster is in another domain. The guests are then in all their own domains. Right now, the guest interaction proxy is in the Veeam domain. It could be moved to the cluster domain if that would eliminate this issue. What is Veeam's recommendation?

[kgabriel]

Hannes,

I received confirmation that our support case has been escalated to the next tier, so thanks for your help if you were involved with that. I am waiting for the new engineer to contact me after they have had a chance to review the case notes and will post another update here if we are able to (hopefully) find a resolution.

[isbmsab]

Hi all,

Wondering if there was a resolution to this issue? I am currently seeing similar logon failures on the VBU server itself every time a backup job is run, and trying to determine the cause of it.
A domain service account is used to handle the services and backup repository access, and there are no job failures, yet.

[kgabriel]

Hi isbmsab,

I don't know if this will help your particular situation but I just wanted to post an update on the results of this support case in case it helps you or anyone else:

Veeam support was ultimately able to determine that the Veeam Installer Service was causing the failed authentication attempts for the proxy computer account, so those failed login events were being logged on the guest VMs every time a backup job ran and the installer service had to install/uninstall the guest agent. Apparently this is hard coded and the only workaround would be to change the installer service to run using a dedicated service account on the proxy, and ensure that the same account is configured on the guest VMs as well so that the authentication attempts are successful. Unfortunately, this was not a feasible option in our scenario so we wound up just having to add an exception to our SIEM platform to ignore the failed login events coming from the proxy computer account.