Veeam trying to authenticate against AD with self signed certificates.

[GeraldS]

After we started to deploy managed instances of the Veeam Agent we noticed an increased activity of failed logins in our Active Directory audit log originating on our Veeam Backup & Replication server.

The "user agents" are in the following form:

Code: Select all

x509n:<s>cn=8b41492a-f228-47a2-b1b5-25dec8af8768,6
x509n:<s>cn=9261cec5-f131-4797-a635-af4472f8e11a,6
x509n:<s>cn=a1a65e62-91a3-48fd-80f1-9d596c47f23d,6

When we searched the Veeam logs for these UUIDs we found out that they correspond to self signed certificates created by Veeam.

These failed logins don't impact the backups, they succeed. It's just that our AD team has asked us to investigate these failed logins.

Why is the Veeam Agent (or VBR) trying to use these self signed certificates to authenticate against Active Directory, and how can we stop it?

[HannesK]

Hello,
sounds strange... what is the support case number for that issue?

Additional to Veeam logs, please do also upload the AD audit logs that we can check that.

Thanks,
Hannes

[GeraldS]

I haven't opened a support case yet. I was hoping someone else might have run in the same problem.

[zd14a]

Hi, we see this kind of login attempts made by Veeam as well. We're also wondering what's happening there and how we can prevent Veeam from doing this.

[HannesK]

Hello,
can you please provide a case number with logs that we can have a look at it?

Thanks,
Hannes

[TDog]

These failed logins don't impact the backups, they succeed. It's just that our AD team has asked us to investigate these failed logins.

Why is the Veeam Agent (or VBR) trying to use these self signed certificates to authenticate against Active Directory, and how can we stop it?

We noticed the same very recently, ever find a solution? I found the offending cert in the local cert store on the agent machine, wonder if I have to deploy a local CA that is trusted by AD to resolve this. https://helpcenter.veeam.com/docs/backu ... ml?ver=110

[redgasgiant]

Howdy, I just noticed that my AD Audit flagged it as well:

A Kerberos authentication ticket (TGT) was requested for X509N:<S>CN=Veeam Backup Enterprise Manager Server Certificate from MyBackupServer.myDomain. Status : Failure. . Error : Bad user name

Did you get resolution?

[HannesK]

Hello,
did you open a support case with Veeam to check what's going on? If yes, can you please post the case number so we can investigate further?

Best regards,
Hannes

[Redhw]

Hi,

We also got the same problem recently and it still happens frequently. Did someone at veeam perhaps forgot to renew an internal certificate ?

[Mildur]

Hello DH

So far we didn't got any case number in this topic.
Please open a case and share the number with us. Then we can analyze this error messages.

Best,
Fabian

[Redhw]

Hi Mildur,

No problem, here you go:
Case #06225025

[gardhy]

I'm getting the same certificate errors as well. Been with support trying to resolve for the last couple of weeks. Case #06172180 if this helps.

[Mildur]

Thank you for the case numbers.
I checked similar cases and their may be a known issue with the certificate.

@Redhw
Let's see what our support team will find out and if it's related to the known issue.

@gardhy
Did you already have tried the last suggestion from our support on August 3rd?

Best,
Fabian

[JJohnson2023]

I'm currently experiencing the same issue, and just submitted Case # 06258240

[Wad4iPod]

Seeing the same issue.

[STGdb]

Sort of the same issue here, but it just started happening after upgrading from v11a to v12. Any update on how to resolve it? I currently have two other support cases open, hoping not to have a third.

[Mildur]

Hello SOSidb

Our customer support team is aware of a known issue regarding EM certificates.
They have a procedure to solve those issues. Please open a case and ask them to check if you are affected by the same issue as Jarred: 06258240.

Best,
Fabian

[Phyxiis]

I had a opened a case end of last week with our MSP/Veeam about this same exact issue (the screenshot from above with ADAudit logging actually) and they stated that because it's not application (Veeam) related, it's a Microsoft issue and to figure it out.... (basically).

We're still on v11a (or whatever the latest v11 is) and only just started noticing this within the past 2 weeks or so. No changes other than I got LDAPS set up on our domain controllers but is not being required or forced on Workstations/Servers (you can run LDAP and LDAPS in parallel with no issues).

We don't use Enterprise Manager, it was set up years ago before I got hired here but worked with Veeam to remove the Enterprise Manager tie-in in the database so that we could auto-renew our license via our MSP. So I don't know that the most recent post by Mildur would be applicable to us, unless some remnant of the EM server is still existent somewhere and expired. No backup jobs fail, and all authentication that I am aware of on the actual VBR server (domain joined VM) work just fine.

My guess is either A) something may be tied up cert-based with the non-existent EM server from years gone by, or B) something else is going on in which I can't figure out with hours of Googling...

EDIT: Our case number for what it's worth 06305132

[cerberus]

Seeing the exact same issue on v12, our auditing tool is flooded with "X509N:<S>CN=Veeam Backup Enterprise Manager Server Certificate" failure events.

Case opened referencing this thread requesting the procedure to resolve this issue as per Fabian, case #06323302.

[Mildur]

Hi Mirza

Thank you.
I can see that our support engineer has provided the steps to solve the issue.
Please execute them and report back if they have solved this issue.

Best,
Fabian

@Phyxiis
If you don't use enterprise manager, then your case is different to other comments in this topic.
The cases I checked in this topic affects environments where Enterprise Manager is installed on the backup server.
We have a known issue regarding the automatically installed certificate. To solve it, we need to create a new self signed certificate, change a config file and delete a value from the enterprise manager database.

[ARHalderberge]

Hi, I'm experiencing same issue as in case #06323302
Enterprise Manager is installed on same server.
We use version 12 (recently installed)
Can you provide the steps to solve the issue?

[MajorWitt]

Hi there @Mildur,

We are seeing the same problems here, we have Enterprise Manager installed on the same server as Veeam Backup & Replication, we are running v12.0.0.1420.

Where can I find more information about how to "create a new self signed certificate, change a config file and delete a value from the enterprise manager database"?

[Mildur]

@MajorWitt
The process is not documented publicly.
Can you please update to our newest release (v12.1) and check the issue again? I can see on our internal system that there were changes around the enterprise manager certificate which may solve this issue. If you cannot update now, please open a support case and get the procedure from our support engineer.

Best,
Fabian

[k21971]

The process is not documented publicly.
Can you please update to our newest release (v12.1) and check the issue again? I can see on our internal system that there were changes around the enterprise manager certificate which may solve this issue. If you cannot update now, please open a support case and get the procedure from our support engineer.

Can we make the fix public? Or if it is, point us to a link? I just upgraded from 11a to 12.1 a few days ago, and I'm now seeing the exact same issue. Thanks.

[lbrown13]

Any updates on this? We have the same issue.

[Mildur]

For now, please contact our support team for the existing workaround.
Meanwhile I started a discussion with support management if it would be possible to provide a KB.

Best,
Fabian

[jcwrks]

FYI

support@veeam.com

At this point if EM is not installed on the same server , there is no current workaround , however Veeam is aware of this issue and will release a future patch to get it fix , what we adviced under these scenarios is to wait for the patch and ignore the alerts for now since they do not affect the backups.

[Mildur]

Hello JW

Thanks for the update.
In the meantime I got an update from support management as well.
We will not provide a KB. Why not? There are two scenarios where we see certificate errors. Our support team must first confirm the scenario a customer is affected by before we can provide a solution.
One issue requires running queries in the configuration database, the other issue requires a private hotfix which can be obtained by a support case for version 12.1.1.56. We plan to include the hotfix in one of our next public patches.

Best,
Fabian

[ShawnKPERS]

I also wanted to confirm that if you are seeing failed login attempts in AD with an account name of a cert that your Veeam backup agent generated and stored in Microsoft's certificate store. Then that is a known issue Veeam Support does offer a hot fix for that you need to get directly from them. If it helps speed up the troubleshooting process with support then reference my case #07202791.

In my case I was managing the backup agents through Veeam B&R and all the failed login attempts would originate from only that server when I checked the AD logs. I would see multiple of the same failed attempts in a single day. The only thing that changed between events was the certificate ID used to login would change depending on what physical server was in that specific backup job running at that time. The physical backup jobs would run with no issues but would just generate the failed login alerts which caused issues for our security team.

Here is one of the certs that were showing up in the logs that was found on one of my physical servers that were backed up by the Veeam Agent:


Here is an example of the AD failed login event that I would get related to that cert:


The login failure event would happen exactly one second before the backup job would run. So I found the Veeam related event that ran at that exact same second in the "Job.VeeamEndpointBackup.log" file:


I believe the highlighted line above was the exact event that generated the failed AD login event.

Support originally thought it might be related "Veeam Backup Enterprise Manager" but I did not have that solution installed.

Hope this is of some help to others out there in the same situation.

[Phyxiis]

I was never notified of this response.

@Mildur: This environment had Enterprise Manager at one point in the past (previous IT people) but subsequently removed it from the environment. I had to open a ticket in the past because our licensing wouldn't renew and had to work with Support to run a database script to delete the enterprise manager from the database.

So perhaps the Veeam system is still trying to authenticate using a Enterprise Manager cert that the EM no longer exists?

I will open a ticket with our MSP (provider of veeam license) to open a ticket with Veeam as it seems related to EM as you state.

Hi Mirza

Thank you.
I can see that our support engineer has provided the steps to solve the issue.
Please execute them and report back if they have solved this issue.

Best,
Fabian

@Phyxiis
If you don't use enterprise manager, then your case is different to other comments in this topic.
The cases I checked in this topic affects environments where Enterprise Manager is installed on the backup server.
We have a known issue regarding the automatically installed certificate. To solve it, we need to create a new self signed certificate, change a config file and delete a value from the enterprise manager database.