Office 365 / Exchange Online's frequent SMTP certificate changes are causing Veeam job emails to fail due to mismatch

[hh_trj]

Case ID 06044505

This issue started the day we upgraded from Veeam 11 to Veeam 12.

We are using Exchange Online as our SMTP server (ourdomain.mail.protection.outlook.com, port 25, SSL enabled) for all of our local SMTP relay needs, including Veeam. The connection is using TLS encryption and Veeam is retaining the certificate thumbprint in its database. If the certificate changes, so does the thumbprint, and Veeam B&R will reject the certificate, not send an email for the job, log an error, and the job enters a Warning state (which also triggers Veeam ONE alarms).

The issue is that Exchange Online is frequently changing its certificate. Over the course of the last month working with support, we've seen 4 different certificates that Exchange Online is presenting. Right now, the only option to get Veeam to trust these additional certificates is for us to first use OpenSSL to establish an SMTP connection, pull the certificate thumbprints currently in use by Exchange Online, then manually modify the Veeam database to add the new thumbprints so Veeam trusts them.

Did something change the way Veeam B&R 12 is checking the certificates?

Ideally, it'd be nice to have some kind of option (hidden in the registry or directly in the settings) that would allow us to disable the thumbprint verification, and simply continue to check the other merits of the certificate (such as trust chain, validity, CRL, etc.). Otherwise, we're stuck waiting for the next job/email failures, then manually pulling the latest certificates and updating the Veeam B&R database, which can be prone to user error.

[Mildur]

Hello HH_TRJ

Thank you for the case number.
I checked our internal system and we plan to solve this bug in the upcoming patch. There were some small changes around the Notification module. Especially to integrate the new OAuth 2.0 mail notification feature.

Unfortunately we don't have a registry key to disable SMTP SSL Checks. One workaround for now that I see is to use Microsoft 365 OAuth 2.0 to send the notifications. Or deploy a small postfix server which will take the mails from Veeam and forward it to "ourdomain.mail.protection.outlook.com" till we have fixed the issue. That would also suit your internal policies to relay everything about Microsoft Exchange Online Protection.

Best,
Fabian

[Pistok]

Hello,

A little workaround for your problem, my system same problem. The main reason of this error the Microsoft's O365 certificate. It changed (renewed) at 2023 may 31.
The Veeam B&R cant handle this (I dont know why), so the solution is simple: You have to change to dummy (unsecured) e-mail setting (i restarted the Veeam B&R), then configure it again to good settins. Then the program checking a certificate about 2 minutes (I don't know why), then configured correctly - and now working as well (3 days ago).

I hop this help for you.

BR: István

[Mildur]

Hi hh_trj and Lukács

This bug is listed as solved with our latest patch for Veeam Backup & Replication (P20230718):
https://www.veeam.com/kb4420

Best,
Fabian

[Pistok]

Thanks Fabian.

[alex_01]

Hi Fabian,

I was reading the release notes of P20230718 but couldn't find anything regarding SMTP certificates or Exchange Online.
Could you please let me know if it is indeed fixed or not?

BR
Alex

[alex_01]

Update: Unfortunately NOT fixed, I had to install the patch on more than 15 servers, but it didn't fix the issue.

[Mildur]

Hello guys

I have news. It seems the entire fix for this issue should make it our upcoming version.
The Beta of the new version is already running in my lab. I will run a test over the next days to check if the issue is solved or not.

Best,
Fabian

[e.rottier]

Same issue here.
There is no new patch since P20230718, but it has been superseeded by 12.1.

Only, also in those release notes I did not read anything about a fix for this issue.

Regards,
Eric

[Mildur]

Hi Eric

We don't mention all bugfixes in our release notes.
I checked our internal system and all bug ids related to this issue are marked as solved.
If you can, please update to v12.1.

Best,
Fabian