-
- Lurker
- Posts: 2
- Liked: never
- Joined: May 18, 2023 1:00 am
- Contact:
Office 365 / Exchange Online's frequent SMTP certificate changes are causing Veeam job emails to fail due to mismatch
Case ID 06044505
This issue started the day we upgraded from Veeam 11 to Veeam 12.
We are using Exchange Online as our SMTP server (ourdomain.mail.protection.outlook.com, port 25, SSL enabled) for all of our local SMTP relay needs, including Veeam. The connection is using TLS encryption and Veeam is retaining the certificate thumbprint in its database. If the certificate changes, so does the thumbprint, and Veeam B&R will reject the certificate, not send an email for the job, log an error, and the job enters a Warning state (which also triggers Veeam ONE alarms).
The issue is that Exchange Online is frequently changing its certificate. Over the course of the last month working with support, we've seen 4 different certificates that Exchange Online is presenting. Right now, the only option to get Veeam to trust these additional certificates is for us to first use OpenSSL to establish an SMTP connection, pull the certificate thumbprints currently in use by Exchange Online, then manually modify the Veeam database to add the new thumbprints so Veeam trusts them.
Did something change the way Veeam B&R 12 is checking the certificates?
Ideally, it'd be nice to have some kind of option (hidden in the registry or directly in the settings) that would allow us to disable the thumbprint verification, and simply continue to check the other merits of the certificate (such as trust chain, validity, CRL, etc.). Otherwise, we're stuck waiting for the next job/email failures, then manually pulling the latest certificates and updating the Veeam B&R database, which can be prone to user error.
This issue started the day we upgraded from Veeam 11 to Veeam 12.
We are using Exchange Online as our SMTP server (ourdomain.mail.protection.outlook.com, port 25, SSL enabled) for all of our local SMTP relay needs, including Veeam. The connection is using TLS encryption and Veeam is retaining the certificate thumbprint in its database. If the certificate changes, so does the thumbprint, and Veeam B&R will reject the certificate, not send an email for the job, log an error, and the job enters a Warning state (which also triggers Veeam ONE alarms).
The issue is that Exchange Online is frequently changing its certificate. Over the course of the last month working with support, we've seen 4 different certificates that Exchange Online is presenting. Right now, the only option to get Veeam to trust these additional certificates is for us to first use OpenSSL to establish an SMTP connection, pull the certificate thumbprints currently in use by Exchange Online, then manually modify the Veeam database to add the new thumbprints so Veeam trusts them.
Did something change the way Veeam B&R 12 is checking the certificates?
Ideally, it'd be nice to have some kind of option (hidden in the registry or directly in the settings) that would allow us to disable the thumbprint verification, and simply continue to check the other merits of the certificate (such as trust chain, validity, CRL, etc.). Otherwise, we're stuck waiting for the next job/email failures, then manually pulling the latest certificates and updating the Veeam B&R database, which can be prone to user error.
-
- Product Manager
- Posts: 10086
- Liked: 2688 times
- Joined: May 13, 2017 4:51 pm
- Full Name: Fabian K.
- Location: Switzerland
- Contact:
Re: Office 365 / Exchange Online's frequent SMTP certificate changes are causing Veeam job emails to fail due to mismatc
Hello HH_TRJ
Thank you for the case number.
I checked our internal system and we plan to solve this bug in the upcoming patch. There were some small changes around the Notification module. Especially to integrate the new OAuth 2.0 mail notification feature.
Unfortunately we don't have a registry key to disable SMTP SSL Checks. One workaround for now that I see is to use Microsoft 365 OAuth 2.0 to send the notifications. Or deploy a small postfix server which will take the mails from Veeam and forward it to "ourdomain.mail.protection.outlook.com" till we have fixed the issue. That would also suit your internal policies to relay everything about Microsoft Exchange Online Protection.
Best,
Fabian
Thank you for the case number.
I checked our internal system and we plan to solve this bug in the upcoming patch. There were some small changes around the Notification module. Especially to integrate the new OAuth 2.0 mail notification feature.
Unfortunately we don't have a registry key to disable SMTP SSL Checks. One workaround for now that I see is to use Microsoft 365 OAuth 2.0 to send the notifications. Or deploy a small postfix server which will take the mails from Veeam and forward it to "ourdomain.mail.protection.outlook.com" till we have fixed the issue. That would also suit your internal policies to relay everything about Microsoft Exchange Online Protection.
Best,
Fabian
Product Management Analyst @ Veeam Software
-
- Lurker
- Posts: 2
- Liked: never
- Joined: Jul 24, 2023 12:55 pm
- Full Name: Lukács István
- Contact:
Re: Office 365 / Exchange Online's frequent SMTP certificate changes are causing Veeam job emails to fail due to mismatc
Hello,
A little workaround for your problem, my system same problem. The main reason of this error the Microsoft's O365 certificate. It changed (renewed) at 2023 may 31.
The Veeam B&R cant handle this (I dont know why), so the solution is simple: You have to change to dummy (unsecured) e-mail setting (i restarted the Veeam B&R), then configure it again to good settins. Then the program checking a certificate about 2 minutes (I don't know why), then configured correctly - and now working as well (3 days ago).
I hop this help for you.
BR: István
A little workaround for your problem, my system same problem. The main reason of this error the Microsoft's O365 certificate. It changed (renewed) at 2023 may 31.
The Veeam B&R cant handle this (I dont know why), so the solution is simple: You have to change to dummy (unsecured) e-mail setting (i restarted the Veeam B&R), then configure it again to good settins. Then the program checking a certificate about 2 minutes (I don't know why), then configured correctly - and now working as well (3 days ago).
I hop this help for you.
BR: István
-
- Product Manager
- Posts: 10086
- Liked: 2688 times
- Joined: May 13, 2017 4:51 pm
- Full Name: Fabian K.
- Location: Switzerland
- Contact:
Re: Office 365 / Exchange Online's frequent SMTP certificate changes are causing Veeam job emails to fail due to mismatc
Hi hh_trj and Lukács
This bug is listed as solved with our latest patch for Veeam Backup & Replication (P20230718):
https://www.veeam.com/kb4420
Best,
Fabian
This bug is listed as solved with our latest patch for Veeam Backup & Replication (P20230718):
https://www.veeam.com/kb4420
Best,
Fabian
Product Management Analyst @ Veeam Software
-
- Lurker
- Posts: 2
- Liked: never
- Joined: Jul 24, 2023 12:55 pm
- Full Name: Lukács István
- Contact:
-
- Lurker
- Posts: 2
- Liked: never
- Joined: Aug 15, 2023 10:31 am
- Full Name: Alexander W
- Contact:
Re: Office 365 / Exchange Online's frequent SMTP certificate changes are causing Veeam job emails to fail due to mismatc
Hi Fabian,
I was reading the release notes of P20230718 but couldn't find anything regarding SMTP certificates or Exchange Online.
Could you please let me know if it is indeed fixed or not?
BR
Alex
I was reading the release notes of P20230718 but couldn't find anything regarding SMTP certificates or Exchange Online.
Could you please let me know if it is indeed fixed or not?
BR
Alex
-
- Lurker
- Posts: 2
- Liked: never
- Joined: Aug 15, 2023 10:31 am
- Full Name: Alexander W
- Contact:
Re: Office 365 / Exchange Online's frequent SMTP certificate changes are causing Veeam job emails to fail due to mismatc
Update: Unfortunately NOT fixed, I had to install the patch on more than 15 servers, but it didn't fix the issue.
-
- Product Manager
- Posts: 10086
- Liked: 2688 times
- Joined: May 13, 2017 4:51 pm
- Full Name: Fabian K.
- Location: Switzerland
- Contact:
Re: Office 365 / Exchange Online's frequent SMTP certificate changes are causing Veeam job emails to fail due to mismatc
Hello guys
I have news. It seems the entire fix for this issue should make it our upcoming version.
The Beta of the new version is already running in my lab. I will run a test over the next days to check if the issue is solved or not.
Best,
Fabian
I have news. It seems the entire fix for this issue should make it our upcoming version.
The Beta of the new version is already running in my lab. I will run a test over the next days to check if the issue is solved or not.
Best,
Fabian
Product Management Analyst @ Veeam Software
-
- Influencer
- Posts: 24
- Liked: 2 times
- Joined: May 06, 2021 1:45 pm
- Contact:
Re: Office 365 / Exchange Online's frequent SMTP certificate changes are causing Veeam job emails to fail due to mismatc
Same issue here.
There is no new patch since P20230718, but it has been superseeded by 12.1.
Only, also in those release notes I did not read anything about a fix for this issue.
Regards,
Eric
There is no new patch since P20230718, but it has been superseeded by 12.1.
Only, also in those release notes I did not read anything about a fix for this issue.
Regards,
Eric
-
- Product Manager
- Posts: 10086
- Liked: 2688 times
- Joined: May 13, 2017 4:51 pm
- Full Name: Fabian K.
- Location: Switzerland
- Contact:
Re: Office 365 / Exchange Online's frequent SMTP certificate changes are causing Veeam job emails to fail due to mismatc
Hi Eric
We don't mention all bugfixes in our release notes.
I checked our internal system and all bug ids related to this issue are marked as solved.
If you can, please update to v12.1.
Best,
Fabian
We don't mention all bugfixes in our release notes.
I checked our internal system and all bug ids related to this issue are marked as solved.
If you can, please update to v12.1.
Best,
Fabian
Product Management Analyst @ Veeam Software
-
- Influencer
- Posts: 20
- Liked: 1 time
- Joined: Aug 17, 2023 8:22 am
- Full Name: Parvez Khan
- Contact:
Re: Office 365 / Exchange Online's frequent SMTP certificate changes are causing Veeam job emails to fail due to mismatc
Dear Team,
Greetings,
We are still encountering the following error on the new version VBR-12.1.2.172:
"An untrusted certificate is installed on smtp.office365.com and secure communication cannot be guaranteed. Untrusted thumbprints. Connect to this server anyway?"
Kindly assist in resolving this issue.
Greetings,
We are still encountering the following error on the new version VBR-12.1.2.172:
"An untrusted certificate is installed on smtp.office365.com and secure communication cannot be guaranteed. Untrusted thumbprints. Connect to this server anyway?"
Kindly assist in resolving this issue.
-
- Product Manager
- Posts: 10086
- Liked: 2688 times
- Joined: May 13, 2017 4:51 pm
- Full Name: Fabian K.
- Location: Switzerland
- Contact:
Re: Office 365 / Exchange Online's frequent SMTP certificate changes are causing Veeam job emails to fail due to mismatc
Hi Parvez
Please contact our customer support team and share your case number if you want to have the issue investigated and resolved.
As a workaround, please try "Microsoft 365 (modern authentication)" instead of "SMTP server (basic authentication)".
Thank you
Fabian
Please contact our customer support team and share your case number if you want to have the issue investigated and resolved.
As a workaround, please try "Microsoft 365 (modern authentication)" instead of "SMTP server (basic authentication)".
Thank you
Fabian
Product Management Analyst @ Veeam Software
Who is online
Users browsing this forum: Google [Bot] and 104 guests