Allow for Mailsystems with chaning SSL certificates for Notifications like Azure HVE/ACS

[fwipe]

Hey,

a support request regarding the topic sent me to post the issue here for further discussion.

Mailnotifications in Veeam B&R are unfortunately very unreliable when using Microsoft infrastructure due to the way Veeam implements stuff. You can either configure OAuth and reauthorize the connection every few weeks when Notifications just stop working or use Systems like High Volume Email or Azure Communication Service with SMTP basic auth as replacement for the disabled normal SMPT basic auth to mailboxes.

We already gave up on all hopes in fixing the OAuth implementation but at least the SMTP basic auth should work with mailclusters.
The problem with mailclusters like HVE or ACS is, as the frontend server loadbalances that the SSL certificate for the given host smtp-lob.office365.com can change regulary every few days thus invalidating the saved SSL cert hash in Veeam and therefor stopping any mailnotifications from sending.


As stated in our support ticket #07668026 here the detailed explanation: [Moderator: Added Case ID to replace Support ID]
Root cause of the problem is regarding the Microsoft policy to no longer allow normal SMTP auth for Exchange Online mailboxes. You already have the ability to grant Mail.Send through OAuth Workflow although you have to reauthorize this connection every few weeks.
The provided alternative for direct SMTP auth from Microsoft would be either Azure Communication Services or High Volume Email (HVE). HVE would be the go-to alternative for sending email notifications.
Unfortunately Microsoft uses its trafficmanager as frontend for smtp-lob.office365.com - the FQDN you have to use for HVE.
As Trafficmanager is some sort of loadbalancer with many unconfigurable frontend servers under *.trafficmanager.net and that even further CNAMEs to other SMTP LOB FQDNs under *.*.*.office.net.

Now the Problem: The SSL Certificate provided by trafficmanager is the FQDN of the node your connection gets resolved during DNS lookup (CNAME/DNS Round Robin) - so the SSL certificate is never for smtp-lob.office365.com.
That alone is not as much of a problem because you can save an exception when you configure the SMTP server for notifications in Veeam B&R. But as in the nature of loadbalancers you get randomly other trafficmanager frontend servers some time later and now Veeam reports on invalid SSL certificates and is unable to send the notification.

For using features like ACS or HVE from Microsoft we would need to disable this part of the SSL certificate validation process for the SMTP connections. It would be fine by checking if the certificate is within its validity range or if its signed by a trusted certificate authority but as the CNAME roundrobins behind smtp-lob.office365.com we need to allow automatically accept changed ssl certificates to happen.
The error provided by Veeam B&R is "Sending e-mail report Details: The remote certificate is invalid according to the validation procedure".

For now we are using a custom made email relay which relays then over HVE - but we would appreciate the possibility of some sort of "ignoring ssl validation errors" on smtp ssl connections.

This might affect other software of you that validates SSL certificates the same way on SMTP connections.



Here is an example log of a backup job after just two days of successful notification mails:
[25.03.2025 03:50:39.815] <17> Info (3) [CReportMailer] Sending e-mail notification, server 'SMTP Server 'smtp-lob.office365.com', port '587', timeout '100000', use ssl 'True'', to 'mail@domain.tld'
[25.03.2025 03:50:40.691] <17> Error (3) Certificate not valid and thumbprint not saved for this certificate.
[25.03.2025 03:50:40.691] <17> Info (4) Certificate trusted.
[25.03.2025 03:50:40.696] <17> Error (3) [CReportMailer] Failed to send email report to 'mail@domain.tld'
[25.03.2025 03:50:40.696] <17> Error (3) The remote certificate is invalid according to the validation procedure. (System.Security.Authentication.AuthenticationException)


Unchecking SSL is not possible as the systems require SSL to connect.

Please give us the option to ignore SSL validation errors for notification mails.

[david.domask]

Hi Florian, welcome to the forums.

Thank you for the detailed explanation -- I believe I recall this behavior as well, but I need some time to review what the options are for handling the SMTP server SSL cert changing periodically. I will update the thread once I have more information.