Configuration backup notification seems to use a different SMTP TLS validation check from other notification

[lucius_the]

I am using Veeam B&R version 11 (build 11.0.0.837) on several hosts.

I was having a problem on some hosts, where the Configuration backup notification could not be send - but all other notifications were being sent without a problem. The configuration backup job would just finish with a warning stating "Cannot sent e-mail report".

After checking the logs I found this two lines:

Code: Select all

[CReportMailer] Failed to send email report to <email-address>
The remote certificate is invalid according to the validation procedure. (System.Security.Authentication.AuthenticationException)

Now, in my case I am using my own mail server and that server has a certificate issued from my private PKI system -> obviously, the root CA from that system is not publicly trusted. So, on some hosts I had my root CA in machine trusted list and on others I didn't have it in three. And those machine are the ones that could not send a configuration backup notification - but were able to send all other email notifications (from backup jobs, backup copy jobs). The moment I imported root CA from my PKI into windows, the notification emails from Configuration backup are also being sent without a problem.

Now, please note again that all other notifications from the same Veeam Backup server were being sent without any issues even before importing the root CA. It was just notification from the configuration backup that didn't get through, due to server cert validation. So it seems that Veeam is using different code to validate mail server certs in case it's sending a configuration backup notification, from the code that is being used when sending a backup job notification, for instance. I find this quite strange.

Now, situations such as this might be rare and this might not touch many users, but I find this behavior problematic for a couple of reasons (below), so I decided to report it:

• receiving a notification when something isn't right is VERY important for any backup operation
• in this case, all other notifications are working normally, only notifications from configuration backups are not being delivered. So in case it stopped working I won't be notified. This is potentially a serious issue, since it's very easy for this to go undetected (unless you also send notifications for Success jobs and also happen to notice that you're missing a mail that you should have received).
• this mail server that I use, I use it to send emails everywhere (including Google and Microsoft servers) and I don't have any problems with delivery at all. Yes, it uses a not-publicly-trused certificate, but that works just fine with pretty much everyone

In any case, I believe Veeam should use the same mechanism regardless of where it's sending notifications from (backup job or configuration job). I see no point in accepting a "self-signed" mail server for some notifications and not accepting it for some other notifications. Ideally: if I sent a test email through Veeam B&R Console and that test went through ok, this should mean that I can have confidence in that notification will be sent when needed.

Hopefully you can fix this. That is, by making mail sending behavior consistent across the board. It doesn't necessarily have to trust mail server with untrusted anchors, but it really should always behave the same.

Cheers,
David

[HannesK]

Hello,
agree with your opinion. To make it easier to fix, could you maybe please upload a full log bundle and post the case number?

Even if you are a free / NFR user, you can create a case. I will make sure it gets directed to the right people.

Thanks,
Hannes

[lucius_the]

Thanks !
Case ID #05061252

[Gostev]

Pretty sure this has to deal with TLS 1.2 and is fixed in 11a (bug 167976).

[lucius_the]

Hm, not likely... It works after I add my cert to trusted store. No workarounds for .NET 4.5 needed. Also all other notifications from B&R work fine. So this doesn't seem to be TLS 1.2 related, as I read in other topics.

Well, ok, I could have tried upgrading to 11a to see if this is also fixed, although it's a different issue. But I would have to try this at a customer sites, where I can replicate the problem easily. And I don't feel like installing an early release on their production... In any case this issue is not urgent for me, I have a fix. This was more to let you know there's a glitch / an inconsistency, that might bite someone if it remains. Cheers

[HannesK]

Thank you for uploading the logs. Agree, it sounds different than in the bug 167976.

I will come back on this.

[HannesK]

Hello,
thanks again for reporting. It is confirmed as bug #350626. The plan is to fix it with V12

Best regards,
Hannes

[gmajestix]

I'm getting the same issue after upgrade to 12.0.0.1420 P20230223:
Sending e-mail report Details: The remote certificate is invalid according to the validation procedure.

[HannesK]

Hello,
can you please provide the support case number for that?

Best regards,
Hannes

[Tom_LeFx]

I suddenly get the same issue for some jobs - they finish, but with warning:
"Sending e-mail report Details: The remote certificate is not valid according to validation procedure"

It worked until yesterday, but suddenly it doesn't want to use my SMTP settings.
If I try to send a test email from that system, the whole Veeam B&R Shell freezes up and a "Please wait, trying to reconnect to backup-server" window starts to flicker like crazy

[Tom_LeFx]

Update:
Could fix my issue - it seemingly has to do with the different approach that v12 takes in verifying the certificate for SSL EMail.
This reddit post brought me to this:
https://www.reddit.com/r/Veeam/comments ... de_to_v12/

What fixed it for me was, to go into global email settings and simply hit "Apply" - then it checked the certificated, notified me, that it is not an officially trusted one (but the correct one from my mail provider) and that was that - Test Email works again, jobs should do

[ertank]

I got same problem with v12 community edition (upgraded from v11).

My case, mail server is a private one with a valid certificate. Certificate was due and renewed from a different trusted brand. This problem emerged after that.

-I could not use domain name in global e-mail settings. After hitting Apply B&R freezes and losts connection to backup server. It could not recover and I have to close it down.
-Using IP number as email server seems to work.
-Trying to to revert to domain name after setting mail server as IP number results in a freeze.

I left it as IP address for now. I have no idea what will it be after next certificate renewal.

[zond80]

Confirm issue mentioned by ertank. Using LE on my own mail server

[enricop]

I had this issue after upgrading to V12 and opened a case March 8th (#05922683).
It took until April 18th for Veeam R&D team to identify the issue: "there's a specific condition in which the certificate retrieval mechanism is not importing all SMTP certificate thumbprints when the mail server is configured in Veeam"

The solution/workaround was to manually modify a tag within an XML field in an SQL table to manually update the certificate thumbprint, the user interface for smtp configuration is still broken, but notifications were sent.
Since a few days notifications are broken again, likely my provider changed certificate, sooo....I have to find out the correct certificate thumbprint, go back to the SQL database and hack the XML field/tag....not really user friendly....

The case was automatically closed because I did not reply to the Automatic Follow Up, I probably should have replied that I need a proper solution, evidently was not obvious.
This is "bound to be fixed in upcoming patches for VBR v12"
FYI: "Latest build: 12.0.0.1420 P20230412 (April 14, 2023)" does not fix the issue.

Good luck!
Enrico

[sysinfma]

We do have to exact same problem since upgrading to V12. Truly terrible!

[sysinfma]

We managed to solve it by generating a new self-signed certificate as support advised to do:

• Generate a new self-signed Certificate:
  https://helpcenter.veeam.com/docs/backu ... ml?ver=120

After generating the certificate (it took a minute to finish) we had to change the smtp hostname to any other host, apply the setting, change it back to the desired smtp host and send a test mail. Otherwise it would not find the new certificate and still return the same error.

[enricop]

Unfortunately asking my email provider to change a perfectly valid public trusted certificate is not an option! (and it would be silly...)

[sysinfma]

I know, this is confusing. Please note that this ist NOT not about the email provider's certificate. It's about some certificate within your Veeam installation. I also don't get the logic behind that, but it does work! Just recreate that certificate within Veeam as described.

[Workbooster]

We managed to solve it by generating a new self-signed certificate as support advised to do:

• Generate a new self-signed Certificate:
  https://helpcenter.veeam.com/docs/backu ... ml?ver=120

I was just investigating the exact same problem at a customers installation, replacing the Veeam internal self signed certificate solfed the problem.
This problem startet on June 1. after the Let's Encrypt cert on our mail server had ben automatically renewed.

[Nestea1]

Dosent work for me, i have a lot of customor with the newest Veeam version 12.0.0.1420 P20230412.
I generate new self signed certifcate and change SMTP server.
One Veeam lost connection to datebase at generate a new certifcate and the other still the same Probblem after accept the certificat for the SMTP server...sending test Mail dosen't work, timeout or Veeam crashes.

This Problem is extremly annoying because the Backups works but nobody trust because missing the mails and you check all veeams manuel.

Disable SSL and go over port 25 if mail provider allow and it works.

[Gostev]

Do they possibly run Veeam on some outdated Windows version without TLS support?

[Nestea1]

I don't think so, Windows Server 2016, 2019, 2022 and Windows 10 22H2.

But after generating new certificates I can SSL over Port 587 with server smtp.gmail.com but another SMTP Server like mail.f2f.ch not.
I accept for both SMTP server the untrusted certificate in Veeam, but I can only send mails with mail.f2f.ch if SSL not allowed and Port 25.
Now I'm a little bit confused, had something changed and not all SMTP server are good enough for Veeam?

[Gostev]

Should not be the case, looks like something is just wrong with that particular SMTP server, since you don't have issues with another one. Or perhaps a Veeam bug that triggers due to some unorthodox SMTP server configuration? Needs to be investigated by support.

[enricop]

@Gostev, as described in my post above, the issue has already been investigated by support, the bug was identified and, according to support, the fix has been developed. See support case #05922683.

What we miss is the fix/patch!!

Initially I was able to make it work hacking an XML field in an SQL table to manually update the certificate thumbprint, for some reason this has stopped working (likely a certificate change by my provider).

Evidently Veeam does not think email notification is important enough to create a dedicated/urgent patch, so we have to wait the "upcoming patches for VBR v12"!
It surprise me that this bug is not even reported in the "Top issues tracker" thread in this forum!
Broken email notification is not considered a top issue? Really?

BTW, when is it planned to be released the next patch for V12??
I really need it badly and it seems I'm not alone.

[Gostev]

The "Top issues tracker" is populated according to the number of support cases, and this particular issues did not result in any significant number of support cases because apparently the issue does not affect most SMTP servers our customers are using. Plus there's a simply workaround of using a different SMTP server.

[EricinIT]

We use Microsoft 365 SMTP servers and all our customers are having this issue. Is there a release date of the patch or is the fix available if I open a support ticket?

Eric

[SergioARGIT]

We are having the same issue with gmail smtp, port 587 and SSL, the workaround is change the username or password y re validate the certificate when you accept the changes, but this solution is just by few time, after restar or something like that you have to do the same process.

[HannesK]

Hello,
and welcome to the forums.

Did you contact support? The ticket above 05922683 mentions a bug #494809 for the "test message". If you see something else, then please open a support ticket and post the case number here for reference.

Best regards,
Hannes

[Mildur]

Hello

Our engineers found a bug related to certificates on the SMTP server. It only affects SMTP configurations where multiple SMTP Servers may be returned by the provided DNS names. Which happens with Gmail and M365.

The fix is already in development and will be released together with our next Cumulative Patch. Unfortunately I don't have an exact date or release window yet to share. We will update this topic after the release so you can test it again.

Best,
Fabian

[Mildur]

Hello

Yesterday, we released the next Cumulative Patch for Veeam Backup & Replication:
https://www.veeam.com/kb4420

Please install this patch and check if the issue still exist.

For v12a we will have more changes around SMTP certification handling.

Best,
Fabian