SSL errors out of nowhere

[Zew]

Hello you everyone!

Case #05291944

So I came into work today and noticed a bunch of failed backup job email alerts. It stated NFC connections errors... ohhh I've been down this road before..

https://zewwy.ca/index.php/2021/04/16/v ... ad-stream/

which of course led me back to Veeams KB (https://www.veeam.com/kb1198) which of course led me to the logs. Which told me...

[17.02.2022 16:30:39.608] < 8428> nfc | SSL connection is required to perform authentication.
[17.02.2022 16:30:39.608] < 8428> nfc | Initializing new SSL connection...
[17.02.2022 16:30:39.733] < 8428> nfc | Establishing connection with the SSL server... Failed.
[17.02.2022 16:30:39.733] < 8428> nfc | Initializing new SSL connection... Failed.
[17.02.2022 16:30:39.733] < 8428> nfc | Opening NFC session with the specified ticket [I'm a bunch of numbers]... Failed.
[17.02.2022 16:30:39.733] < 8428> nfc | Connecting to NFC session. Target host: [im an esxi host]. Storage: [im a storage unit]. VI SOAP connection ID: [im vcenter]. Failed.
[17.02.2022 16:30:39.733] < 8428> | ERR |Failed to initiate NFC session. Target host: [im an esxi host]. VI connection ID: [I'm vcenter]. Storage MOID: [Im a storage unit].
[17.02.2022 16:30:39.733] < 8428> | ERR |SSL error, code: [0].error:00000000:lib(0):func(0):reason(0)
[17.02.2022 16:30:39.733] < 8428> | >> |SSL_connect() function call has failed.
[17.02.2022 16:30:39.733] < 8428> | >> |Failed to establish connection with the SSL server.
[17.02.2022 16:30:39.733] < 8428> | >> |Cannot initialize new SSL connection.
[17.02.2022 16:30:39.733] < 8428> | >> |Cannot initialize SSLv3 connection
[17.02.2022 16:30:39.733] < 8428> | >> |Authd handshake has failed.

I found this thread vmware-vsphere-f24/nfc-storage-connecti ... 30613.html

1) No changes were made before this error started (no updates Windows(Veeam) or VMware(ESXi or vCenter))
2) I have no tried to "enabled SSLv3) nor would I really want to?
3) Veeam is up to date AFAIK (11.0.0.837)
4) Time is synced through n through

Seriously what gives??

[Dima P.]

Hello Zew,

It's hard to say without detailed debug log analysis, so please keep working with our support team and let us know if how the investigation goes or if you need any assistance. Thank you!

[Zew]

I think I may have discovered the root of the problem. will report back in a bit.

[Zew]

It appears a firewall application must have made a change, some packets were noticed as blocked/denied. (though no changes were actually made in terms of the rules themselves.)

An adjustment was made to allow the packets (that should have always been allowed). The jobs appear to be working. I plan to open a support case with the vendor in question in hopes to get an answer as to why this issue started to occur.

[soncscy]

@Zew,

Thank you for your feedback, as my team _constantly_ battles with firewall heuristics on client sites and this gobbles up a lot of time.

For the viewing audience, let me explain what @Zew experienced and let me $@%K your mind with how modern firewall/AV heuristics work.

Let's assume that you have a protection solution that scans all data packets received by a given server.

When you have such a live scan, you have to make a decision:

1. Have fast scans which check for known entities but have a low "unique-ness" value, meaning a non-zero chance that a legit packet and a risky packet return the same value
2. Have slow scans where every value is unique, but there is a performance penalty on the connection stream to ensure that there is never a collision between a legitimate packet and an malicious one.

You can understand the challenge here! Being "too accurate" means your connection is effectively throttled by the AV/FW solution. But, too loose of policies means you might get malicious packets.

No one except you can determine valid/malicious packets, and you must come up with a definition that is appropriate. I will _not_ try to do this, but my hope is anyone reading this understands that security is never 'one and done', you have to consider your attack vector!

I hope this helps some sysadmins with their investigations to understand how legitimate traffic might be flagged as malicious because AV/FW vendors prefer to err on caution instead of reality.

You must investigate yourself and plan accordingly! It is tough, but a reasonable requirement I think!

[Zew]

In this case the rule wasn't even too tight (based solely on the application type itself), but based on what should be the standard for any application it "sees" in general, which was working fine, till the vendors auto update definitions got updated. not sure what happened, still waiting on vendor.