Veeam intelligent diagnostics failure - Antivirus exclusion needed

[mwgbr]

Hi,

since the upgrade from Veeam One and B&R to v13 we had the error "Veeam intelligent diagnostics failure".

After some troubleshooting with support (ticket is: 07927953) our solution was to exclude the process "c:\Program Files\PowerShell\7\pwsh.exe" from SSL/TLS decryption in our antivirus solution (ESET). I wanted to note it down here, if someone has the same problem.

It was a bit tricky, because:
- we never had to set any exclusions for Veeam to work correcty (but we have set the recommended ones for performance reasons)
- powershell 7 seems new in Veeam v13
- antivirus software did not show any problems like blocked connections or processes in IDS/HIPS/Firewall/HTTPS logs...
- we already tried to reinstall / repair several times
- the data collection overview of B&R in the webinterface of Veeam One was "Healthy" and "installed"
- support saw some powershell related problems in the logs, but it did not look like an antivirus related thing

[david.domask]

Hi mwgbr,

Thanks for sharing the details, and sorry to hear about the troubles.

This is indeed a known situation and has to do with some of the changes done in v13. Effectively, the new gRPC protocol sometimes conflicts with services (read: AV) that does various HTTP(s) inspection actions such as certificate replacement / proxying. By default, Powershell is included in many AV's default monitoring, and the AV tries to do what it's set to do; gRPC (correctly) detects a problem and terminates the connection.

The solution you used is the correct one.

[kevin.boddy]

Hi,

We are having many problems with the latest Veeam ONE. These same errors mentioned about intelligent diagnostics etc. but this was present in v8 as well.
We also have the Veeam ONE just stop working. Veeam ONE client won't connect. No notifications are being sent. After a reboot it starts working again.

Could this be related to this PowerShell 7 gRPC? Your documentation doesn't exactly define what should be excluded in the AV. What exactly makes up "backup infrastructure activity"?

Thanks
Kevin

[david.domask]

Hi Kevin,

I'm not personally aware of issues with this and Veeam One, so best to open a Support Case and let Support review and advise. The Topic Creator's issue may be initiated by Veeam One, but one of the core mechanics behind the Veeam Intelligent Diagnostics is Powershell hence why adding the powershell executable to the allowList helped.

The documentation from the link above notes the issue:

Antivirus software with features like SSL/TLS Filtering may block connectivity for backup infrastructure components

It's best to follow KB1999 and ensure that the executables listed there are allowListed for features like SSL / TLS filtering or HTTP(s) inspection.

[kevin.boddy]

Hi,

The PowerShell exclusion is not mentioned in your documentation or KB but you're confirming it's a known situation. Why is it not documented anywhere?

[david.domask]

Hi,

Please check my first post, the link to the release notes has this information:

Antivirus software with features like SSL/TLS Filtering may block connectivity for backup infrastructure components (e.g., remote PowerShell or Remote Backup Console). Ensure backup infrastructure activity is whitelisted.

Unfortunately the behavior will vary depending on the AV vendor -- as noted above, we observed this behavior with ESET, but other AV with similar features may also result in the same behavior.

[kevin.boddy]

Hi,

As mentioned your linked documentation is vague at best. Surely as the software creator you know what components are required for Veeam to function correctly and as Veeam directly installed this version of PowerShell and is a dependancy your documentation should include these types of AV exclusions seeing as it is a known situation?

What other known situations are there that require manual intervention from end users that are not in your documentation?

Thanks
Kevin

[david.domask]

We're actively working to improve our KB articles on the recommended configuration for antivirus software and will publish updates to the existing KB1999 once completed.

The powershell is noted specifically in the article, but we are working on improvements to the KB articles to better explain what specific actions and binaries need to be excluded.

For v13, the Powershell exclusion discussed in this topic is the main one that I am aware of where it can cause prominent issues unless the powershell binary itself is allowListed.