[v12] What's Veeam ONE trying to do with Linux VMs via DCOM?

[doktornotor]

I'm just playing with this product again. Overall nice, looking at the logs though, I noticed it's spamming the event log with errors.

There are 3 Linux Hyper-V VMs on the testing Hyper-V server that's being monitored (1 Ubuntu LTS, 2 Debian LTS). For each of them, the log gets spammed with these entries (DistributedCOM event ID 10028):

Code: Select all

DCOM was unable to communicate with the computer fe80::215:5dff:fe00:1234 using any of the configured protocols; requested by PID      708 (C:\Program Files\Veeam\Veeam ONE\Veeam ONE Monitor Server\VeeamDCS.exe), while activating CLSID {8BC3F05E-D86B-11D0-A075-00C04FB68820}.

The error is repeated 3 times per Linux VM every hour (besides the link-local IPv6, it tried to use the global IPv6 and also IPv4 IPs of those machines). This results in hundreds of errors daily and thousands weekly.

Generally, not sure what's Veeam ONE trying to do with Linux machines via DCOM but I'm pretty sure the effort is futile.

[RomanK]

Hello doktornotor,

Veeam ONE uses the Packet Privacy authentication level (the highest one) by default but due to Windows OS-related issues sometimes the resulting authentication level could ignore the configured values and use the second level. Usually, it didn't cause any issues but the DCOM hardened level becomes mandatory. We could face the event spam because some packets ignored the required authentication level.

There were security updates for the MS servers but I am not sure if there is something specific for Linux VMs. Please open a support case and provide the case ID in this thread so we could collect all the additional details and investigate it accordingly.

Thanks

[doktornotor]

Thanks. There are no DCOM errors logged for Windows VMs. I am just testing/exploring the product with a trial license, not sure about opening tickets.

What I meant is that non-Windows VMs should be skipped from processing of whatever ONE is trying to do with these periodic jobs. There's no DCOM server running on Linux to answer.

[RomanK]

Hello doktornotor,

I just wanted to update the thread. The issue was recorded and reproduced. The reported behavior should be improved in the next versions. But please, keep in mind that things are subject to change.

Thanks

[doktornotor]

Thanks for the update, it's not really an urgent issue, just nice to get fixed eventually.

[Alkochm]

Sorry for hijacking this topic. I just don't want to create duplicates. I think that the cause is the same.

After updating from v11 to v12 (12.1.0.3208) we see that VeeamOne server is trying to connect via DCOM to absolutely every guest VM that it can see.
No guest OS credentials is specified anywhere and it is not possible - it is a completely separate network segment.
It wasn't the case with v11 and there is no option to disable this behavior in v12.

On VeeamOne server we see a lot of 10028 EventID erros in System eventlog.
It is some kind of job that occurs exactly every 1 hour.
It tries to connect to port 135/TCP on every VM and also fails authentication because it uses account of VeeamOne service which triggers SIEM. Also it uses NTLM which is a no-no.

If there is any configuration option to disable this behavior - that would be much appreciated.

Error sample:
Log Name: System
Source: Microsoft-Windows-DistributedCOM
Date: 30.01.2024 13:12:24
Event ID: 10028
Task Category: None
Level: Error
Keywords: Classic
User: DOMAIN\VEEAMONE
Computer: VEEAMONE.DOMAIN.LOCAL
Description:
DCOM was unable to communicate with the computer 192.168.10.23 using any of the configured protocols; requested by PID 2434 (C:\Program Files\Veeam\Veeam ONE\Veeam ONE Monitor Server\VeeamDCS.exe).

[jorgedlcruz]

Hello Alexey,
I have checked internally, and we do not trigger this by default, so it is unexpected.

Meanwhile:

• Please open a support ticket so we can track and this as an issue, add it to our current workload, historical data etc.
• See if you do not have any Custom alarm looking for Process State, and if so, look at the assignments
  
• A bit more extreme, but try disabling monitoring of the VMs on a few VMs to see if this behaviour stops (of course this disables VONE monitoring of the VM, likce CPU/RAM/etc that we get through vSphere)
  
• Can you confirm what hypervisor you running? And what VMS guest OS you have affected?

Let us know the answer to all the points listed here

[Alkochm]

Support ticket: #07113029
Zero custom alarms, only predefined
Disabling monitoring of the VMs is not affecting this behavior
Hypervisor: Hyper-V (Windows Server 2016 / 2019)
VM guest OS: Windows Server 2016 / 2019 / 2022

[jorgedlcruz]

I see, we can let our Engineers work on the support case.

I am starting to suspect that to the Hyper-V VMs, in order to do real-time monitoring we try to connect to them using the ports described here, under Microsoft Windows VM:

• https://helpcenter.veeam.com/docs/one/d ... ml?ver=120

Let's try another non intrusive change, can you follow the steps in this picture, and disable all the counters for Hyper-V VMs?


And let us know after?

Thank you

[Alkochm]

Ok. I've been there.
First I tried to disable "Process CPU usage", "Process memory usage" and "Running services".
https://dl.dropboxusercontent.com/scl/f ... 53gqd&dl=0
No effect.

Then I disabled all Hyper-V Virtual Machine metrics.
https://dl.dropboxusercontent.com/scl/f ... 7e9mb&dl=0
No effect. (No metrics for VMs obviously)

I don't think it is metrics. Simply because metrics are collected relatively often, like once in a 1 or 5 minutes.
But this activity occurs exactly every 60 minutes starting from Monitoring Service restart.

[jorgedlcruz]

That sounds very strange then. Let our Engineers trying to pin point the issue.

As far as I can see no changes were made on Hyper-v daya collection between versions.

Will monitor the case and see what is happening.

Thank you

[Alkochm]

There is nothing to wait on this case anymore.
I'm using Community Edition and case have been closed: "Unfortunately, due to low support staff availability we were unable to process your request."

I can argue that this is a security issue since VeeamOne server is spraying its service NTLM credentials all over the network.
Oh well.

[jorgedlcruz]

Thank you Alexey,
Yes that is current policy for Community users of any Veeam product.

We will keep investigating it and see if we can reproduce it, so far we couldn't and we didn't saw that behaviour.

Will post here if we get to the bottom of it, thank you so much.

[SnakeSK]

Same on my end. Windows, linux, doesnt matter, it even tries ipv6

[jorgedlcruz]

Hello,
Would you mind opening a support ticket so we can explore what is happening with logs, etc.

Thank you!

[SnakeSK]

Already have it since december, but on another issue. 07051424