Real-time performance monitoring and troubleshooting
Post Reply
doktornotor
Enthusiast
Posts: 94
Liked: 29 times
Joined: Mar 07, 2018 12:57 pm
Contact:

[v12] What's Veeam ONE trying to do with Linux VMs via DCOM?

Post by doktornotor »

I'm just playing with this product again. Overall nice, looking at the logs though, I noticed it's spamming the event log with errors.

There are 3 Linux Hyper-V VMs on the testing Hyper-V server that's being monitored (1 Ubuntu LTS, 2 Debian LTS). For each of them, the log gets spammed with these entries (DistributedCOM event ID 10028):

Code: Select all

DCOM was unable to communicate with the computer fe80::215:5dff:fe00:1234 using any of the configured protocols; requested by PID      708 (C:\Program Files\Veeam\Veeam ONE\Veeam ONE Monitor Server\VeeamDCS.exe), while activating CLSID {8BC3F05E-D86B-11D0-A075-00C04FB68820}.
The error is repeated 3 times per Linux VM every hour (besides the link-local IPv6, it tried to use the global IPv6 and also IPv4 IPs of those machines). This results in hundreds of errors daily and thousands weekly.

Generally, not sure what's Veeam ONE trying to do with Linux machines via DCOM but I'm pretty sure the effort is futile. :wink:
RomanK
Veeam Software
Posts: 648
Liked: 170 times
Joined: Nov 01, 2016 11:26 am
Contact:

Re: [v12] What's Veeam ONE trying to do with Linux VMs via DCOM?

Post by RomanK »

Hello doktornotor,

Veeam ONE uses the Packet Privacy authentication level (the highest one) by default but due to Windows OS-related issues sometimes the resulting authentication level could ignore the configured values and use the second level. Usually, it didn't cause any issues but the DCOM hardened level becomes mandatory. We could face the event spam because some packets ignored the required authentication level.

There were security updates for the MS servers but I am not sure if there is something specific for Linux VMs. Please open a support case and provide the case ID in this thread so we could collect all the additional details and investigate it accordingly.

Thanks
doktornotor
Enthusiast
Posts: 94
Liked: 29 times
Joined: Mar 07, 2018 12:57 pm
Contact:

Re: [v12] What's Veeam ONE trying to do with Linux VMs via DCOM?

Post by doktornotor »

Thanks. There are no DCOM errors logged for Windows VMs. I am just testing/exploring the product with a trial license, not sure about opening tickets.

What I meant is that non-Windows VMs should be skipped from processing of whatever ONE is trying to do with these periodic jobs. There's no DCOM server running on Linux to answer.
RomanK
Veeam Software
Posts: 648
Liked: 170 times
Joined: Nov 01, 2016 11:26 am
Contact:

Re: [v12] What's Veeam ONE trying to do with Linux VMs via DCOM?

Post by RomanK » 1 person likes this post

Hello doktornotor,

I just wanted to update the thread. The issue was recorded and reproduced. The reported behavior should be improved in the next versions. But please, keep in mind that things are subject to change.

Thanks
doktornotor
Enthusiast
Posts: 94
Liked: 29 times
Joined: Mar 07, 2018 12:57 pm
Contact:

Re: [v12] What's Veeam ONE trying to do with Linux VMs via DCOM?

Post by doktornotor »

Thanks for the update, it's not really an urgent issue, just nice to get fixed eventually.
Alkochm
Influencer
Posts: 16
Liked: 1 time
Joined: Aug 02, 2016 3:30 pm
Full Name: Alexey
Contact:

Re: [v12] What's Veeam ONE trying to do with Linux VMs via DCOM?

Post by Alkochm »

Sorry for hijacking this topic. I just don't want to create duplicates. I think that the cause is the same.

After updating from v11 to v12 (12.1.0.3208) we see that VeeamOne server is trying to connect via DCOM to absolutely every guest VM that it can see.
No guest OS credentials is specified anywhere and it is not possible - it is a completely separate network segment.
It wasn't the case with v11 and there is no option to disable this behavior in v12.

On VeeamOne server we see a lot of 10028 EventID erros in System eventlog.
It is some kind of job that occurs exactly every 1 hour.
It tries to connect to port 135/TCP on every VM and also fails authentication because it uses account of VeeamOne service which triggers SIEM. Also it uses NTLM which is a no-no.

If there is any configuration option to disable this behavior - that would be much appreciated.

Error sample:
Log Name: System
Source: Microsoft-Windows-DistributedCOM
Date: 30.01.2024 13:12:24
Event ID: 10028
Task Category: None
Level: Error
Keywords: Classic
User: DOMAIN\VEEAMONE
Computer: VEEAMONE.DOMAIN.LOCAL
Description:
DCOM was unable to communicate with the computer 192.168.10.23 using any of the configured protocols; requested by PID 2434 (C:\Program Files\Veeam\Veeam ONE\Veeam ONE Monitor Server\VeeamDCS.exe).
jorgedlcruz
Veeam Software
Posts: 1383
Liked: 620 times
Joined: Jul 17, 2015 6:54 pm
Full Name: Jorge de la Cruz
Contact:

Re: [v12] What's Veeam ONE trying to do with Linux VMs via DCOM?

Post by jorgedlcruz »

Hello Alexey,
I have checked internally, and we do not trigger this by default, so it is unexpected.

Meanwhile:
  • Please open a support ticket so we can track and this as an issue, add it to our current workload, historical data etc.
  • See if you do not have any Custom alarm looking for Process State, and if so, look at the assignments
    Image
  • A bit more extreme, but try disabling monitoring of the VMs on a few VMs to see if this behaviour stops (of course this disables VONE monitoring of the VM, likce CPU/RAM/etc that we get through vSphere)
    Image
  • Can you confirm what hypervisor you running? And what VMS guest OS you have affected?
Let us know the answer to all the points listed here :)
Jorge de la Cruz
Senior Product Manager | Veeam ONE @ Veeam Software

@jorgedlcruz
https://www.jorgedelacruz.es / https://jorgedelacruz.uk
vExpert 2014-2024 / InfluxAce / Grafana Champion
Alkochm
Influencer
Posts: 16
Liked: 1 time
Joined: Aug 02, 2016 3:30 pm
Full Name: Alexey
Contact:

Re: [v12] What's Veeam ONE trying to do with Linux VMs via DCOM?

Post by Alkochm »

Support ticket: #07113029
Zero custom alarms, only predefined
Disabling monitoring of the VMs is not affecting this behavior
Hypervisor: Hyper-V (Windows Server 2016 / 2019)
VM guest OS: Windows Server 2016 / 2019 / 2022
jorgedlcruz
Veeam Software
Posts: 1383
Liked: 620 times
Joined: Jul 17, 2015 6:54 pm
Full Name: Jorge de la Cruz
Contact:

Re: [v12] What's Veeam ONE trying to do with Linux VMs via DCOM?

Post by jorgedlcruz »

I see, we can let our Engineers work on the support case.

I am starting to suspect that to the Hyper-V VMs, in order to do real-time monitoring we try to connect to them using the ports described here, under Microsoft Windows VM: Let's try another non intrusive change, can you follow the steps in this picture, and disable all the counters for Hyper-V VMs?
Image

And let us know after?

Thank you
Jorge de la Cruz
Senior Product Manager | Veeam ONE @ Veeam Software

@jorgedlcruz
https://www.jorgedelacruz.es / https://jorgedelacruz.uk
vExpert 2014-2024 / InfluxAce / Grafana Champion
Alkochm
Influencer
Posts: 16
Liked: 1 time
Joined: Aug 02, 2016 3:30 pm
Full Name: Alexey
Contact:

Re: [v12] What's Veeam ONE trying to do with Linux VMs via DCOM?

Post by Alkochm »

Ok. I've been there.
First I tried to disable "Process CPU usage", "Process memory usage" and "Running services".
https://dl.dropboxusercontent.com/scl/f ... 53gqd&dl=0
No effect.

Then I disabled all Hyper-V Virtual Machine metrics.
https://dl.dropboxusercontent.com/scl/f ... 7e9mb&dl=0
No effect. (No metrics for VMs obviously)

I don't think it is metrics. Simply because metrics are collected relatively often, like once in a 1 or 5 minutes.
But this activity occurs exactly every 60 minutes starting from Monitoring Service restart.
jorgedlcruz
Veeam Software
Posts: 1383
Liked: 620 times
Joined: Jul 17, 2015 6:54 pm
Full Name: Jorge de la Cruz
Contact:

Re: [v12] What's Veeam ONE trying to do with Linux VMs via DCOM?

Post by jorgedlcruz »

That sounds very strange then. Let our Engineers trying to pin point the issue.

As far as I can see no changes were made on Hyper-v daya collection between versions.

Will monitor the case and see what is happening.

Thank you
Jorge de la Cruz
Senior Product Manager | Veeam ONE @ Veeam Software

@jorgedlcruz
https://www.jorgedelacruz.es / https://jorgedelacruz.uk
vExpert 2014-2024 / InfluxAce / Grafana Champion
Alkochm
Influencer
Posts: 16
Liked: 1 time
Joined: Aug 02, 2016 3:30 pm
Full Name: Alexey
Contact:

Re: [v12] What's Veeam ONE trying to do with Linux VMs via DCOM?

Post by Alkochm »

There is nothing to wait on this case anymore.
I'm using Community Edition and case have been closed: "Unfortunately, due to low support staff availability we were unable to process your request."

I can argue that this is a security issue since VeeamOne server is spraying its service NTLM credentials all over the network.
Oh well.
jorgedlcruz
Veeam Software
Posts: 1383
Liked: 620 times
Joined: Jul 17, 2015 6:54 pm
Full Name: Jorge de la Cruz
Contact:

Re: [v12] What's Veeam ONE trying to do with Linux VMs via DCOM?

Post by jorgedlcruz » 1 person likes this post

Thank you Alexey,
Yes that is current policy for Community users of any Veeam product.

We will keep investigating it and see if we can reproduce it, so far we couldn't and we didn't saw that behaviour.

Will post here if we get to the bottom of it, thank you so much.
Jorge de la Cruz
Senior Product Manager | Veeam ONE @ Veeam Software

@jorgedlcruz
https://www.jorgedelacruz.es / https://jorgedelacruz.uk
vExpert 2014-2024 / InfluxAce / Grafana Champion
SnakeSK
Service Provider
Posts: 56
Liked: 9 times
Joined: Feb 09, 2019 5:06 pm
Contact:

Re: [v12] What's Veeam ONE trying to do with Linux VMs via DCOM?

Post by SnakeSK »

Same on my end. Windows, linux, doesnt matter, it even tries ipv6
jorgedlcruz
Veeam Software
Posts: 1383
Liked: 620 times
Joined: Jul 17, 2015 6:54 pm
Full Name: Jorge de la Cruz
Contact:

Re: [v12] What's Veeam ONE trying to do with Linux VMs via DCOM?

Post by jorgedlcruz »

Hello,
Would you mind opening a support ticket so we can explore what is happening with logs, etc.

Thank you!
Jorge de la Cruz
Senior Product Manager | Veeam ONE @ Veeam Software

@jorgedlcruz
https://www.jorgedelacruz.es / https://jorgedelacruz.uk
vExpert 2014-2024 / InfluxAce / Grafana Champion
SnakeSK
Service Provider
Posts: 56
Liked: 9 times
Joined: Feb 09, 2019 5:06 pm
Contact:

Re: [v12] What's Veeam ONE trying to do with Linux VMs via DCOM?

Post by SnakeSK »

Already have it since december, but on another issue. 07051424
Post Reply

Who is online

Users browsing this forum: No registered users and 3 guests