Unable to open Veeam Collector event log (SCOM)

[philipev]

Hi

I'm getting this a lot:

Date and Time: 15/05/2014 06:24:33
Log Name: Operations Manager
Source: Health Service Modules
Event Number: 26002
Level: 1
Logging Computer: www.www.net
User: N/A
Description:
The Windows Event Log Provider was unable to open the Veeam Collector event log on computer 'www.www.net' for reading. The provider will retry opening the log every 30 seconds. Most recent error details: The specified channel could not be found. Check channel configuration. One or more workflows were affected by this. Workflow name: many Instance name: many Instance ID: many Management group: OMSSE

Event Data:

< DataItem type =" System.XmlData " time =" 2014-05-15T06:24:33.9360195+01:00 " sourceHealthServiceId =" 00648DEC-59F6-A7CD-8146-E63F140D7E7A " >
< EventData >
< Data > OMSSE </ Data >
< Data > many </ Data >
< Data > many </ Data >
< Data > many </ Data >
< Data > Veeam Collector </ Data >
< Data > 30 </ Data >
< Data > The specified channel could not be found. Check channel configuration. </ Data >
< Data > www.www.net </ Data >
< Data />
</ EventData >
</ DataItem >

This is a physical server - one of the Management Servers in our SCOM 2012 instance. What over-ride might I have missed or what monitor may be trying to do this?

Many thanks

Philipev

[sergey.g]

Hi Philip,

We would highly appreciate additional information about the issue. The problem occurs when one of objects created by our Management Pack became orphaned and because of that Management Server is trying to calculate its health. Management Servers may not have all necessary Veeam VMware components installed (VMware Collector) and because of that you will see an error message like the one you mentioned in your post.

We are designing our Management Packs, so that all Veeam objects should be managed by the specific SCOM computer (groups and some other containers - by SCOM management servers, VMware objects by Veeam VMware Collectors). However there could be circumstances when Veeam object may become orphaned and will "fall" to the root SCOM Management Servers resource pool. It could be environment with frequent creation and deletion of virtual machines, or environment with frequently migrating VMs from monitored to un-monitored servers, we are trying to collect as much information as we can about these environments to prevent orphaned objects in the next version of our product.

We think that your case is unique, looks like your management servers "picked-up" an object with workflows targeted to the Veeam Collector log, so this could be virtual switch, or a host or a datastore. Could you please run the following script in your SCOM command shell and provide us with the output? The script will list all objects which are being managed by root resource pool, so that we can check if there are any unwanted(not groups, not containers) orphaned objects in your environment.

https://drive.google.com/file/d/0BxwETG ... sp=sharing

If you have a single orphaned object - this could be because of some unplanned reboot of the SCOM agent which was perfoming discovery operation, in this case "Remove-SCOMDisabledClassInstance" can help you fix the issue. If the issue re-occurs or you have multiple unwanted orphaned objects - we need to know more information about your environment. I would recommend you to open a case with our awesome techsupport team - they can suggest a workaround for the issue and they can help us collect additional information, so that we can further investigate the issue and prevent it in the next versions of our Management Packs.

Thanks.

[philipev]

Hi there - apologies for the delayed response. I don't seem to be able to resolve that URL (for the script download). Could you email a copy to me with a mail-gate friendly extension? Many thanks

[sergey.g]

Hi, I've sent you a DM

[sepj12927]

Hi

I have the same issue with all my 4 management servers. I usually help to run a rebuild full topology. But it would be good to know if the situation could be avoided.

/Per

[Kiwifulla]

Hi there
I ran the script supplied and can see 4 VMs in the output that are indeed the ones appearing in the detailed events in the Event Log on the OPSMGR servers - and still appearing under the "All VMs" SCOM Veeam MP view, even though these VMs were deleted a week ago from vCenter, and removed from SCOM (from Agent Managed). We have run the "Remove-SCOMDisabledClassInstance" commandlet and run a Veeam "Rebuild Full Topology" which doesn't get rid of them either.

We logged a Veeam Support call the last time this happened with 2 VMs about 6 months back, however the solution we were provided then was to uninstall/reinstall the MP. This was not practical for us - especially as the dependencies become a nightmare etc., so we instead logged a call with Microsoft who gave us some SQL scripts to remove the dead entries (which worked). This happened again a few months ago and we had to do the SQL hacks again for these VMs.

Now it seems every time we decommission a VM this happens. Is there a special method for the MP/Collector when decommissioning VMs? Surely removing the managed agent and rebuilding the topology should get rid of them?

Cheers

Steve

[sergey.g]

Hi Steve,

I would highly appreciate more details about the issue. Which version of our Management Pack are you using? Do you have any overrides for Veeam MP discoveries (VMs discovery, VMGUEST to Ops Mgr Agent discovery)?

I assume solution from Microsoft was about cleaning up data for decomissioned SCOM agents. There is a known limitation with such a process - you have to wait 3 days before SCOM removes records about deleted or decomissioned agent or when you need to install a management server or a gateway on the same machine.

Could you provide us with more information about your process of decomissioning VMs? Do you just shutdown a VM and then delete it from vCenter inventory or you uninstall Ops Mgr agent and remove it from your active derectory first?

The issue exists because our VM to Agent script connects the dots: VM data gathered by our MP and Ops Mgr agent data from windows computer running inside this VM. And when one part of this relationship is undiscovered (Veeam data), the other part (ghost Ops Mgr agent) may "lock" the object in the database. We beleive that next time our dicsovery script is executed (once a day by default) - it should remove the orphaned relationship and the issues should go away, but something may be happening in your environment which doesn't allow SCOM to "release" this discovery packet because windows computer data is still "floating" somewhere in the database.

So, I would highly appreciate additional information about this issue.

Thanks

[Kiwifulla]

Hi Sergey

We are running Veeam 6.5 (6.5.0.1549). We have not overridden any discoveries at all (only the two from the out-of-box Veeam override MP however these appear to be related to backups).

The solution from Microsoft is attached (two SQL scripts). I assumed this process cleaned up data for the VM objects rather than the decommissioned SCOM agents (or in fact both?). Once running these two scripts, within minutes the rouge VMs disappeared from the "All VMs" MP view.

However, the error Events (ID 26004) were still occurring every 12 minutes on the Management Servers (for hours), so we ran "Remove-SCOMDisabledClassInstance" again (not sure if this was necessary) and re-triggered the Veeam Collector topology resend. We waited for hours however in the end we also had to stop the SCOM services on the Management Servers, clear the cache and restart the services again. This stopped the Events from reoccurring and the SCOM alert cleared automatically.

The VM decom process (I am told by our SysOps Team) is to delete the agent from "Agent Managed" in the SCOM console first, then to shutdown the VM. Supposedly they leave the VM in vCenter for a week (just in case) and then delete it.

Your assistance is much appreciated. I haven't run the Microsoft solution this time as I thought I would see if there is a preferred Veeam resolution or fix for this.

Thanks again

Steve

Code: Select all

--Run the following SP to mark the related entities to deleted.
USE OperationsManager
CREATE TABLE #DIRTY(TypedManagedEntityId UNIQUEIDENTIFIER)
INSERT INTO #DIRTY(TypedManagedEntityId)
SELECT TypedManagedEntityId FROM TypedManagedEntity WHERE BaseManagedEntityId IN ( SELECT BME.BaseManagedEntityId FROM BaseManagedEntity BME WHERE BME.DisplayName like '%<agent name>%' )
DECLARE @EntityId UNIQUEIDENTIFIER
DECLARE @TimeGenerated DATETIME
DECLARE myCursor CURSOR FOR
SELECT TypedManagedEntityId FROM #DIRTY
OPEN myCursor
FETCH NEXT FROM myCursor INTO @EntityId
WHILE @@FETCH_STATUS = 0
BEGIN
      SET @TimeGenerated = GETUTCDATE()
      EXEC dbo.p_TypedManagedEntityDelete @EntityId, @TimeGenerated
      FETCH NEXT FROM myCursor INTO @EntityId END
CLOSE myCursor
DEALLOCATE myCursor
DROP TABLE #DIRTY
 
--Run the following SP to purge the deleted items to let it take effective immediately in  SCOM console.
USE [OperationsManager]
declare @rowcount int
DECLARE @Err int
DECLARE @GroomingThresholdUTC datetime
set @GroomingThresholdUTC = getutcdate()
EXEC @Err = dbo.p_DiscoveryDataPurgingByTypedManagedEntity @GroomingThresholdUTC,250, @rowcount
EXEC @Err = dbo.p_DiscoveryDataPurgingByRelationship @GroomingThresholdUTC,250, @rowcount
EXEC @Err = dbo.p_DiscoveryDataPurgingByBaseManagedEntity @GroomingThresholdUTC,250, @rowcount