Unable to open the nworksEventLog on the RMS??

[maekee]

After upgrading to nWorks VMware Management Pack for OpsMgr to the latest version 5.5 i began to get this alerts (EventID's) on the OpsMgr RMS:

Event ID: 25002
Event Source: Health Service Modules
Event Description: The Windows Event Log Provider was unable to open the nworksEventLog event log on computer 'rmsserver.domain.local' for reading. The Provider will retry opening the log every 30 seconds. Most recent error details: The system cannot find the file specified.


So why is OpsMgr trying to open a Eventlog that dont exist on our RMS, the nworksEventLog is located on the VIC-server.
I cannot override this monitor per EventLog and cannot disable it because then it wont monitor the other Eventlogs on the RMS.

Help

/Maekee

[Alec King]

Hi maekee,

This is an unusual problem....I don't see this in our test environment. It seems the MP is trying to use the RMS as a target for some rules or monitors which depend on nworksEventLog.

Did you ever have nworks Collector or Management Center components installed on the RMS? Perhaps there are registry keys remaining, or maybe Ops Mgr has not fully cleaned up the configuration. If you look in the Enterprise Manager dashboard view, do you see the RMS there - has RMS been 'discovered' as a Collector or an Enterprise Manager server?

Please open a case with our support - if you could send all logfiles, and details of your configuration, we'll investigate.

Thanks!
Alec

[maekee]

Hi Alec,

Yes, i have not seen this before.

Its only the Managements Pack that is imported on the RMS, no SW is installed there.

I have checked the Registry that holds the Event Logs installed on the RMS (HKLM\SYSTEM\CurrentControlSet\Services\Eventlog)
There is no nworksEventLog-key there. If i look in _Management Center Dashboard the RMS is not detected as a Collector.

Where does this get the nWorksEventlog from?

/Maekee

[maekee]

I created a support case.. thnks

/maekee

[GDavies]

Hi

Did you manage to resolve this? I have the same problem on version 5.0.3.248

Cheers

Graham

[Vitaliy S.]

Graham,

As far as I know this is a known issue, and it has been already fixed in our upcoming version of MP 5.5

Thanks!

[Windessy]

it was fixed in MP 5.5 RC2 exactly.

[spujado]

Hi,
since three days now I am experiencing the same error on our RMS. We are running R2cu1 on a 2008 sp2 server with Nmworks MP 5.5.1.493 and the only thing that has changed is that this recommended hotfix (http://support.microsoft.com/kb/981263) has been applied.
Any help will be welcome

Tkx

[Irina Simkina]

Hi Spujado!

could you please open the support case using the support@veeam.com and provide us the detailed error message (with something like:
Most recent error details: xxx.
One or more workflows were affected by this.
Workflow name: nworks.VMware.VEM.VMHOSTDATASTORE.Monitor.diskPercUsed
Instance name: xxx
Instance ID: {xxx}
Management group: xxx
)
or the complete nworksEventLog wich is better.

Thank you!
Irina

[dsloyer]

As of 5.7, this issue has still not been fixed. Case was opened with Veeam and I was instructed to rebuild topology. Full topology rebuild did not correct the issue. It was explained to me that a managed object was orphaned and defaulted to the RMS as a collector, but the RMS has no collector services installed. Nothing Veeam realted is installed other than the nWorks 5.7 managment pack and the nworks.VMware.2007R2RequiredOverrides MP. This issue still persists with no resolution from Veeam.

Date and Time: Date/Time
Log Name: Operations Manager
Source: Health Service Modules
Event Number: 26004
Level: 1
Logging Computer: RMS-NODEB.mydomain.com
User: N/A
Description:
The Windows Event Log Provider is still unable to open the nworksEventLog event log on computer 'RMS-NODEB.mydomain.com'. The Provider has been unable to open the nworksEventLog event log for 149040 seconds. Most recent error details: The specified channel could not be found. Check channel configuration. One or more workflows were affected by this. Workflow name: many Instance name: many Instance ID: many Management group: MyManagementGroup.

Event Data:

< DataItem type =" System.XmlData " time =" TIME " sourceHealthServiceId =" 29D87AB8-9F3F-37C1-2AFC-D6FCB4361715 " (HealthServiceID of RMS VirtualComputerName)>
< EventData >
< Data > MyManagementGroup </ Data >
< Data > many </ Data >
< Data > many </ Data >
< Data > many </ Data >
< Data > nworksEventLog </ Data >
< Data > 149040 </ Data >
< Data > The specified channel could not be found. Check channel configuration. </ Data >
< Data > RMS-NODEB.mydomain.com </ Data >
< Data />
</ EventData >
</ DataItem >

[Vitaly.Mzokov]

Hello David,

I am so much sorry to hear that you have not resolved the issue yet. Let me discuss your problem with my colleagues at support department.

Could you please let me know a support case number you have for this issue? Our records indicate that you have 2 cases opened with us:

• 00125725
  00125734

Which one would be best to use?

-VM.

[dsloyer]

00125725 is the correct case number for this issue.

[Vitaly.Mzokov]

David,

As far as I see from our tracking system, our engineer has replied you back on your corporate email address and asked for log files from nworks Enterprise Manager console for further investigation. That message dated November,8th.
Could you please log on to the Customer Portal and check the case's status and its email chain? Also you should be able to upload all the necessary files via Customer Portal.

-VM.

[felyjos]

Hello,

Any information on this I have the same error MP 5.7.0.1022

Thanks,
Dom

[Vitaly.Mzokov]

Dom,

Unfortunately, this is still under investigation by R&D. Could you please open a support case with us, so we've got way more information to deeply analyze the behavior? Thanks!

-VM.

[toa1970]

Hi,

I have nWorks 5.7.0.1054, and get the same error. How to solve this?

[kidnapoff]

Hi!
We're experiencing the same problem.
nWorks 5.7.0.1054
SCOM 2007 R2 CU6
RMS has been deployed in failover cluster configuration.

[sergey.g]

Hi All,

I'll try to explain why we think this is happening and what can be done to fix the issue.

Symptoms:
Periodically you receive error message about Management Server not being able to access nworks logs or nworks context in WMI. We have a KB article about that. http://www.veeam.com/KB1600

Cause:
Because of the nature of our product we create quite large amount of objects which are not managed or controlled by virtualized infrastructure, all VMs, ESX hosts, Datastores and vCenter servers exist on top of infrastructure which is running inside VMs. In SCOM most of objects(at least objects which have workflows) should be managed by a certain Health Service (Agent or Management Server). For instance if SQL MP creates a database or a table object - it should be managed by Health Service which is running on this SQL server. If something happens with this Health Service, but the created object still exists and is not managed by any entity - it will be managed by RMS. That's a standard SCOM behavior.

In case of our MP, all virtual infrastructure objects are managed by collector servers and the respective Health Service which is installed on this machine. But there are few exceptions:

1. VM <->Windows Coumputer relationship. It's disabled be default, but if you enable it, our MP will "ask" all SCOM agents to create this relationship which also creates a "ghost" virtual machine object which is not managed by any Health Service(it cannot be managed by this agent, because all VMs are managed by collector computers) which should merge with "real" VM thus creating a relationship between virtualized environment and virtualization infrastructure.
2. Split vSphere clusters - on our collector you can split a cluster if it has too many objects. To support this, our collectors create several version of the same cluster objects and only one of them are managed by a collector machine (because other objects are just "ghost" objects and should not be managed).

Usually everything should work like a charm - our MP creates relationship, "ghost" VM merges with real VM which is managed by collector - everything is working as it should. BUT under certain circumstances(most common is different VM name and NETBIOS name of the computer inside) this "ghost" object may never merge with real VM and at this point it will be orphaned and will be managed by RMS, when RMS tries to manage it, it will try to open nworks event log, which RMS usually doesn't have - that's why you see errors.

Same thing with split clusters, usually only real cluster is active in SCOM and collector manages it, while "ghost" cluster objects from other collectors just "merge" with the real one. BUT, if collector with real cluster is experiencing issues uploading data to SCOM, "ghost" version of the same cluster may arrive to SCOM earlier and RMS will try to manage it. Which is ... yep, same errors again.

There is also one more thing - sometimes our customers just kill VMs with agents running inside. This is not a very healthy practice for SCOM, it's better to uninstall the agent first. Now, if you killed a VM, our MP will remove it form topology, but If our MP created a "ghost" VM object and this object has been merged with "real" VM, when real VM is removed - SCOM may keep "ghost" version of it because stub object of SCOM agent is still present in the database. And RMS could still try to manage this "ghost" version of killed VM.

Resolution:
- First of all, if you are using VM<->agent relationship discovery, make sure VM name and netbios name of the guest os is the same. If this cannot be done, create a group for VMs and create an override for this group for "VMGUEST contains OpsMgr Agent" discovery. You will need to enable "removeRelationsip" property of this discovery. This will instruct SCOM to remove "ghost" VM objects.

- When you are decomissioning a VM, remove SCOM agent from this machine before killing it. I beleive there should be some process in place for decomissioning a VM (i.e. remove a machine from AD and any other corporate systems including SCOM). If you still have stub objects in SCOM, try to run "Remove-DisabledMonitoringObject" command in SCOM shell ("Remove-SCOMDisabledClassInstance" in SCOM 2012). If you still have stub VMs, try to do the same thing I suggested for VMs with different names above.

- Perform "Rebuild full topology" task in the nworks Enterprise Manager UI, this should do the final "clean-up" of the nworks topology.

The issue with split cluster should be self-healed when "real" verison of cluster objects arrive to SCOM. However there are a lot of other cases - for instance SCOM agent crashed, or our collector or VM with collector has been lost due to some disaster. All such cases should be investigated, I would recommend to open a support ticket with our support team - we will need to know a lot of things, like all logs from your nworks deployment.

Also I've created a script for SCOM 2007 (let me know if you need one for SCOM 2012) which displays cluster and VM objects which are orphaned. I've attached it to the post. The script could help you determine which objects are causing the issue with Event Logs or WMI.

Let me know if you have any other questions.

P.S. Of course we are going to re-factor MP architecture to avoid situations like this in our future versions.

[kidnapoff]

In our case "VMGUEST contains OpsMgr Agent" discovery not enabled.

[sergey.g]

Hello Alexey,

Do you have split clusters in your nworks Enterprise Manager configuration? Could you also run the script in SCOM shell? Could you run it like this and let us check the csv file?

"c:\folder_where_script_is_located\list_orphaned_vms_clusters_scom2k7.ps1 | export-csv c:\orphaned_nworks_objects.csv"

Were you able to try "Remove-DisabledMonitoringObject" in SCOM shell or "Rebuild full topology" in our UI?

I would really appreciate your help.

Thanks.

[toa1970]

Hi,

I have nWorks 5.7.0.1054, and get the same error. "VMGuest contains OpsMgr Agent" disabled. Running SCOM 2012. I get the error when using Agentless monitoring. No split clusters in out nworks Enterprise Manager configuration. I have one Windows Server 2000, monitored using Agentless agent.

[kidnapoff]

We do not use "split clusters" feature yet/
script shows 4 VMs

[sergey.g]

Hi,

I have nWorks 5.7.0.1054, and get the same error. "VMGuest contains OpsMgr Agent" disabled. Running SCOM 2012. I get the error when using Agentless monitoring. No split clusters in out nworks Enterprise Manager configuration. I have one Windows Server 2000, monitored using Agentless agent.

Helo Torben,

Could you try to run the script I've mentioned in my previous post and check if orphaned VM is an agentless monitored one? We're making series of tests on our end too.

We do not use "split clusters" feature yet/
script shows 4 VMs

Do you use agentless monitoring to monitor windows computers? Could you send me the results of the script in a csv file? I think you should be able to do this via a direct message on this forum.

Thanks.

[toa1970]

Hi,

Running SCOM2012, so I can't use the script

[sergey.g]

Hello Torben,

I'm attaching a version for SCOM2012. (let me know if you experience any issues with it)

[toa1970]

Hi,

I get the following error:

Code: Select all

Get-SCRelationshipInstance : Cannot validate argument on parameter 'TargetInstance'. The argument is null. Supply a non
-null argument and try the command again.
At C:\Temp\list_orphaned_vms_clusters_scom2012.ps1:46 char:60
+     $rel_instance=get-scomrelationshipinstance -targetinstance <<<<  $orphaned_candidates_clusters[$i] | where-object
 {$_.relationshipid -eq $rel_manages1.Id -or $_.relationshipid -eq $rel_manages2.Id} | select-object -first 1
    + CategoryInfo          : InvalidData: (:) [Get-SCRelationshipInstance], ParameterBindingValidationException
    + FullyQualifiedErrorId : ParameterArgumentValidationError,Microsoft.SystemCenter.Core.Commands.GetSCRelationshipI
   nstanceCommand

[sergey.g]

Hi Torben,

Looks like you just don't have ESX clusters in your infrastructure, did you get any other objects as orphaned?

I'm also going to attach a new script which is very simple, nonetheless it should more accurately detect orphaned objects. I would apprecaite if you can give it a try in your environment.

Script description:

According to my investigations, looks like an object becomes orphaned when it doesn't have a health service to run workflows assigned for the object. Some management entity is required to run workflows, such an object cannot just drift in SCOM DB. If there is no health service associated with an object (this could happen because of many reasons) then RMS(in SCOM 2007) or Management Servers Resource Pool(SCOM2012) will try to run workflows assigned to the object. In order to detect such scenario I'm using "Microsoft.SystemCenter.ManagementActionPointManagesEntity" relationship. When certain object should be managed by RMS, SCOM creates such a relationship. Then, after some time, the corresponding health service may appear again and our orphaned object will be re-assigned, at this point it will have two relationships of "Microsoft.SystemCenter.ManagementActionPointManagesEntity" class. So, I'm just taking all Veeam VM and Cluster objects and checking if they have "Microsoft.SystemCenter.ManagementActionPointManagesEntity" relationship with RMS and if such a relationship is the only of "Microsoft.SystemCenter.ManagementActionPointManagesEntity" class.

[wellssh]

Hello,

We are running SCOM 2012 SP1 UR4, and have imported Veeam nWorks MP 6.5 and this issue is occurring on 6 of our Management Servers as well.
"The Windows Event Log Provider is still unable to open the Veeam VMware event log on computer 'xxxxxxxxx001.americas.PPDI.LOCAL'.
The Provider has been unable to open the Veeam VMware event log for 720 seconds.

Most recent error details: The specified channel could not be found. Check channel configuration.
One or more workflows were affected by this.
Workflow name: many
Instance name: many
Instance ID: many
Management group: PPDI"

Should I open a support case?

Thanks,
Sven

[sergey.g]

Hi Sven,

Sorry for a delay in my response. Also, please, accept our appologies for the inconvenience caused by the issue. We think the issue could occur when VMs are migrating very often from host to host, cluster to cluster, or when they often are being created and deleted. Could you provide us with details about your environment - do you have a lot of VM changing their host or a lot of VM deletions?

Currently we are working to find a solution for the next version of our Management Pack, so feedback from you would be highly valuable.

I would also recommend you to open a support ticket, our customer care team should be able to find a workaround for your situation.

Thanks.