Unable to open the nworksEventLog on the RMS??

Unleash the power of System Center for vSphere and Hyper-V | Veeam Task Manager for Hyper-V

Re: Unable to open the nworksEventLog on the RMS??

Veeam Logoby toa1970 » Tue Dec 18, 2012 8:32 am

Hi,

I have nWorks 5.7.0.1054, and get the same error. How to solve this?
toa1970
Influencer
 
Posts: 18
Liked: never
Joined: Fri Oct 15, 2010 6:51 pm
Full Name: Torben Andersen

Re: Unable to open the nworksEventLog on the RMS??

Veeam Logoby kidnapoff » Thu Dec 20, 2012 9:32 am

Hi!
We're experiencing the same problem.
nWorks 5.7.0.1054
SCOM 2007 R2 CU6
RMS has been deployed in failover cluster configuration.
kidnapoff
Novice
 
Posts: 3
Liked: never
Joined: Thu Dec 20, 2012 9:27 am
Full Name: Alexey V Ryzhkov

Re: Unable to open the nworksEventLog on the RMS??

Veeam Logoby sergey.g » Thu Dec 20, 2012 1:58 pm 2 people like this post

Hi All,

I'll try to explain why we think this is happening and what can be done to fix the issue.

Symptoms:
Periodically you receive error message about Management Server not being able to access nworks logs or nworks context in WMI. We have a KB article about that. http://www.veeam.com/KB1600

Cause:
Because of the nature of our product we create quite large amount of objects which are not managed or controlled by virtualized infrastructure, all VMs, ESX hosts, Datastores and vCenter servers exist on top of infrastructure which is running inside VMs. In SCOM most of objects(at least objects which have workflows) should be managed by a certain Health Service (Agent or Management Server). For instance if SQL MP creates a database or a table object - it should be managed by Health Service which is running on this SQL server. If something happens with this Health Service, but the created object still exists and is not managed by any entity - it will be managed by RMS. That's a standard SCOM behavior.

In case of our MP, all virtual infrastructure objects are managed by collector servers and the respective Health Service which is installed on this machine. But there are few exceptions:

1. VM <->Windows Coumputer relationship. It's disabled be default, but if you enable it, our MP will "ask" all SCOM agents to create this relationship which also creates a "ghost" virtual machine object which is not managed by any Health Service(it cannot be managed by this agent, because all VMs are managed by collector computers) which should merge with "real" VM thus creating a relationship between virtualized environment and virtualization infrastructure.
2. Split vSphere clusters - on our collector you can split a cluster if it has too many objects. To support this, our collectors create several version of the same cluster objects and only one of them are managed by a collector machine (because other objects are just "ghost" objects and should not be managed).

Usually everything should work like a charm - our MP creates relationship, "ghost" VM merges with real VM which is managed by collector - everything is working as it should. BUT under certain circumstances(most common is different VM name and NETBIOS name of the computer inside) this "ghost" object may never merge with real VM and at this point it will be orphaned and will be managed by RMS, when RMS tries to manage it, it will try to open nworks event log, which RMS usually doesn't have - that's why you see errors.

Same thing with split clusters, usually only real cluster is active in SCOM and collector manages it, while "ghost" cluster objects from other collectors just "merge" with the real one. BUT, if collector with real cluster is experiencing issues uploading data to SCOM, "ghost" version of the same cluster may arrive to SCOM earlier and RMS will try to manage it. Which is ... yep, same errors again.

There is also one more thing - sometimes our customers just kill VMs with agents running inside. This is not a very healthy practice for SCOM, it's better to uninstall the agent first. Now, if you killed a VM, our MP will remove it form topology, but If our MP created a "ghost" VM object and this object has been merged with "real" VM, when real VM is removed - SCOM may keep "ghost" version of it because stub object of SCOM agent is still present in the database. And RMS could still try to manage this "ghost" version of killed VM.

Resolution:
- First of all, if you are using VM<->agent relationship discovery, make sure VM name and netbios name of the guest os is the same. If this cannot be done, create a group for VMs and create an override for this group for "VMGUEST contains OpsMgr Agent" discovery. You will need to enable "removeRelationsip" property of this discovery. This will instruct SCOM to remove "ghost" VM objects.

- When you are decomissioning a VM, remove SCOM agent from this machine before killing it. I beleive there should be some process in place for decomissioning a VM (i.e. remove a machine from AD and any other corporate systems including SCOM). If you still have stub objects in SCOM, try to run "Remove-DisabledMonitoringObject" command in SCOM shell ("Remove-SCOMDisabledClassInstance" in SCOM 2012). If you still have stub VMs, try to do the same thing I suggested for VMs with different names above.

- Perform "Rebuild full topology" task in the nworks Enterprise Manager UI, this should do the final "clean-up" of the nworks topology.

The issue with split cluster should be self-healed when "real" verison of cluster objects arrive to SCOM. However there are a lot of other cases - for instance SCOM agent crashed, or our collector or VM with collector has been lost due to some disaster. All such cases should be investigated, I would recommend to open a support ticket with our support team - we will need to know a lot of things, like all logs from your nworks deployment.

Also I've created a script for SCOM 2007 (let me know if you need one for SCOM 2012) which displays cluster and VM objects which are orphaned. I've attached it to the post. The script could help you determine which objects are causing the issue with Event Logs or WMI.

Let me know if you have any other questions.

P.S. Of course we are going to re-factor MP architecture to avoid situations like this in our future versions.
Attachments
Veeam Logo list_orphaned_vms_clusters_scom2k7.zip
script that displays all orphaned objects
(1.06 KiB) Downloaded 165 times
sergey.g
Veeam Software
 
Posts: 453
Liked: 75 times
Joined: Wed May 02, 2012 1:49 pm
Full Name: Sergey Goncharenko

Re: Unable to open the nworksEventLog on the RMS??

Veeam Logoby kidnapoff » Fri Dec 21, 2012 6:01 am

In our case "VMGUEST contains OpsMgr Agent" discovery not enabled.
kidnapoff
Novice
 
Posts: 3
Liked: never
Joined: Thu Dec 20, 2012 9:27 am
Full Name: Alexey V Ryzhkov

Re: Unable to open the nworksEventLog on the RMS??

Veeam Logoby sergey.g » Fri Dec 21, 2012 8:45 am 1 person likes this post

Hello Alexey,

Do you have split clusters in your nworks Enterprise Manager configuration? Could you also run the script in SCOM shell? Could you run it like this and let us check the csv file?

"c:\folder_where_script_is_located\list_orphaned_vms_clusters_scom2k7.ps1 | export-csv c:\orphaned_nworks_objects.csv"

Were you able to try "Remove-DisabledMonitoringObject" in SCOM shell or "Rebuild full topology" in our UI?

I would really appreciate your help.

Thanks.
sergey.g
Veeam Software
 
Posts: 453
Liked: 75 times
Joined: Wed May 02, 2012 1:49 pm
Full Name: Sergey Goncharenko

Re: Unable to open the nworksEventLog on the RMS??

Veeam Logoby toa1970 » Fri Dec 21, 2012 9:02 am

Hi,

I have nWorks 5.7.0.1054, and get the same error. "VMGuest contains OpsMgr Agent" disabled. Running SCOM 2012. I get the error when using Agentless monitoring. No split clusters in out nworks Enterprise Manager configuration. I have one Windows Server 2000, monitored using Agentless agent.
toa1970
Influencer
 
Posts: 18
Liked: never
Joined: Fri Oct 15, 2010 6:51 pm
Full Name: Torben Andersen

Re: Unable to open the nworksEventLog on the RMS??

Veeam Logoby kidnapoff » Mon Dec 24, 2012 8:52 am

We do not use "split clusters" feature yet/
script shows 4 VMs
kidnapoff
Novice
 
Posts: 3
Liked: never
Joined: Thu Dec 20, 2012 9:27 am
Full Name: Alexey V Ryzhkov

Re: Unable to open the nworksEventLog on the RMS??

Veeam Logoby sergey.g » Mon Dec 24, 2012 11:01 am 1 person likes this post

toa1970 wrote:Hi,

I have nWorks 5.7.0.1054, and get the same error. "VMGuest contains OpsMgr Agent" disabled. Running SCOM 2012. I get the error when using Agentless monitoring. No split clusters in out nworks Enterprise Manager configuration. I have one Windows Server 2000, monitored using Agentless agent.


Helo Torben,

Could you try to run the script I've mentioned in my previous post and check if orphaned VM is an agentless monitored one? We're making series of tests on our end too.

kidnapoff wrote:We do not use "split clusters" feature yet/
script shows 4 VMs


Do you use agentless monitoring to monitor windows computers? Could you send me the results of the script in a csv file? I think you should be able to do this via a direct message on this forum.

Thanks.
sergey.g
Veeam Software
 
Posts: 453
Liked: 75 times
Joined: Wed May 02, 2012 1:49 pm
Full Name: Sergey Goncharenko

Re: Unable to open the nworksEventLog on the RMS??

Veeam Logoby toa1970 » Mon Jan 07, 2013 9:38 am

Hi,

Running SCOM2012, so I can't use the script
toa1970
Influencer
 
Posts: 18
Liked: never
Joined: Fri Oct 15, 2010 6:51 pm
Full Name: Torben Andersen

Re: Unable to open the nworksEventLog on the RMS??

Veeam Logoby sergey.g » Thu Jan 10, 2013 12:16 pm 1 person likes this post

Hello Torben,

I'm attaching a version for SCOM2012. (let me know if you experience any issues with it)
Attachments
Veeam Logo list_orphaned_vms_clusters_scom2012.zip
(816 Bytes) Downloaded 129 times
sergey.g
Veeam Software
 
Posts: 453
Liked: 75 times
Joined: Wed May 02, 2012 1:49 pm
Full Name: Sergey Goncharenko

Re: Unable to open the nworksEventLog on the RMS??

Veeam Logoby toa1970 » Tue Jan 22, 2013 8:22 am

Hi,

I get the following error:
Code: Select all
Get-SCRelationshipInstance : Cannot validate argument on parameter 'TargetInstance'. The argument is null. Supply a non
-null argument and try the command again.
At C:\Temp\list_orphaned_vms_clusters_scom2012.ps1:46 char:60
+     $rel_instance=get-scomrelationshipinstance -targetinstance <<<<  $orphaned_candidates_clusters[$i] | where-object
 {$_.relationshipid -eq $rel_manages1.Id -or $_.relationshipid -eq $rel_manages2.Id} | select-object -first 1
    + CategoryInfo          : InvalidData: (:) [Get-SCRelationshipInstance], ParameterBindingValidationException
    + FullyQualifiedErrorId : ParameterArgumentValidationError,Microsoft.SystemCenter.Core.Commands.GetSCRelationshipI
   nstanceCommand
toa1970
Influencer
 
Posts: 18
Liked: never
Joined: Fri Oct 15, 2010 6:51 pm
Full Name: Torben Andersen

Re: Unable to open the nworksEventLog on the RMS??

Veeam Logoby sergey.g » Tue Jan 22, 2013 12:06 pm 1 person likes this post

Hi Torben,

Looks like you just don't have ESX clusters in your infrastructure, did you get any other objects as orphaned?

I'm also going to attach a new script which is very simple, nonetheless it should more accurately detect orphaned objects. I would apprecaite if you can give it a try in your environment.

Script description:

According to my investigations, looks like an object becomes orphaned when it doesn't have a health service to run workflows assigned for the object. Some management entity is required to run workflows, such an object cannot just drift in SCOM DB. If there is no health service associated with an object (this could happen because of many reasons) then RMS(in SCOM 2007) or Management Servers Resource Pool(SCOM2012) will try to run workflows assigned to the object. In order to detect such scenario I'm using "Microsoft.SystemCenter.ManagementActionPointManagesEntity" relationship. When certain object should be managed by RMS, SCOM creates such a relationship. Then, after some time, the corresponding health service may appear again and our orphaned object will be re-assigned, at this point it will have two relationships of "Microsoft.SystemCenter.ManagementActionPointManagesEntity" class. So, I'm just taking all Veeam VM and Cluster objects and checking if they have "Microsoft.SystemCenter.ManagementActionPointManagesEntity" relationship with RMS and if such a relationship is the only of "Microsoft.SystemCenter.ManagementActionPointManagesEntity" class.
Attachments
Veeam Logo orphaned_vms_and_clusters_scom2012.zip
(512 Bytes) Downloaded 121 times
sergey.g
Veeam Software
 
Posts: 453
Liked: 75 times
Joined: Wed May 02, 2012 1:49 pm
Full Name: Sergey Goncharenko

Re: Unable to open the nworksEventLog on the RMS??

Veeam Logoby wellssh » Mon Mar 03, 2014 12:44 pm

Hello,

We are running SCOM 2012 SP1 UR4, and have imported Veeam nWorks MP 6.5 and this issue is occurring on 6 of our Management Servers as well.
"The Windows Event Log Provider is still unable to open the Veeam VMware event log on computer 'xxxxxxxxx001.americas.PPDI.LOCAL'.
The Provider has been unable to open the Veeam VMware event log for 720 seconds.

Most recent error details: The specified channel could not be found. Check channel configuration.
One or more workflows were affected by this.
Workflow name: many
Instance name: many
Instance ID: many
Management group: PPDI
"

Should I open a support case?

Thanks,
Sven
wellssh
Influencer
 
Posts: 14
Liked: never
Joined: Wed Feb 02, 2011 3:23 pm
Full Name: Sven Wells

Re: Unable to open the nworksEventLog on the RMS??

Veeam Logoby sergey.g » Wed Mar 05, 2014 8:23 am

Hi Sven,

Sorry for a delay in my response. Also, please, accept our appologies for the inconvenience caused by the issue. We think the issue could occur when VMs are migrating very often from host to host, cluster to cluster, or when they often are being created and deleted. Could you provide us with details about your environment - do you have a lot of VM changing their host or a lot of VM deletions?

Currently we are working to find a solution for the next version of our Management Pack, so feedback from you would be highly valuable.

I would also recommend you to open a support ticket, our customer care team should be able to find a workaround for your situation.

Thanks.
sergey.g
Veeam Software
 
Posts: 453
Liked: 75 times
Joined: Wed May 02, 2012 1:49 pm
Full Name: Sergey Goncharenko

Previous

Return to Veeam Management Pack for Microsoft System Center



Who is online

Users browsing this forum: No registered users and 4 guests

cron