VMware Tools Heartbeat Status

Unleash the power of System Center for vSphere and Hyper-V | Veeam Task Manager for Hyper-V

Re: VMware Tools Heartbeat Status

Veeam Logoby Vitaliy S. » Wed Dec 23, 2009 10:39 am

Arkadiusz,

That's right, you can perform WBEMTEST (as it is described in our Guides) on the Collector Machine to see whether collector established communication with your hosts or not. Usually it takes few minutes, depends on the size of your virtual infrastructure
Vitaliy S.
Veeam Software
 
Posts: 19564
Liked: 1104 times
Joined: Mon Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov

Re: VMware Tools Heartbeat Status

Veeam Logoby sameerdave » Mon Jan 04, 2010 9:44 pm

We have seen these alerts in loads in our environment.
And finally Veeam found that there is a bug which they will update in their next version.

So currently you will have to live with these alerts as there is no work around. Sadly, I receive around 2k alerts per day, OR we have to disable the VM monitoring, which would not make sense..

Wish we knew about this buy before we bought the product..
sameerdave
Enthusiast
 
Posts: 29
Liked: never
Joined: Wed Sep 30, 2009 2:42 pm
Full Name: Sameer Dave

Re: VMware Tools Heartbeat Status

Veeam Logoby Alec King » Mon Jan 04, 2010 10:46 pm

Hi Sameer Dave,

I am sorry you are receiving a lot of alerts; and we will of course assist you to get the best nworks configuration for your environment.

However I have to point out that if there IS a "bug" - it is in Virtual Center, not in the nworks MP.
All that our Collector does, is check the 'VMware Tools Heartbeat Status' for each VM, exactly as the VMware API publishes it. If this status is 'Red', or 'Yellow' - then we generate an alert.

This 'VMTools Heartbeat Status' has been shown in the past to be somewhat unstable/unreliable - *as published in Virtual Center, not in our product!*
Sometimes when we check the vCenter API, we get a status of Red or Yellow, for no apparent reason. Within a few minutes, the status returns to Green - again for no apparent reason. This is why we introduced a Timer in the MP Monitor, so that we would only alert if the Status *remained* Red/Yellow for 2 intervals, rather than alert immediately.

It seems that in some environments, the default (2 intervals = 10 minutes) is not enough - so you can either
* increase the interval by modifying the parameter 'CorrelationInterval' on the Monitor.
* disable the VMTools Heartbeat Monitor (not this does not disable all "VM monitoring" - only this particular state event. All VM performance data, all other VC events, including power changes etc, will still be captured)

So - we are happy to assist you, as I said, to get the best value from the nworks MP. But the 'bottom line' is that we are always dependent on the data that is published by the VMware API, whether that is performance data, events, or VM state-change as in this case.

I hope that clears up the issue, please let me know.

For *anyone* receiving these 'Heartbeat' alerts - please feed back to this thread if you are experiencing this issue, and whether increasing CorrelationInterval helps. This is the feedback we need, we can take it to VMware, and work with them to find a solution for this instability in the Heartbeat value we get from the VI-API.

Thanks!
Alec

Senior Product Manager
nworks
Alec King
Veeam Software
 
Posts: 700
Liked: 116 times
Joined: Sun Jan 01, 2006 1:01 am

Re: VMware Tools Heartbeat Status

Veeam Logoby krowczynski » Tue Jan 05, 2010 10:56 am

Hello,

where can I increase the CorrelationInterval for the hearbeat alert please?
krowczynski
Enthusiast
 
Posts: 62
Liked: never
Joined: Thu Dec 17, 2009 6:34 pm
Full Name: Arkadiusz Krowczynski

Re: VMware Tools Heartbeat Status

Veeam Logoby sameerdave » Tue Jan 05, 2010 1:55 pm

Thanks Alec,
From what i understood, that this would be fixed in the 5.5 version, as in this case we are receiving old alerts from the vc.

Anyways, in ref to your override, I see the Correlations Interval which is at 600, and Correlation count which is at 1. Whats the difference between count and interval?
So i guess the correlations interval is in seconds, and we can change that settings from 600 to 1200 or 900..

Krowczynski - You can find these counters by going to Authoring-Monitors - Serach for "VMWare Tools Heartbeat" and then clicking on override settings for the monitors..
sameerdave
Enthusiast
 
Posts: 29
Liked: never
Joined: Wed Sep 30, 2009 2:42 pm
Full Name: Sameer Dave

Re: VMware Tools Heartbeat Status

Veeam Logoby Alec King » Tue Jan 05, 2010 4:12 pm

Hi SameerDave,

CorrelationInterval is the Timer period before an alert will be raised. CorrelationCount is the number of events required to 'reset' the timer.

So two examples for this VM Heartbeat Monitor could be -

1. This example does NOT generate an alert:

nworks requests Heartbeat Status from the VI-API, and gets "Red"
nworks throws Heartbeat=Red event
Event triggers Timer start for Monitor (CorrelationInterval = 600 seconds)
After 300 seconds, nworks requests Heartbeat Status from the VI-API, and gets "Green"
nworks throws Heartbeat=Green event
As CorrelationCount=1, this 1 event resets the Monitor.
No alert is fired.

2. This example WILL generate an alert:

nworks requests Heartbeat Status from the VI-API, and gets "Red"
nworks throws Heartbeat=Red event
Event triggers Timer start for Monitor (CorrelationInterval = 600 seconds)
After 300 seconds, nworks requests Heartbeat Status from the VI-API, and still gets "Red"
After 600 seconds, nworks requests Heartbeat Status from the VI-API, and still gets "Red"
CorrelationInterval Timer has expired at 600s.
Alert is generated.

So, to answer your question - yes you can override that CorrelationInterval value to 900, 1200, 1500....some multiple of your baseline nworks Collector interval (default 300s)
Note - do not override CorrelationCOUNT. We only throw an event for State-change; we don't throw repeating events. So CorrelationCount should stay as "1".
Increasing that CorrelationInterval out-of-box is one change we will probably make in MP v5.5.
And at the same time we will continue working with VMware to try and track down the reasons for the instability in the value reported by the VI-API...this is the root of the problem.

Thanks!
Alec
Alec King
Veeam Software
 
Posts: 700
Liked: 116 times
Joined: Sun Jan 01, 2006 1:01 am

Re: VMware Tools Heartbeat Status

Veeam Logoby sameerdave » Tue Jan 05, 2010 4:51 pm

Thanks Alec for the detailed explanation.

In ref to the VMware Tools hearbeat, we can change the correlationinterval, that is fine. But i do get quite a huge number of alerts for "VM CPU Alarm changed to Red". We get like 3-10 every minute. While talking to support, they said these are old events that are getting generated. Would you have any workaround for this? I don't think these have any override parameters, but if you could help us here, that would be great.

Thanks
Sameer
sameerdave
Enthusiast
 
Posts: 29
Liked: never
Joined: Wed Sep 30, 2009 2:42 pm
Full Name: Sameer Dave

Re: VMware Tools Heartbeat Status

Veeam Logoby Alec King » Tue Jan 05, 2010 5:42 pm

Hi Sameer,

Regarding the 'VM CPU Alarm' alerts -
These alerts are based on the Alarms as configured in Virtual Center. As with Heartbeat, there is an event that is thrown when the CPU Usage changes to "Red" (as Virtual Center is configured to see it). And there is an MP Monitor that responds to those events.

These Alarms run in parallel, at the same time as the Monitor for VM CPU usage that is included in the MP. Our CPU Usage Monitor is configured in SCOM to respond to the actual value of cpuUsedPct metric.

So, really you have 2 different alerts that are watching the same thing - VM CPU.
One threshold (for VM CPU Alarm event) is configured in VC. That's "VM CPU Alarm changed to Red/Yellow" as seen in SCOM.
One threshold (for VM CPU Usage %) is configured in SCOM. That's "VM CPU Usage has exceeded threshold" as seen in SCOM.

I would advise you to use the MP CPU Usage Monitors only. They are configured within SCOM, so it is much easier to set overrides. And the alerts are more informative - they will give you the actual value in %, rather than just saying "to Red" or "to Yellow".

So you could just disable the Alarm for VM CPU - either by disabling the MP Monitors (for Red, and Yellow) - or by disabling the Alarm itself, within Virtual Center. it is just providing duplicate information. Our MP "CPU Usage threshold" Monitor is more useful.

Regarding what support said, that you are receiving old (cached?) events that are repeating..? Can you give me the Support Ticket number? I will look into that for you.

Cheers,
Alec
Alec King
Veeam Software
 
Posts: 700
Liked: 116 times
Joined: Sun Jan 01, 2006 1:01 am

Re: VMware Tools Heartbeat Status

Veeam Logoby krowczynski » Tue Jan 05, 2010 6:12 pm

Hello Alec,

where can I find the Authoring-Monitors to change the setting for heartbeat messeges. I get to many of them...
krowczynski
Enthusiast
 
Posts: 62
Liked: never
Joined: Thu Dec 17, 2009 6:34 pm
Full Name: Arkadiusz Krowczynski

Re: VMware Tools Heartbeat Status

Veeam Logoby sameerdave » Tue Jan 05, 2010 6:19 pm

Thanks Alec. The support ticket is ID#517901. These are alerts we are getting from old logs.

Krowczynski - Yes, its under the SCOM console - Authoring - Expand Managed Pack Objects and Monitors. There do a find for VMWare Tools Heartbeat, and you would see the monitors there. You can right click on these monitors and create an override.
sameerdave
Enthusiast
 
Posts: 29
Liked: never
Joined: Wed Sep 30, 2009 2:42 pm
Full Name: Sameer Dave

Re: VMware Tools Heartbeat Status

Veeam Logoby Alec King » Tue Jan 05, 2010 6:20 pm

Hi Krowczynski,

In SCOM Console, you find Authoring section in the navigation buttons in bottom left. (you need to be SCOM Administrator with Authoring privileges)
When you are in Authoring, find Management Pack Objects - Monitors in the tree view.
Then you can search for 'nworks' using the 'Look For' box. (make sure you clear any existing Scope)
Find the Target "VMGuest Virtual Machine" in the main view, expand it and and you will see the Heartbeat Monitors (Red, and Yellow) and you can change them there in the usual way (right click and override)

Cheers
Alec
Alec King
Veeam Software
 
Posts: 700
Liked: 116 times
Joined: Sun Jan 01, 2006 1:01 am

Re: VMware Tools Heartbeat Status

Veeam Logoby krowczynski » Tue Jan 05, 2010 6:38 pm

Hi,

thanks for quick answer.

Ok, I went under Monitor, but annot find Target "VMGuest Virtual Machine". :oops:
krowczynski
Enthusiast
 
Posts: 62
Liked: never
Joined: Thu Dec 17, 2009 6:34 pm
Full Name: Arkadiusz Krowczynski

Re: VMware Tools Heartbeat Status

Veeam Logoby krowczynski » Tue Jan 05, 2010 6:51 pm

Ok found it thx guys for your help.
I chnaged it from 1200 to 2400, now I will see if any warnings came up.
krowczynski
Enthusiast
 
Posts: 62
Liked: never
Joined: Thu Dec 17, 2009 6:34 pm
Full Name: Arkadiusz Krowczynski

Re: VMware Tools Heartbeat Status

Veeam Logoby Alec King » Tue Jan 05, 2010 7:04 pm

Thanks Krowczynski!

BTW - Make sure you change the CorrelationInterval for BOTH Monitors - "to Red", and "to Yellow". They should match.

Let us know how it goes,
Cheers
Alec
Alec King
Veeam Software
 
Posts: 700
Liked: 116 times
Joined: Sun Jan 01, 2006 1:01 am

Re: VMware Tools Heartbeat Status

Veeam Logoby krowczynski » Tue Jan 26, 2010 12:59 pm

Hi,

I disabled this alarm in my SCOM.
I also think that will soon came out an update for this problem as written in this post under VMware community.

http://communities.vmware.com/message/1460065#1460065
krowczynski
Enthusiast
 
Posts: 62
Liked: never
Joined: Thu Dec 17, 2009 6:34 pm
Full Name: Arkadiusz Krowczynski

PreviousNext

Return to Veeam Management Pack for Microsoft System Center



Who is online

Users browsing this forum: No registered users and 2 guests