VMware Tools Heartbeat Status

[krowczynski]

Hi,

I am new to nworks and have a quick question.
I get several times a day the messege "nworks:VMware Tools Heartbeat Status change to Yellow" after 10 or 15 Minutes the warning went awwy.
Looking on my ESX host I don not see something critcal throght this vms.

Have anyone else this issue or should I set the threhold to more than 10 Minutes?

Thanks for answers.

[Vitaliy S.]

Hello Arkadiusz,

Could you check that all VMware Tools are all up to date, also check the physical memory allocation for the virtual machines that raise that alerts?

Basically, we get all the info from VMware API. The alert is generated if the tools are not responding for 5-10 minutes as far as I remember, but sometimes VC may experinence some latency issues and still telling via API that the tools gone red/yellow.

We get this message from the Collector which tracks the change in VMwareVMStats.guestHeartbeatStatus field in VI API.

The trigger is related to VMware monitors and tell us that status. You should look at VM\Summary Tab\General\VMware Tools for full information, also all additional information you can see at VMwareVMStats.guestHeartbeatStatus using VMware MOB (Managed Object Browser)

To check that field status please follow these instructions:

1. connect to your VC via ip address using browser
2. then choose browse objects managed by VirtualCenter (it is located at the bottom, on the right corner)
3. after that choose "content ServiceContent: content"
4. then click to "rootFolder ManagedObjectReference:Folder:" your Value
5. next please choose "childEntity ManagedObjectReference:ManagedEntity" your Value

If you try to refresh the page you'll see that the status might change,that is actually the source where we get the info.

Thank you

[krowczynski]

Well under the mob following I see.
Under summary all Tools are ok.

[Vitaliy S.]

Arkadiusz,

I've deleted the mob output cause it's hard to read it here and it's too long, however have you had a chance to see this field changing while you refresh the page? Do you often see that alert coming? Do you see that for particular VMs or for all of them? You can also log a support ticket (support@veeam.com) with us, probably there will be additional information in the log files.

Thank you!

[krowczynski]

Hello Vitaly,

just opened a servicerequest to support@veeam.com
Let's wait what happen.

I will give you feddback as far as I know something more.

[krowczynski]

Hi,

get following answer from support.

Hello Arkadiusz,

The matter is that we do not generate this alert, we just trigger it when the status of VMWare tools in MOB changes. You may observe for some time the status of tools in MOB, refreshing the page and you will see that VMWare tools status changes. So, this should equip you well to open a support case with VMWare and find out what`s wrong with VMWare tools. Feel free to put us in CC for this.

Have I to monitor the status on the MOB page of my vcenter server?

[Vitaliy S.]

Arkadiusz,

Actually, we do monitor this field on all the changes via MOB by ourselves, so if you see any alert about VMware Tools status it means that VMwareVMStats.guestHeartbeatStatus has changed. Basically, there is no need in doing it by yourself (refreshing the page), you were advised to do that just to prove that we do not generate this alert about VMware Tools Status, cause we just inform you upon everything we get through VMware API by means of our Management Pack.
Generally, if you see this alert for only specifc VMs you may try to find what's in common between them, on the other hand if you see these alerts for random VMs, I also believe that VMware could give you more information on that and shed some light on the strange Tools behaviour.

By the way, what version of MP are you currently using? I'm asking this, cause we've increased the correlation Interval for VMware Tools Heartbeat in our latest update - Version 5.0.3 Build 248.

Thank you.

[krowczynski]

Hi,

I am running the version of Management center version 5.0.2.194.
Should I am doing an upgrade.

Thanks for answer.

[Vitaliy S.]

Arkadiusz,

Yes, please do. After you are upgraded please monitor your virtual environment for a couple of days, and if you still see those alerts coming, please let me know.

Thank you.

[krowczynski]

Hi,

what is the best way to update all the things?
Thank you in advanced!!

[Vitaliy S.]

Essentially, you should re-install all the components one by one, and then re-import the MP itself to SCOM server.

[krowczynski]

OK, a few minutes ago I did the following.
I have imported the MgMt Pack to scom, installed all components one by one, rebooted the server and after that I could not connect to Management Center --> http://localhost:4430/.

Luckily I created a snapshot and couls role back to old version.

So the last state ist that I have only imported the MgMt Pack to the SCOm and all other products are based on the old version.

[Vitaliy S.]

Arkadiusz,

For detailed assistance please refer to our support team so they could assist you with the upgrade procedure, we can do a webex session if needed.

[krowczynski]

Ok I've done so, thanks for you very fast answers!
Great job!

[krowczynski]

OK found the error and installation went fine.
Now I don't see the whole _Datacenter topology in my SCOM.

Have I to wait a few minutes until the scom collect all data once again??

[Vitaliy S.]

Arkadiusz,

That's right, you can perform WBEMTEST (as it is described in our Guides) on the Collector Machine to see whether collector established communication with your hosts or not. Usually it takes few minutes, depends on the size of your virtual infrastructure

[sameerdave]

We have seen these alerts in loads in our environment.
And finally Veeam found that there is a bug which they will update in their next version.

So currently you will have to live with these alerts as there is no work around. Sadly, I receive around 2k alerts per day, OR we have to disable the VM monitoring, which would not make sense..

Wish we knew about this buy before we bought the product..

[Alec King]

Hi Sameer Dave,

I am sorry you are receiving a lot of alerts; and we will of course assist you to get the best nworks configuration for your environment.

However I have to point out that if there IS a "bug" - it is in Virtual Center, not in the nworks MP.
All that our Collector does, is check the 'VMware Tools Heartbeat Status' for each VM, exactly as the VMware API publishes it. If this status is 'Red', or 'Yellow' - then we generate an alert.

This 'VMTools Heartbeat Status' has been shown in the past to be somewhat unstable/unreliable - *as published in Virtual Center, not in our product!*
Sometimes when we check the vCenter API, we get a status of Red or Yellow, for no apparent reason. Within a few minutes, the status returns to Green - again for no apparent reason. This is why we introduced a Timer in the MP Monitor, so that we would only alert if the Status *remained* Red/Yellow for 2 intervals, rather than alert immediately.

It seems that in some environments, the default (2 intervals = 10 minutes) is not enough - so you can either
* increase the interval by modifying the parameter 'CorrelationInterval' on the Monitor.
* disable the VMTools Heartbeat Monitor (not this does not disable all "VM monitoring" - only this particular state event. All VM performance data, all other VC events, including power changes etc, will still be captured)

So - we are happy to assist you, as I said, to get the best value from the nworks MP. But the 'bottom line' is that we are always dependent on the data that is published by the VMware API, whether that is performance data, events, or VM state-change as in this case.

I hope that clears up the issue, please let me know.

For *anyone* receiving these 'Heartbeat' alerts - please feed back to this thread if you are experiencing this issue, and whether increasing CorrelationInterval helps. This is the feedback we need, we can take it to VMware, and work with them to find a solution for this instability in the Heartbeat value we get from the VI-API.

Thanks!
Alec

Senior Product Manager
nworks

[krowczynski]

Hello,

where can I increase the CorrelationInterval for the hearbeat alert please?

[sameerdave]

Thanks Alec,
From what i understood, that this would be fixed in the 5.5 version, as in this case we are receiving old alerts from the vc.

Anyways, in ref to your override, I see the Correlations Interval which is at 600, and Correlation count which is at 1. Whats the difference between count and interval?
So i guess the correlations interval is in seconds, and we can change that settings from 600 to 1200 or 900..

Krowczynski - You can find these counters by going to Authoring-Monitors - Serach for "VMWare Tools Heartbeat" and then clicking on override settings for the monitors..

[Alec King]

Hi SameerDave,

CorrelationInterval is the Timer period before an alert will be raised. CorrelationCount is the number of events required to 'reset' the timer.

So two examples for this VM Heartbeat Monitor could be -

1. This example does NOT generate an alert:

nworks requests Heartbeat Status from the VI-API, and gets "Red"
nworks throws Heartbeat=Red event
Event triggers Timer start for Monitor (CorrelationInterval = 600 seconds)
After 300 seconds, nworks requests Heartbeat Status from the VI-API, and gets "Green"
nworks throws Heartbeat=Green event
As CorrelationCount=1, this 1 event resets the Monitor.
No alert is fired.

2. This example WILL generate an alert:

nworks requests Heartbeat Status from the VI-API, and gets "Red"
nworks throws Heartbeat=Red event
Event triggers Timer start for Monitor (CorrelationInterval = 600 seconds)
After 300 seconds, nworks requests Heartbeat Status from the VI-API, and still gets "Red"
After 600 seconds, nworks requests Heartbeat Status from the VI-API, and still gets "Red"
CorrelationInterval Timer has expired at 600s.
Alert is generated.

So, to answer your question - yes you can override that CorrelationInterval value to 900, 1200, 1500....some multiple of your baseline nworks Collector interval (default 300s)
Note - do not override CorrelationCOUNT. We only throw an event for State-change; we don't throw repeating events. So CorrelationCount should stay as "1".
Increasing that CorrelationInterval out-of-box is one change we will probably make in MP v5.5.
And at the same time we will continue working with VMware to try and track down the reasons for the instability in the value reported by the VI-API...this is the root of the problem.

Thanks!
Alec

[sameerdave]

Thanks Alec for the detailed explanation.

In ref to the VMware Tools hearbeat, we can change the correlationinterval, that is fine. But i do get quite a huge number of alerts for "VM CPU Alarm changed to Red". We get like 3-10 every minute. While talking to support, they said these are old events that are getting generated. Would you have any workaround for this? I don't think these have any override parameters, but if you could help us here, that would be great.

Thanks
Sameer

[Alec King]

Hi Sameer,

Regarding the 'VM CPU Alarm' alerts -
These alerts are based on the Alarms as configured in Virtual Center. As with Heartbeat, there is an event that is thrown when the CPU Usage changes to "Red" (as Virtual Center is configured to see it). And there is an MP Monitor that responds to those events.

These Alarms run in parallel, at the same time as the Monitor for VM CPU usage that is included in the MP. Our CPU Usage Monitor is configured in SCOM to respond to the actual value of cpuUsedPct metric.

So, really you have 2 different alerts that are watching the same thing - VM CPU.
One threshold (for VM CPU Alarm event) is configured in VC. That's "VM CPU Alarm changed to Red/Yellow" as seen in SCOM.
One threshold (for VM CPU Usage %) is configured in SCOM. That's "VM CPU Usage has exceeded threshold" as seen in SCOM.

I would advise you to use the MP CPU Usage Monitors only. They are configured within SCOM, so it is much easier to set overrides. And the alerts are more informative - they will give you the actual value in %, rather than just saying "to Red" or "to Yellow".

So you could just disable the Alarm for VM CPU - either by disabling the MP Monitors (for Red, and Yellow) - or by disabling the Alarm itself, within Virtual Center. it is just providing duplicate information. Our MP "CPU Usage threshold" Monitor is more useful.

Regarding what support said, that you are receiving old (cached?) events that are repeating..? Can you give me the Support Ticket number? I will look into that for you.

Cheers,
Alec

[krowczynski]

Hello Alec,

where can I find the Authoring-Monitors to change the setting for heartbeat messeges. I get to many of them...

[sameerdave]

Thanks Alec. The support ticket is ID#517901. These are alerts we are getting from old logs.

Krowczynski - Yes, its under the SCOM console - Authoring - Expand Managed Pack Objects and Monitors. There do a find for VMWare Tools Heartbeat, and you would see the monitors there. You can right click on these monitors and create an override.

[Alec King]

Hi Krowczynski,

In SCOM Console, you find Authoring section in the navigation buttons in bottom left. (you need to be SCOM Administrator with Authoring privileges)
When you are in Authoring, find Management Pack Objects - Monitors in the tree view.
Then you can search for 'nworks' using the 'Look For' box. (make sure you clear any existing Scope)
Find the Target "VMGuest Virtual Machine" in the main view, expand it and and you will see the Heartbeat Monitors (Red, and Yellow) and you can change them there in the usual way (right click and override)

Cheers
Alec

[krowczynski]

Hi,

thanks for quick answer.

Ok, I went under Monitor, but annot find Target "VMGuest Virtual Machine".

[krowczynski]

Ok found it thx guys for your help.
I chnaged it from 1200 to 2400, now I will see if any warnings came up.

[Alec King]

Thanks Krowczynski!

BTW - Make sure you change the CorrelationInterval for BOTH Monitors - "to Red", and "to Yellow". They should match.

Let us know how it goes,
Cheers
Alec

[krowczynski]

Hi,

I disabled this alarm in my SCOM.
I also think that will soon came out an update for this problem as written in this post under VMware community.

http://communities.vmware.com/message/1460065#1460065