v5 false alarm "Host hardware sensor status changed"

[alubel]

We have updated 2 of our ESX hosts and firmware to latest version a few weeks ago (4.1 260247) along with monitor v5. These are identically configured sun X4150's. within the vsphere console both hosts report "hardware status" is normal. Within the monitor v5 console, one of the hosts has an alarm going off every 10 minutes with alarm detail stating "Sensor VMware Rollup Health State equal Unknown" and in the hardware tab showing "VMware rollup health state" with a question mark. I have tried resetting the sensors on the affected host with no resolution.

Is there anymore information I could provide that would help in resolving this?

[Vitaliy S.]

Hello Andy, it would be helpful, if you could look through a VeeamDCS.log file and see if there are any errors related to the second ESX host. Besides, I believe we may also check the sensor's status with Managed Object Browser to compare the behavior... Thanks!

[Vitaliy S.]

Andy, please use the instructions I've sent you via PM to check the sensor state with MOB. I would appreciate if you could post your results back here. Thanks!

[alubel]

both are exactly the same, thanks for the advice - any other idea what could cause this?

https://host1/mob/?moid=ha-host&doPath= ... SensorInfo

Code: Select all

aseUnits	string	""
currentReading	long	0
dynamicProperty	DynamicProperty[]	Unset
dynamicType	string	Unset
healthState	ElementDescription	
NAME	TYPE	VALUE
dynamicProperty	DynamicProperty[]	Unset
dynamicType	string	Unset
key	string	"green"
label	string	"Green"
summary	string	"Sensor is operating under normal conditions"
name	string	"VMware Rollup Health State"
rateUnits	string	Unset
sensorType	string	"system"
unitModifier	int	0

https://host2/mob/?moid=ha-host&doPath= ... SensorInfo

Code: Select all

baseUnits	string	""
currentReading	long	0
dynamicProperty	DynamicProperty[]	Unset
dynamicType	string	Unset
healthState	ElementDescription	
NAME	TYPE	VALUE
dynamicProperty	DynamicProperty[]	Unset
dynamicType	string	Unset
key	string	"green"
label	string	"Green"
summary	string	"Sensor is operating under normal conditions"
name	string	"VMware Rollup Health State"
rateUnits	string	Unset
sensorType	string	"system"
unitModifier	int	0

[Vitaliy S.]

For future readers: this is vCenter/ESX hardware interop issue that can be confirmed by browsing vCenter Server's MOB, which also displays unknown state for the VMware Rollup Health state hardware sensor. This is where Veeam Monitor obtains its data from.

[jchampion01]

I am experiencing the same thing on my Dell R710 Servers 2 of the 3 are just fine, but my first one keeps having the Host hardware sensor status changed, for VMware Rollup Health and Drive Bay X Parity Check in progress. In the VI client the hardware status is fine on this drive array and host. Also physically there is little to no activty on the drives and the PERC card has not alerted me of any drive/array parity checks.
I have check the MOB for the VMware Rollup Health Status and it shows, current reading 0, Unset, and green status. Also the sensors for the hard drives and they raid array all show "green" status in MOB and VI CLient. Veeam Monitor keeps reporting this as an issue every 10 minutes.

Was there a resolution to this or, just live with false alarms? Which seems to defeat the purpose of monitoring...
Am i missing something here.

[Vitaliy S.]

Hello Jeremy, could you please clarify whether you were using vCenter Sever MOB page or ESX host's one? The information about VMware Rollup Health State might be different in those cases.

[jchampion01]

I was using the MOB page on the ESXi host, not the vCenter Server. I will check the vCenter Server MOB page and see if I can find my way around to check the sensor settings and values.

[Vitaliy S.]

Yes, please do, because the issue we had with Andy related to vCenter server displaying incorrect status for this particular sensor, even though ESX host was reporting correctly.

[jchampion01]

Could you tell me what links / objects i need to click on to drill down into that information, or the url path?

[jchampion01]

Alright i think i found it on the vCenter Server MOB page.
It looks like this host (esxi01) on this vCenter Server MOB does show "Yellow" as the status for "VMware Rollup Health" and for the the RAID Array drives under "Drive X Parity Check In Progress - Assert".

VMware Rollup Health
https://shvcenter01/mob/?moid=host%2d20 ... SensorInfo

Code: Select all

[11] HostNumericSensorInfo Name Type Value 
baseUnits string "" 
currentReading long 0 
dynamicProperty DynamicProperty[] Unset 
dynamicType string Unset 
healthState ElementDescription Name Type Value 
dynamicProperty DynamicProperty[] Unset 
dynamicType string Unset 
key string "yellow" 
label string "Yellow" 
summary string "Sensor is operating under conditions that are non-critical" 
 
name string "VMware Rollup Health State" 
rateUnits string Unset 
sensorType string "system" 
unitModifier int 0 
 

Drive Parity Check
https://shvcenter01/mob/?moid=host%2d20 ... SensorInfo

Code: Select all

[131] HostNumericSensorInfo Name Type Value 
baseUnits string "" 
currentReading long 0 
dynamicProperty DynamicProperty[] Unset 
dynamicType string Unset 
healthState ElementDescription Name Type Value 
dynamicProperty DynamicProperty[] Unset 
dynamicType string Unset 
key string "yellow" 
label string "Yellow" 
summary string "Sensor is operating under conditions that are non-critical" 
 
name string "Disk Drive Bay 1 Drive 4: Parity Check In Progress - Assert" 
rateUnits string Unset 
sensorType string "Storage" 
unitModifier int 0 
 

[Vitaliy S.]

Jeremy, that's good that we were able to nail it. The resolution provided by Andy was the following, you may also try it, should help:

Issue resolved by:
1. disable EVC on cluster.
2. vmotion machines over to secondary node.
3. maintenance mode / evict "faulted" node from cluster.
4. remove "faulted" node from vcenter.
5. log into "faulted" node via ILOM, restart management agents
6. re-add node back into vcenter.
7. re-add node to cluster.
8. re-enable EVC.

Thanks,
Vitaliy

[Jpurdy79]

I'm receiving the same alarms since my upgrade to 4.1. How do I access the sensor states in the Managed Object Browser?

[Vitaliy S.]

Hello Jonathan, here you go:

1. Connect to your vCenter Server via Internet browser - https://vCenter_Server_name/mob/
2. Choose content ServiceContent content
3. Navigate to rootFolder ManagedObjectReference:Folder ha-folder-root (ha-folder-root)
4. Find childEntity ManagedObjectReference:ManagedEntity[] ha-datacenter (ha-datacenter)
5. Choose hostFolder ManagedObjectReference:Folder ha-folder-host (host)
6. Then find childEntity ManagedObjectReference:ManagedEntity[] ha-compute-res
7. Choose host ManagedObjectReference:HostSystem[] ha-host
8. Finally, go to runtime HostRuntimeInfo runtime
9. On doing that, navigate to healthSystemRuntime HealthSystemRuntime healthSystemRuntime
10. Click on systemHealthInfo HostSystemHealthInfo systemHealthInfo
11. Find numericSensorInfo HostNumericSensorInfo[] numericSensorInfo
12. Compare the value in the following string name string "VMware Rollup Health State"

Hope it helps! If you need any assistance with this, please contact our support team directly.

[Jpurdy79]

Thanks. I'm going to open a ticket with VMware support. What I'm seeing is that when viewing the "VMware Rollup Health State" string from the vCenter MOB, the key is "unknown" and the summary is "Cannot report on the current health state of the element". Whereas, if I veiw the "VMware Rollup Health State" string from the Host MOB, the key is "green" and the summary is "Sensor is operating under normal conditions".

[Vitaliy S.]

Jonathan, you're welcome. I would appreciate it if you could update this topic with VMware's official resolution.

[Jpurdy79]

Will do.

[axs@moffatt.com]

A quick note on this thread. I had the same issue as described above and removing and rejoining the host did fix this issue but seems to have broken my replication jobs completely. What I have learned now is that if you reboot the host server without first putting it into MM it will show this fault. The solution for me was to put the server into MM, reboot, bring server out of MM and the "viola" no more error. I wish I would have known this before I removed and rejoined the host as it looks like I am going to need to rebuild and start over on all of my replication jobs...

[aqualityplace]

We found simply putting the host into maitenance mode and then exit maitenance mode cleared the error

[Vitaliy S.]

Michael, thanks for finding time to update this topic.

[RumataRus]

[Merged with existing discussion]

I often see alarm "Host hardware sensor status changed" warning though there is no errors/warnings in vCenter about hardware status.
Is it possible to reduce the sensitivity of host hardware sensors in Veeam Monitor?

[Vitaliy S.]

Dmitry,

This alarm is being triggered by a certain condition defined in the alarm properties. You can use Reporting tab to adjust the triggering frequency. However, If I were you I would try to investigate why you have this status changed for the host.

[RumataRus]

Dmitry,
However, If I were you I would try to investigate why you have this status changed for the host.

How? vCenter Alarms and Logs are clean.
BTW, if I clear ALL alarms in Virtual Infrastructure I receive "Host hardware sensor status changed" warning in about 10 seconds on two of my three hosts.
If I do it again, I receive the same again on the same hosts. It is stable.

[RumataRus]

We found simply putting the host into maintenance mode and then exit maintenance mode cleared the error

Wow! It really helps!

[Fiskepudding]

Great post. Altho it is old, I just want to chip in my exprience.

"VMware Rollup Health State" reported: "Cannot report on the current health state of the element"

This was when browsing the MOB on the vCetner server
When browsing directly on the HOST it reported: "Sensor is operating under normal conditions"

We took the HOST to manintanance mode and rebooted it. As suggested.
Took it out of maintanance mode, waited 5 minutes and looked in vCenter's MOB.
Still the same issue, "Cannot report on the current health state of the element".

Waited even longer, logged of MOB and back on. Nothing changed.

Finaly we rebooted the vCenter server, and all was good when it came back online.
So a tip migt be to also restart vCenter.

[Vitaliy S.]

Thanks for sharing your resolution with the community.