Since v6.5 upgrade "Host hardware sensor status changed"

[rimvydukas]

Hi,

I've updated my vcenter infrastructure to patch 1 today. And all of our servers started to show "Host hardware status changed" warnings:( All our servers are Proliant DL380 ones.

[Vitaliy S.]

Hello,

What does vSphere Client show for the hardware sensors in question?

Thanks!

[rimvydukas]

Client shows the same as other users clients were showing. I have warnings only with Proc cache sensors. Read policy shows Unknown:/

[Vitaliy S.]

Then you need to disable these sensors on the Hardware Tab in Monitor client if you do not want to receive these warning messages any longer.

[rimvydukas]

Yes, I did that already:) Can you explain why were there no errors prior this patch?

[Vitaliy S.]

If this was caused by ESX(i) host update, then most likely this status had different status prior installing the patch and that is why you didn't receive any notifications.

[rimvydukas]

I've updated only Veeam One and not the ESXi servers,

[Vitaliy S.]

Oh, but initially you said the opposite Anyway, we have re-configured this rule to fire alarms for unknown state of the sensors, because if sensor has this state, this definitely should be investigated. You can always revert to the previous behavior by removing trigger for "unknown" state in the alarm configuration. Thanks!

[rimvydukas]

Vitaly,

I'm sorry for my unclear first post:( I wanted to say that I've updated Veeam One to patch 1 and started to experience such behaviour. So I've suppressed all alarms for all of unknown sensors. But it seems that I still have problems. Sensors about proc level cache appeared as unchecked after some time on several ESXi servers so I have status warnings again. I checked to suppress alarms one more time and after some time the same situation - sensors got unchecked on some servers:/

[Vitaliy S.]

Can you please let our technical team take a look at your configuration to verify that everything is suppressed correctly? You may also remove this rule from the alarm configuration (via Alarm Management) to revert to prior patch 1 behavior.

[rimvydukas]

Hi,

I checked the following checkboxes and now these checkboxes got unchecked again.



And I noticed one more thing - one time when checkboxes are unchecked I can see sensors as Proc1, unknown cache sensor, unknown cache sensor, unknown cache sensor, Proc2, unknown cache sensor, unknown cache sensor, unknown cache sensor and other time I can see these sensors as Proc1, Proc2, unknown cache sensor, unknown cache sensor, unknown cache sensor, unknown cache sensor, unknown cache sensor, unknown cache sensor.

Maybe because of that they get unchecked???

[Vitaliy S.]

Hello,

I doubt that. If you select to suppress alarms, these checkboxes should stay unchecked no matter what happens. Can you please open a support case on this issue?

Thanks!

[rimvydukas]

Ok, I opened the case. But I noted one more thing today.

I've got an e-mail today from Veeam monitoring that on one esxi server problem with unknown sensors was resolved. I went to monitoring console, checked processor sensors and saw that there were no unknown cache sensors at all, neither checked, nor unchecked. I've got another warning from the same esxi server after some time about unknown sensors. Whent to console, checked processor sensors and saw that all unknown cache sensors are there and all of them are unchecked. So I started to suspect the following thing - I supress alarms for these sensors, esxi server comes to normal state. These sensors disappear after some time - esxi is still in ok state. Sensors appear again and veeam thinks that there are new sensors appeared and throws warning. What do you think?

[Vitaliy S.]

Hmm...sensors cannot just disappear and then show up again. I wonder whether it is related to collection jobs or ESXi server itself. Do you see the same behavior for all hosts? Our debug logs will definitely point out the problem. Can you please tell me your ticket number?

[rimvydukas]

Maybe its strange for you, but I saw this with my eyes:) As for now I can see warnings only on ESXi servers which are in HA clusters (sensors appearing as unchecked). My ticked is 00258726. I was offered to disable specific monitoring rule at all, but I want to be able supress alerts for unknown sensors and not disable this rule.

[Vitaliy S.]

Thanks for posting your ticket number, I will follow up with the support team on this.

[AlexBingen]

Hi all,

I just upgraded VEEAM One from 6.5 to V7. Right after the upgrade (that went without issues) I noticed all three ESX hosts created a warning simultaneously:

Sensor "Battery on Controller 0 Charging ..." equal Warning
Sensor "CPU1 Level-1 Cache DeviceID:1041.01" equal Unknown
Sensor "CPU1 Level-2 Cache DeviceID:1042.01" equal Unknown
Sensor "CPU1 Level-3 Cache DeviceID:1043.01" equal Unknown
Sensor "CPU2 Level-1 Cache DeviceID:1041.02" equal Unknown
Sensor "CPU2 Level-2 Cache DeviceID:1042.02" equal Unknown
Sensor "CPU2 Level-3 Cache DeviceID:1043.02" equal Unknown
Sensor "VMware Rollup Health State" equal Unknown

However the hardware sensors in VMWare show no problems. Due to the fact this happens on all three hosts exactly the same time, right after the upgrade i'm thinking this could be a bug.
Specs Evironment:
ESXi Host: DELL Poweredge R610, 2 socket Intel Xeon 5504, 98gb RAM
VMWare: Vsphere 4.1.0 Essentials
Storage DELL SAN Equallogic PS4000E + PS4100X storage pool.

Has anyone seen this before?
Thanks for your input.

[hafesta]

Same problem here since the update, except the batteryproblem. There are only question marks shown at the hardwarestatus-tab of the hosts in veeam one monitor.

ESXi Hosts: Intel S2600GZ, 2x Intel E5-2640, 256GB -> ESXi 5.1.0, 1157734
vShere 5.1.0 Build 1123961

[Vitaliy S.]

Please check these sensors in vSphere Client, pretty sure you will see the same unknown status for them.

[Terrum]

I have three Dell PE620 servers each running ESXi 5.5, all joined in a cluster, vCenter version is 5.5.
In Veeam One Monitor v7 I got one alarm (warning) similar across all three hosts:

Sensor "CPU1 Level-1 Cache DeviceID:1041.01" equal Unknown
Sensor "CPU1 Level-2 Cache DeviceID:1042.01" equal Unknown
Sensor "CPU1 Level-3 Cache DeviceID:1043.01" equal Unknown
Sensor "CPU2 Level-1 Cache DeviceID:1041.02" equal Unknown
Sensor "CPU2 Level-2 Cache DeviceID:1042.02" equal Unknown
Sensor "CPU2 Level-3 Cache DeviceID:1043.02" equal Unknown

One of hosts also got "Sensor "VMware Rollup Health State" equal Unknown" added in the list and that's the most interesting one. I can suppress warnings like "CPU1 Level-1 Cache..." by selecting Suppress Alarm checkbox for each item with a question mark under Hardware/Processors, but I couldn't find any for "VMware Rollup Health State"...

I found VMware kb that seems relevant (though it's for vSphere v5.1 and below - http://kb.vmware.com/selfservice/micros ... Id=2020665). I followed the steps suggested to verify if my host is running as expected and I confirmed that the "VMware Rollup Health State" on the server returns "Sensor is operating under normal conditions".

So, the host is running without any issues, and in vCenter Client I see no alerts or warning from any of sensors. Why then "Sensor "VMware Rollup Health State" equal Unknown" warning comes up in Veeam One?

[Vitaliy S.]

Hello,

I can suppress warnings like "CPU1 Level-1 Cache..." by selecting Suppress Alarm checkbox for each item with a question mark under Hardware/Processors, but I couldn't find any for "VMware Rollup Health State"...

This sensor should be located under System category, please check it out.

So, the host is running without any issues, and in vCenter Client I see no alerts or warning from any of sensors. Why then "Sensor "VMware Rollup Health State" equal Unknown" warning comes up in Veeam One?

That's not expected, all sensors should have the same state as in vSphere Client. Can you please contact our support team and let me know you case number?

Thank you!

[Terrum]

Hi Vitaliy. I found "VMware Rollup Health State" under System category, and its status is "Green" - so far everything is the same as in vSphere Client, but I still have "Sensor "VMware Rollup Health State" equal Unknown" under Alarms, which just doesn't make any sense...

My case number: 00483875

Thank you!
Anton

[Vitaliy S.]

Have you tried resolving this alarm manually?

[Terrum]

No, I didn't try to resolve the alarm. However, it seems that changing Assign severity from Warning to Informational for the Sensor state "Unknown" under Alarm settings/Rules/Hardware sensor/Sensor state helped in my case, as the message "Sensor "VMware Rollup Health State" equal Unknown" was gone and didn't come back even after I changed severity back to Warning. I used info from this Veeam kb - http://www.veeam.com/kb_articles.html/kb1703

[Vitaliy S.]

Hi Anton,

The reason why it didn't come back is that the state was reported as healthy in vSphere Client and I believe resolving the alarm manually would give the same effect.

Thanks for the update!

[Terrum]

Hi Anton,

The reason why it didn't come back is that the state was reported as healthy in vSphere Client and I believe resolving the alarm manually would give the same effect.

Thanks for the update!

In this situation I expect that Veeam One clears the alarm automatically - same way how vSphere Client does it.

That sounds odd to have to manually resolve an alarm which is not relevant anymore. I think this behavior makes these warnings/alarms very confusing, at least visually - how can I differentiate a real (true) alarm from the one that doesn't represent an existing issue anymore (false) ? Was that fixed in the recent R2 patch for Veeam One?

[Vitaliy S.]

Yes, the default behavior is resolve all alarms automatically. The reason why I have proposed to do that was just to check that this sensor is collected properly.

Yes, hardware sensors monitoring logic has been adjusted to address this.

[Terrum]

Yes, the default behavior is resolve all alarms automatically. The reason why I have proposed to do that was just to check that this sensor is collected properly.

Yes, hardware sensors monitoring logic has been adjusted to address this.

Vitaliy,
I'm on the latest Veeam One v7 R2 build (version 7.0.0.912) and I'm seeing the following alarm in Veeam One on one of ESXi hosts:

Description
Sensor "VMware Rollup Health State" equal Warning
Sensor "System Board 1 Drive 0 2: Parity Check In Progress - Assert" equal Warning
Sensor "System Board 1 Drive 0 3: Parity Check In Progress - Assert" equal Warning

A few days ago I had a real disk parity check event on this host and I saw corresponding Yellow warning in vCenter Client while the check was in progress. However, the check finished a day ago, the "Parity Check" sensors in vCenter Client changed status from Yellow to Green, so everything should be back to normal, but according to Veeam One the check is still in progress and when I look under Hardware tab I see an exclamation mark on disk parity check sensor! How is that possible if Veeam One should reflect the same sensors status as vCenter Client?

After installing R2 patch, I tried to resolve the alarm manually, suppress and release it, even rebooted vCenter server and the server where Veeam One runs, but the alarm is still coming back every time. I'm guessing something is wrong with my Veeam One installation. My support case number is 00483875.

Thanks!

[Vitaliy S.]

I will ask our dev team to take a look at this on Monday, thanks for posting your ticket number. All sensors should have the same view in both Veeam ONE and vSphere Client.

[Vitaliy S.]

This issue has been fixed in the patch for Veeam ONE v7R2. You can download it using the same KB link > http://forums.veeam.com/viewtopic.php?f=28&t=19698