Since v6.5 upgrade "Host hardware sensor status changed"

[AlainRussell]

I've seen this error on all our hosts since the 6.5 upgrade. Pre 6.5 this error could be cleared by entering maintenance mode and then exiting. Since v6.5 I cannot clear this error .. I've tried the maintenance mode fix, restarting the host, restarting Veeam one but nothing seems to clear the error.

Does anyone have any ideas?
Thanks
Alain

[Vitaliy S.]

Hi Alain,

Could you please tell me what sensor has changed its status? What does vSphere Client show for the same sensor?

Thanks!

[AlainRussell]

Hi Vitaliy,

VCenter is not showing any errors currently (All Hardware sensors are showing as Normal), Veeam One is showing all the sensors for each host as "equal Unknown".

Thanks
Alain

[Vitaliy S.]

In v6.5 we have switched to a more advanced host hardware collection method (CIM over XML) and for some sensors VMware doesn't have any state and returns “unknown” state, which leads to triggered alarms. Please remove the rule that triggers "Host hardware sensor status changed" alarm when sensor changes its state to “Unknown”. Hope this helps!

[AlainRussell]

Thanks, that sorted it!

[ch@switch-it.se]

Hi!

I have the same problem with HP ProLiant servers since the upgrade to 6.5.

Is it an ok (safe) solution to disable that rule? I have had issues earlier with servers that randomly reported all sensors with status unkown. I mean there is a risk that a faulty server reports status unknown. If all sensors are reporting unknown, then you don't know if server hardware are ok. It could be stuck in status unkown and you think everything is good.

/Christian

[Vitaliy S.]

Hi Christian,

Mostly you should be concerned about sensors which have either warning or error health state, but your point is also valid. Since "Unknown" status is retrieved for both, sensors without any state and sensors that have just switched to this state, I'm not sure there is way to differentiate these two conditions.

As a workaround you may want to leave this rule, but assign a different severity level (info) when host hardware sensor switches to "unknown" status.

Thank you!

[Vitaly.Mzokov]

In addition to Vitaliy's post...Prior to v6.5 we've seen that some hardware sensors didn't return the correct status due to a known VMware issue - when host's sensors display wrong status via vCenter Server's Managed Objects Browser (MOB). In v6.5 we are using more accurate hardware monitoring protocol CIM-XML which completely eliminates such behavior. Be aware that vSphere Client uses the same protocol to obtain sensors' states.

Moreover, since Veeam ONE v6.5 the hardware monitoring became more granular, and you might notice way more sensors available in v6.5 in comparison to v6.0. But, as Vitaliy S. has already mentioned, for some sensors VMware doesn't have any state and returns just “unknown” status. Please keep in mind that the correct state for "well-known sensors" is always available via CIM-XML (which is utilized by Veeam ONE by default).
If NOT, then you have to talk to VMware and get this issue addressed ASAP.

Following this logic, in most cases "unknown" state will be returned for sensors that VMware doesn't have any information about. Thus, answering your question, Christian - "Yes, I think this is a safe solution to disable the rule after upgrade"!

Hope this helps!
-VM.

[ch@switch-it.se]

Thank you guys for the info/comments

I will disable the rule.

/Christian

[cffit]

We also have current HP Proliant DL360 G6 and G7 Servers that showed fine until the 6.5 upgrade, and now they all show this error. This is pretty mainstream server hardware and it surprises me that the solution is to disable the monitor.

If I do call into VMWare support, what do I ask of them since their monitoring is showing up fine in their client?

[Vitaliy S.]

Christensen,

If hardware sensor in vSphere Client has a "healthy" state while Monitor Client reports the opposite for the same sensor, then you need to contact our support team for further investigation. Your issue might not be related to the problem discussed in this thread.

Thanks!

[cffit]

I take that back, in VMware these sensors also show as unknown too.

Here's the issue though, I have the HP branded ESXi 5 install, and just to make sure, I installed the HP Offline bundle with all the newest CIM agents. The sensors still show as "Unknown" in VMWare. I had someone I know that has HP, Dell and IBM servers check theirs out. They say that on all three platforms they have the same sensors show up as "Unknown" too. And he has the HP branded ESXi 5 install on them.

With this said, if those three server brands have this issue, I can't imagine any server out there where all the sensors communicate and report properly. Is there any case where they do all work? And VEEAM 6.5 is happy and clear (without manually disabling this alarm)? Has anyone had any luck with this?

[Vitaliy S.]

The hosts in my lab are not branded ones, so I don't have a rich set of hardware sensors to monitor, only few critical sensors are available and all of them communicate their status properly.

[cffit]

Christian Hellemar, did you have any luck with your HP Proliant servers in getting them to work? Or did you just remove the rule that monitors them?

[ijefferson]

Hi

I'm having the same issue since upgrading, on a Dell T410 the following sensors are reporting as unknown:

Code: Select all

Sensor "Disk Drive Bay 1 Cable SAS A 0: Connected - Assert" equal Unknown 
Sensor "Disk Drive Bay 1 Cable SAS B 0: Connected - Assert" equal Unknown 
Sensor "System Board 1 Status 0: Connected - Assert" equal Unknown 
Sensor "DIMM_A1" equal Unknown Sensor "DIMM_A2" equal Unknown 
Sensor "DIMM_A3" equal Unknown Sensor "DIMM_A4" equal Unknown 
Sensor "DIMM_B1" equal Unknown Sensor "DIMM_B2" equal Unknown 
Sensor "DIMM_B3" equal Unknown Sensor "DIMM_B4" equal Unknown 
Sensor "vmnic0" equal Unknown Sensor "vmnic1" equal Unknown 
Sensor "vmnic2" equal Unknown Sensor "vmnic3" equal Unknown 
Sensor "System Board 1 Power Optimized 0: OEM - Assert" equal Unknown 
Sensor "System Board 1 Power Optimized 0: Unknown - Deassert" equal Unknown 
Sensor "Disk Drive Bay 1 Presence 0: Absent - Deassert" equal Unknown 
Sensor "Disk Drive Bay 1 Presence 0: Present - Assert" equal Unknown 
Sensor "Power Supply 1 Presence 0: Absent - Deassert" equal Unknown 
Sensor "Power Supply 1 Presence 0: Present - Assert" equal Unknown 
Sensor "Power Supply 2 Presence 0: Absent - Deassert" equal Unknown 
Sensor "Power Supply 2 Presence 0: Present - Assert" equal Unknown 
Sensor "Processor 1 Presence 0: Absent - Deassert" equal Unknown 
Sensor "Processor 1 Presence 0: Present - Assert" equal Unknown 
Sensor "Processor 2 Presence 0: Absent - Deassert" equal Unknown 
Sensor "Processor 2 Presence 0: Present - Assert" equal Unknown 
Sensor "System Board 1 HEATSINK PRES 0: Absent - Deassert" equal Unknown 
Sensor "System Board 1 HEATSINK PRES 0: Present - Assert" equal Unknown 
Sensor "System Board 1 iDRAC6 Ent PRES 0: Absent - Assert" equal Unknown 
Sensor "System Board 1 iDRAC6 Ent PRES 0: Present - Deassert" equal Unknown 
Sensor "CPU1 Level-1 Cache DeviceID:1041.01" equal Unknown 
Sensor "CPU1 Level-2 Cache DeviceID:1042.01" equal Unknown 
Sensor "CPU1 Level-3 Cache DeviceID:1043.01" equal Unknown
Sensor "CPU2 Level-1 Cache DeviceID:1041.02" equal Unknown 
Sensor "CPU2 Level-2 Cache DeviceID:1042.02" equal Unknown 
Sensor "CPU2 Level-3 C

[Vitaliy S.]

Hi Ian, can you please check hardware status for these sensors in vSphere Client? Also please look through this topic for the possible explanation of this situation. Thanks!

[cffit]

ijefferson, I have the exact same sensors reporting as unkown and have been unable to resolve my issue. If you look in vCenter you will proabably see these listed as unknown as well. Since you are using Dell and I am using HP, and we both have this issue, I can't understand what server platform would actually work and not report back unkown on so many sensors.

So far it sounds like the only two other people that have had this issue have disabled/removed the rule and nobody has actually got things to work properly. Has anyone on this post been able to get the sensors to show up properly? If so, what was your path to the solution? I understand that VEEAM is just pulling what vCenter is able to report on, so technically the issue is there and not in VEEAM, but at the same time if HP and Dell systems aren't getting vCenter the sensor information then that tells me there must be very few server vendors that can. If this is the case then is it really a good idea to have VEEAM use this information by default?

[ijefferson]

To be fair, a lot of them I'm suprised it's picked up. It seems to go into a lot of details.

Attached screenshot of my sensors window (presuming it's the right bit, I'm still fairly new to this) and the unknown ones all seem to appear where there is no status rightly or wrongly (should there be a status for vmnic's for example?)



Any advice appreciated.

Edit: The only one which is correctly reported as Unknown in Veeam and is Unknown in vSphere is from my other host correctly reporting the 2nd Power supply as unknown because it doesn't currently have one.

So if I'm just supposed to ignore these sensors from the rule, could you guide me through the quickest way to do this? Also presumably if vSphere isn't returning a status to Veeam, then logically Veeam should ignore the status too rather then flagging as unknown?

Thanks again.

(editted again due to me awful mistyping!)

[Vitaliy S.]

Hi Ian,

Thanks for the screenshot, currently our R&D team is researching on how to workaround this issue and do not trigger this alarm for "stateless" sensors (for ex. vmnic and so on). This will allow you to be notified when one of the sensors that should have the state goes to "unknown" state.

Thanks!

[ijefferson]

Thanks Vitaliy

So just sit tight and wait for the time being?

Cheers

Ian

[Vitaliy S.]

Yes, once I have more details, I will update this topic.

[cffit]

Thanks Vitaliy. That's what I was hoping to hear. And thanks VEEAM support for being responsive to the customer!

[connelp]

Just to add, I have just upgraded to 6.5 and am having exactly the same issue.

What's strange for me is that if look in VSphere I can see the details of, for example, all my DIMMs but if I look in VEEAM One it tells me they are 'unknown'.

[Vitaliy S.]

The status should be the same in both Veeam ONE Monitor Client and vSphere Client, if you see a different picture, then you should open the support case.

[cffit]

connelp,

IF you expand the details of the DIMM (click the + sign next to each one) it will show serveral things like Capacity, Speed, Memory Type, Status, Chip Type and Number. The Status is what reads "Unknown" for me and thus triggers the alarm in VEEAM. What do you see listed after "Status" here? I suspect it is "Unknown". At least it's that way for me.

[monkeymagic79]

Hi All

I am also getting this for my company and one of our customers who we have updated. Veeam Monitor shows an unknown status but in vCentre we see all the details and status as green.

I was watching this forum as Vitaliy mentioned R&D were looking into this and he would report back. are you now saying I need to raise a support case for this issue. If the answer is yes does this suggest you now have a fix?

Many thanks

[Vitaliy S.]

I believe there are two different issues discussed in this thread, let me sum them up to avoid confusion:

1. Hardware sensor alarm is triggered for "stateless" sensors, the sensors that do not have any state in vSphere Client (see screenshot from Ian Jefferson, vmnic sensors).
2. Some sensors might have "unknown" state in the status column in Monitor Client while having status "Normal" in vSphere Client (and vice versa).

Our R&D team has found the way to resolve the first issue with "stateless" sensors and is already working on the hotfix. It's better to open the support case to be notified when this fix becomes available.

As to the second one, then there is no fix for that, and this issue should be investigated by our and VMware support teams.

[connelp]

connelp,

IF you expand the details of the DIMM (click the + sign next to each one) it will show serveral things like Capacity, Speed, Memory Type, Status, Chip Type and Number. The Status is what reads "Unknown" for me and thus triggers the alarm in VEEAM. What do you see listed after "Status" here? I suspect it is "Unknown". At least it's that way for me.

Ah yes, that's the case exactly. I'm running Proliant G5s BTW.

[cffit]

Any update from VEEAM on whether this is going to be addressed in any upcoming patch? I read that the B&R product has the first patch coming out later this week and just wondered if the same was true for this product. Thanks!

[Vitaliy S.]

Hi Christensen,

Yes, we are also going to release a patch for Veeam ONE that should address this issue. This patch should be available very soon (the time-frame depends on the pilot deployment results).

Thanks!