Hardware sensors collection failure after V8 upgrade

[lwalker]

Wondering if anyone can help out with this one.

We just upgraded a few of our clients to Veeam Backup and replication 8 as well as Veeam ONE 8

We are running ESXi v5.5
-I have verified there is no firewall between my Veeam ONE VM and my ESXi Host
-I have verified the CIM Server service is running on my ESXi Host
-I have tried different user accounts to connect to ESXi, Currently I am using the root user, still getting the error.

I have went through the following KB Article and all criteria has been met..
www.veeam.com/kb1568

This is happening on multiple VeeamONE installations for my clients, all using ESXi v5.5. Some with Vcenter some without Vcenter.

[Vitaliy S.]

Hi Lynn,

Can you please tell me if you had any similar issues while being on version 7? Do you have a case opened, so I could discuss this issue with the R&D team?

Thank you!

[VladV]

I haven't open a case yet, and I will, but would like to add that we are also seeing this issue which hasn't manifested itself in v7. The warnings are:

Code: Select all

Sensor "Fan Device 1 Fan 1 Present 0" equal Unknown
Sensor "Fan Device 2 Fan 2 Present 0" equal Unknown
Sensor "Fan Device 3 Fan 3 Present 0" equal Unknown
Sensor "Fan Device 4 Fan 4 Present 0" equal Unknown
Sensor "Fan Device 5 Fan 5 Present 0" equal Unknown
Sensor "Power Supply 1 PS1 Fan Fail 0" equal Unknown
Sensor "Power Supply 2 PS2 Fan Fail 0" equal Unknown
Sensor "Power Supply 1 PS1 Status 0: Presence detected" equal Unknown
Sensor "Power Supply 2 PS2 Status 0: Presence detected" equal Unknown
Sensor "System Board 1 Button 0: Power Button pressed" equal Unknown
Sensor "System Board 1 Button 0: Reset Button pressed" equal Unknown
Sensor "I/O Module 1 IO Mod Presence 0" equal Unknown
Sensor "I/O Module 1 SAS Mod Presence 0" equal Unknown
Sensor "Processor 1 CATERR 0" equal Unknown
Sensor "Processor 1 CPU Missing 0" equal Unknown
Sensor "Processor 1 P1 ERR2 0" equal Unknown
Sensor "Processor 1 P1 MSID Mismatch 0" equal Unknown
Sensor "Processor 1 P1 Status 0: Presence detected" equal Unknown
Sensor "Processor 2 P2 ERR2 0" equal Unknown
Sensor "Processor 2 P2 MSID Mismatch 0" equal Unknown
Sensor "Processor 2 P2 Status 0: Presence detected" equal Unknown
Sensor "Drive Backplane 1 HDD 0 Status 0: Drive Present" equal Unknown
Sensor "Drive Backplane 1 HDD 0 Status 0: Hot Spare" equal Unknown
Sensor "Drive Backplane 1 HDD 0 Status 0: Unknown" equal Unknown
Sensor "Drive Backplane 1 HDD 1 Status 0: Unknown" equal Unknown
Sensor "Drive Backplane 1 HDD 2 Status 0: Drive Present" equal Unknown
Sensor "Drive Backplane 1 HDD 2 Status 0: Hot Spare" equal Unknown
Sensor "Drive Backplane 1 HDD 2 Status 0: Unknown" equal Unknown
Sensor "Drive Backplane 1 HDD 3 Status 0: Drive Present" equal Unknown
Sensor "Drive Backplane 1 HDD 3 Status 0: Hot Spare" equal Unknown
Sensor "Drive Backplane 1 HDD 3 Status 0: Unknown" equal Unknown
Sensor "Drive Backplane 1 HDD 4 Status 0: Drive Present" equal Unknown
Sensor "Drive Backplane 1 HDD 4 Status 0: Hot Spare" equal Unknown
Sensor "Drive Backplane 1 HDD 4 Status 0: Unknown" equal Unknown
Sensor "Drive Backplane 1 HDD 5 Status 0: Drive Present" equal Unknown
Sensor "Drive Backplane 1 HDD 5 Status 0: Hot Spare" equal Unknown
Sensor "Drive Backplane 1 HDD 5 Status 0: Unknown" equal Unknown
Sensor "Drive Backplane 1 HDD 6 Status 0: Drive Present" equal Unknown
Sensor "Drive Backplane 1 HDD 6 Status 0: Hot Spare" equal Unknown
Sensor "Drive Backplane 1 HDD 6 Status 0: Unknown" equal Unknown
Sensor "Drive Backplane 1 HDD 7 Status 0: Drive Present" equa

All our ESXi hosts are included in VMWare HCL.

[Vitaliy S.]

Vlad, I believe this is a different issue, since you do have collection job working, but these sensors do not have any state in the vSphere Client and thus reported as unknown.

[VladV]

That's the thing, vpshere client sees the HW info fine. For example:



The case nr. is 00673118

[Vitaliy S.]

Ok, thanks! I have forwarded this to our dev team.

[lwalker]

Case number 00672756

I havent seen any of the same problems with version 7. All was working in v7, no alerts on any of my v7 boxes. i upgraded Veeam Backup and Veeam ONE to v8 when I saw these problems, all systems had v7 installed before.

[Vitaliy S.]

Lynn, in version 7 we've been using MOB and CIM-Smash protocol to monitor hardware sensors, and since there were issues with MOB method due to VMware API we have decided to make a switch to CIM-Smash way of monitoring ESXi host hardware sensors. Let's see what our support team can say after reviewing the debug logs.

[lwalker]

Im actually wondering if we've got something else going on here.

V7, while I havent been getting alerts for CIM integration errors, when I go into the hardware tab in Veeam ONE for my host, it doesnt look like its pulling any data.

Looks like it may not be v8 that is the problem, but maybe an added alert/error added into v8 warning me of this CIM connection problem.

I've got a call into support, just waiting for the tech to call me back.

THANKS

[Vitaliy S.]

Yes, we have specifically changed our triggers to monitor CIM service availability on the monitored host. One thing I would recommend to check while waiting our technical guy to call you back - can you ping your hosts directly via FQDN from the Veeam ONE server? Just want to make sure that vCenter Server is not related to this problem.

[mcwill]

Case number 00674415

Just to add that we are also seeing this issue, V8 can only poll 1 out of our 3 hosts (all identical) whereas V7 could see all 3.

V8 server can successfully ping all 3 hosts by FQDN.

Regards,
Iain

[Vitaliy S.]

Ok, thanks for the case ID, will discuss it with the devs, could be CIM Smash service unavailability issue...

[macgyver]

hello

same issue here with V8

testlab : no VCenter

veeam-one v8 Server (fresh install ) + 2 x esxi hosts all identical ( hardware , ESXi Version 5.5.0,2143827, fresh install )

same firewall settings
same credentials
same services running ( cim )

first host shows Hardware !
second host shows no hardware ( empty tab ) !


Test : adding above Hosts to VCenter Server > Hardware is fine on both Hosts.

ideas welcome !

Regards,

mac

[Vitaliy S.]

Hi Mac,

What about CIM-Smash service on the second host? Is it running?

Thanks!

[macgyver]

Hi Vitaly,

do you mean :

/etc/init.d # /etc/init.d/sfcbd-watchdog test
Checking core components of sfcbd are running...
Component sfcb-ProviderMa is running.
Component sfcb-HTTP-Daemo is running.
Component sfcb-HTTPS-Daem is running.
All components are running.


The CIM service is up and running



I compared both serves / services - exactly the same.

How / where can I restart only CIM-Smash in ESXi 5.5 ?

I rebooted both servers - no change

Both Servers were freshly installed, same time and date - same vsphere image - so why should the servers behave differently ?

Regards

Mac

[Vitaliy S.]

In the case from Iain, the issue seems to be related to names resolution (our support team has find out it after reviewing job logs). Let's see what our dev team can say after looking through your debug information.

[mcwill]

Yes, on our system the cause was the affected vSphere hosts; Configuration -> DNS and Routing -> Host Identification -> Name = localhost

Changing the above so it matched the DNS entry solved the problem.

Regards,
Iain

[macgyver]

Hi,

issue resolved with veeam Support

thanks

Regards

Mac

[northrundicandus]

It appears that the new CIM-SMASH protocol method of Hardware Sensors collection now requires DNS resolution of the individual ESX hosts to work correctly. Previously with v7 I only needed the vCenter server to be resolvable, and not for the individual ESX hosts managed by that vCenter.

Does this mean that customers who add their ESX servers to vCenter by IP address are out of luck when it comes to this kind of monitoring?

[macgyver]

hello,

no your are not out of luck,

we have the same setup ( IP )

just make sure the veeamone server is able to resolve ( reverse lookup ) the dns names ( FQDN ) of the esxi host(s) in your vcenter(s) !

you can do this by static dns entries in your dns server ( more flexible ) or just use your local hosts file on the veeam one server = manual adding ip addresses and FQDN,
if later IP addresses or names change please don't forget to change them here as well.

This solved our problems with the hardware tab / display in Veeam One 8 ( and Veeam One 7 as well )

Regards

Mac

[Vitaliy S.]

Previously with v7 I only needed the vCenter server to be resolvable, and not for the individual ESX hosts managed by that vCenter.

Does this mean that customers who add their ESX servers to vCenter by IP address are out of luck when it comes to this kind of monitoring?

Previously we were using vSphere API to monitor hardware sensors, but when we have raised one issue with VMware regarding this method, they have advised us not to use this method and use CIM-SMASH.

The case with ESXi servers added by IP address, should not be an issue. Let me check it with the dev team tomorrow.

[northrundicandus]

The case with ESXi servers added by IP address, should not be an issue. Let me check it with the dev team tomorrow.

I determined this is an issue if the customer defined a hostname and DNS domain in the ESX install, but added the ESX server to vCenter with an IP address instead. Adding a hosts file entry on the Veeam Monitor server is a quick fix for this until the customer can remove and add the server back to vCenter using a hostname instead.

[Vitaliy S.]

Is there any reason to add ESXi host via IP address instead of its hostname? Just trying to confirm that I didn't miss any use case.

[northrundicandus]

I've seen this a lot in the field with small vSphere deployoments. Customers don't know they need to add a DNS record for their ESX hosts to their AD integrated DNS servers, so they end up adding their ESX hosts to vCenter by IP address instead.

[Vitaliy S.]

I never remember my host IP address, but I do know its hostname all the time.

Ok, we will discuss this scenario internally and I will update the thread with the requirements for CIM-smash monitoring feature.

[macgyver]

Hi Vitaliy,

this would be great !

i never thought of the solution veeamsupport came up with.

on the one hand Veeamone/CIM-Smash can use ip-addresses for all servers but on the other hand for backwards communication to the infrastructure it must be able to resolve the fqdn of the esxi servers for hardware monitoring

no chance of working with ip addresses !

maybe you can explain this to me

it's by design ?

Regards

Mac

[firmdale]

Case # 00675767

Same issue here. Adding DNS records resolved issue with ONE connecting to hosts but still seeing below:

Description
Sensor "Memory" equal Warning
Sensor "PROC 2 DIMM 4: Physical Memory Module Socket=04 on ProcessorBoard=2" equal Warning
Sensor "System Board 8 Memory: Uncorrectable ECC" equal Alert
Sensor "System Chassis 1 UID Light" equal Unknown
Sensor "System Chassis 2 Sys. Health LED" equal Unknown
Sensor "Proc 1 Level-1 Cache" equal Unknown
Sensor "Proc 1 Level-2 Cache" equal Unknown
Sensor "Proc 1 Level-3 Cache" equal Unknown
Sensor "Proc 2 Level-1 Cache" equal Unknown
Sensor "Proc 2 Level-2 Cache" equal Unknown
Sensor "Proc 2 Level-3 Cache" equal Unknown
Sensor "Disk or Disk Bay 1 C1 P1I Bay 1: Drive Present" equal Unknown
Sensor "Disk or Disk Bay 1 C1 P1I Bay 1: Hot Spare" equal Unknown
Sensor "Disk or Disk Bay 2 C1 P1I Bay 2: Drive Present" equal Unknown
Sensor "Disk or Disk Bay 2 C1 P1I Bay 2: Hot Spare" equal Unknown
Sensor "Disk or Disk Bay 3 C1 P1I Bay 3: Drive Present" equal Unknown
Sensor "Disk or Disk Bay 3 C1 P1I Bay 3: Hot Spare" equal Unknown
Sensor "Disk or Disk Bay 4 C1 P1I Bay 4: Drive Present" equal Unknown
Sensor "Disk or Disk Bay 4 C1 P1I Bay 4: Hot Spare" equal Unknown
Sensor "Disk or Disk Bay 5 C1 P2I Bay 5: Drive Present" equal Unknown
Sensor "Disk or Disk Bay 5 C1 P2I Bay 5: Hot Spare" equal Unknown
Sensor "Disk or Disk Bay 6 C1 P2I Bay 6: Drive Present" equal Unknown
Sensor "Disk or Disk Bay 6 C1 P2I Bay 6: Hot Spare" equal Unknown
Sensor "Disk or Disk Bay 7 C1 P2I Bay 7: Drive Present" equal Unknown
Sensor "Disk or Disk Bay 7 C1 P2I Bay 7: Hot Spare" equal Unknown
Sensor "Disk or Disk Bay 8 C1 P2I Bay 8: Drive Present" equal Unknown
Sensor "Disk or Disk Bay 8 C1 P2I Bay 8: Hot Spare" equal Unknown
Sensor "Power Supply 1 Power Supply 1: Presence detected" equal Unknown
Sensor "Power Supply 2 Power Supply 2: Presence detected" equal Unknown

[Vitaliy S.]

These sensors do not health state in the vSphere Client (this column) for them is empty, that is why they are reported as unknown. We are working on the workaround for this behavior. Thanks!

[Vitaliy S.]

i never thought of the solution veeamsupport came up with.

on the one hand Veeamone/CIM-Smash can use ip-addresses for all servers but on the other hand for backwards communication to the infrastructure it must be able to resolve the fqdn of the esxi servers for hardware monitoring

no chance of working with ip addresses !

maybe you can explain this to me

it's by design ?

If you have ESXi hosts added by IP address, you can still make it work by following either of these ways:

1. adjust "hosts" files on the Veeam ONE server in order to resolve host name
2. do the required changes in the vCenter Server config (Configuration -> DNS and Routing -> Host Identification)

The explanation of this requirement is pretty straight forward. When the API call is sent to the host to get the data for hardware sensors, ESXi host returns it's name (configured in the option 2 above) and then this name is used to establish the connection tunnel for data retrieval. If Veeam ONE cannot resolve the name of the host, then it cannot connect to it, thus hardware sensors data is not available.

In version 7 we were using different mechanism of getting this data, and when we have discovered an issue with the API, have decided to switch to a more reliable way of monitoring hardware senors.

Hope this makes sense!

[macgyver]

Hi Vitaliy,

all clear as day
If I'd known about this earlier

Thanks

Regards

Mac