Monitoring and reporting for Veeam Data Platform
lwalker
Novice
Posts: 4
Liked: 1 time
Joined: Nov 06, 2014 9:53 pm
Full Name: Lynn Walker
Contact:

Hardware sensors collection failure after V8 upgrade

Post by lwalker » 1 person likes this post

Wondering if anyone can help out with this one.

We just upgraded a few of our clients to Veeam Backup and replication 8 as well as Veeam ONE 8

We are running ESXi v5.5
-I have verified there is no firewall between my Veeam ONE VM and my ESXi Host
-I have verified the CIM Server service is running on my ESXi Host
-I have tried different user accounts to connect to ESXi, Currently I am using the root user, still getting the error.

I have went through the following KB Article and all criteria has been met..
www.veeam.com/kb1568

This is happening on multiple VeeamONE installations for my clients, all using ESXi v5.5. Some with Vcenter some without Vcenter.
Vitaliy S.
VP, Product Management
Posts: 27375
Liked: 2799 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: Hardware sensors collection failure after V8 upgrade

Post by Vitaliy S. »

Hi Lynn,

Can you please tell me if you had any similar issues while being on version 7? Do you have a case opened, so I could discuss this issue with the R&D team?

Thank you!
VladV
Expert
Posts: 224
Liked: 25 times
Joined: Apr 30, 2013 7:38 am
Full Name: Vlad Valeriu Velciu
Contact:

Re: Hardware sensors collection failure after V8 upgrade

Post by VladV »

I haven't open a case yet, and I will, but would like to add that we are also seeing this issue which hasn't manifested itself in v7. The warnings are:

Code: Select all

Sensor "Fan Device 1 Fan 1 Present 0" equal Unknown
Sensor "Fan Device 2 Fan 2 Present 0" equal Unknown
Sensor "Fan Device 3 Fan 3 Present 0" equal Unknown
Sensor "Fan Device 4 Fan 4 Present 0" equal Unknown
Sensor "Fan Device 5 Fan 5 Present 0" equal Unknown
Sensor "Power Supply 1 PS1 Fan Fail 0" equal Unknown
Sensor "Power Supply 2 PS2 Fan Fail 0" equal Unknown
Sensor "Power Supply 1 PS1 Status 0: Presence detected" equal Unknown
Sensor "Power Supply 2 PS2 Status 0: Presence detected" equal Unknown
Sensor "System Board 1 Button 0: Power Button pressed" equal Unknown
Sensor "System Board 1 Button 0: Reset Button pressed" equal Unknown
Sensor "I/O Module 1 IO Mod Presence 0" equal Unknown
Sensor "I/O Module 1 SAS Mod Presence 0" equal Unknown
Sensor "Processor 1 CATERR 0" equal Unknown
Sensor "Processor 1 CPU Missing 0" equal Unknown
Sensor "Processor 1 P1 ERR2 0" equal Unknown
Sensor "Processor 1 P1 MSID Mismatch 0" equal Unknown
Sensor "Processor 1 P1 Status 0: Presence detected" equal Unknown
Sensor "Processor 2 P2 ERR2 0" equal Unknown
Sensor "Processor 2 P2 MSID Mismatch 0" equal Unknown
Sensor "Processor 2 P2 Status 0: Presence detected" equal Unknown
Sensor "Drive Backplane 1 HDD 0 Status 0: Drive Present" equal Unknown
Sensor "Drive Backplane 1 HDD 0 Status 0: Hot Spare" equal Unknown
Sensor "Drive Backplane 1 HDD 0 Status 0: Unknown" equal Unknown
Sensor "Drive Backplane 1 HDD 1 Status 0: Unknown" equal Unknown
Sensor "Drive Backplane 1 HDD 2 Status 0: Drive Present" equal Unknown
Sensor "Drive Backplane 1 HDD 2 Status 0: Hot Spare" equal Unknown
Sensor "Drive Backplane 1 HDD 2 Status 0: Unknown" equal Unknown
Sensor "Drive Backplane 1 HDD 3 Status 0: Drive Present" equal Unknown
Sensor "Drive Backplane 1 HDD 3 Status 0: Hot Spare" equal Unknown
Sensor "Drive Backplane 1 HDD 3 Status 0: Unknown" equal Unknown
Sensor "Drive Backplane 1 HDD 4 Status 0: Drive Present" equal Unknown
Sensor "Drive Backplane 1 HDD 4 Status 0: Hot Spare" equal Unknown
Sensor "Drive Backplane 1 HDD 4 Status 0: Unknown" equal Unknown
Sensor "Drive Backplane 1 HDD 5 Status 0: Drive Present" equal Unknown
Sensor "Drive Backplane 1 HDD 5 Status 0: Hot Spare" equal Unknown
Sensor "Drive Backplane 1 HDD 5 Status 0: Unknown" equal Unknown
Sensor "Drive Backplane 1 HDD 6 Status 0: Drive Present" equal Unknown
Sensor "Drive Backplane 1 HDD 6 Status 0: Hot Spare" equal Unknown
Sensor "Drive Backplane 1 HDD 6 Status 0: Unknown" equal Unknown
Sensor "Drive Backplane 1 HDD 7 Status 0: Drive Present" equa
All our ESXi hosts are included in VMWare HCL.
Vitaliy S.
VP, Product Management
Posts: 27375
Liked: 2799 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: Hardware sensors collection failure after V8 upgrade

Post by Vitaliy S. »

Vlad, I believe this is a different issue, since you do have collection job working, but these sensors do not have any state in the vSphere Client and thus reported as unknown.
VladV
Expert
Posts: 224
Liked: 25 times
Joined: Apr 30, 2013 7:38 am
Full Name: Vlad Valeriu Velciu
Contact:

Re: Hardware sensors collection failure after V8 upgrade

Post by VladV »

That's the thing, vpshere client sees the HW info fine. For example:

Image

The case nr. is 00673118
Vitaliy S.
VP, Product Management
Posts: 27375
Liked: 2799 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: Hardware sensors collection failure after V8 upgrade

Post by Vitaliy S. »

Ok, thanks! I have forwarded this to our dev team.
lwalker
Novice
Posts: 4
Liked: 1 time
Joined: Nov 06, 2014 9:53 pm
Full Name: Lynn Walker
Contact:

Re: Hardware sensors collection failure after V8 upgrade

Post by lwalker »

Case number 00672756

I havent seen any of the same problems with version 7. All was working in v7, no alerts on any of my v7 boxes. i upgraded Veeam Backup and Veeam ONE to v8 when I saw these problems, all systems had v7 installed before.
Vitaliy S.
VP, Product Management
Posts: 27375
Liked: 2799 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: Hardware sensors collection failure after V8 upgrade

Post by Vitaliy S. »

Lynn, in version 7 we've been using MOB and CIM-Smash protocol to monitor hardware sensors, and since there were issues with MOB method due to VMware API we have decided to make a switch to CIM-Smash way of monitoring ESXi host hardware sensors. Let's see what our support team can say after reviewing the debug logs.
lwalker
Novice
Posts: 4
Liked: 1 time
Joined: Nov 06, 2014 9:53 pm
Full Name: Lynn Walker
Contact:

Re: Hardware sensors collection failure after V8 upgrade

Post by lwalker »

Im actually wondering if we've got something else going on here.

V7, while I havent been getting alerts for CIM integration errors, when I go into the hardware tab in Veeam ONE for my host, it doesnt look like its pulling any data.

Looks like it may not be v8 that is the problem, but maybe an added alert/error added into v8 warning me of this CIM connection problem.

I've got a call into support, just waiting for the tech to call me back.

THANKS
Vitaliy S.
VP, Product Management
Posts: 27375
Liked: 2799 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: Hardware sensors collection failure after V8 upgrade

Post by Vitaliy S. »

Yes, we have specifically changed our triggers to monitor CIM service availability on the monitored host. One thing I would recommend to check while waiting our technical guy to call you back - can you ping your hosts directly via FQDN from the Veeam ONE server? Just want to make sure that vCenter Server is not related to this problem.
mcwill
Enthusiast
Posts: 64
Liked: 10 times
Joined: Jan 16, 2010 9:47 am
Full Name: Iain McWilliams
Contact:

Re: Hardware sensors collection failure after V8 upgrade

Post by mcwill »

Case number 00674415

Just to add that we are also seeing this issue, V8 can only poll 1 out of our 3 hosts (all identical) whereas V7 could see all 3.

V8 server can successfully ping all 3 hosts by FQDN.

Regards,
Iain
Vitaliy S.
VP, Product Management
Posts: 27375
Liked: 2799 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: Hardware sensors collection failure after V8 upgrade

Post by Vitaliy S. »

Ok, thanks for the case ID, will discuss it with the devs, could be CIM Smash service unavailability issue...
macgyver
Influencer
Posts: 15
Liked: 2 times
Joined: Jan 20, 2011 6:03 pm
Contact:

Re: Hardware sensors collection failure after V8 upgrade

Post by macgyver »

hello

same issue here with V8

testlab : no VCenter

veeam-one v8 Server (fresh install ) + 2 x esxi hosts all identical ( hardware , ESXi Version 5.5.0,2143827, fresh install )

same firewall settings
same credentials
same services running ( cim )

first host shows Hardware !
second host shows no hardware ( empty tab ) !


Test : adding above Hosts to VCenter Server > Hardware is fine on both Hosts.

ideas welcome !

Regards,

mac
Vitaliy S.
VP, Product Management
Posts: 27375
Liked: 2799 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: Hardware sensors collection failure after V8 upgrade

Post by Vitaliy S. »

Hi Mac,

What about CIM-Smash service on the second host? Is it running?

Thanks!
macgyver
Influencer
Posts: 15
Liked: 2 times
Joined: Jan 20, 2011 6:03 pm
Contact:

Re: Hardware sensors collection failure after V8 upgrade

Post by macgyver »

Hi Vitaly,

do you mean :

/etc/init.d # /etc/init.d/sfcbd-watchdog test
Checking core components of sfcbd are running...
Component sfcb-ProviderMa is running.
Component sfcb-HTTP-Daemo is running.
Component sfcb-HTTPS-Daem is running.
All components are running.


The CIM service is up and running



I compared both serves / services - exactly the same.

How / where can I restart only CIM-Smash in ESXi 5.5 ?

I rebooted both servers - no change

Both Servers were freshly installed, same time and date - same vsphere image - so why should the servers behave differently ?

Regards

Mac
Vitaliy S.
VP, Product Management
Posts: 27375
Liked: 2799 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: Hardware sensors collection failure after V8 upgrade

Post by Vitaliy S. »

In the case from Iain, the issue seems to be related to names resolution (our support team has find out it after reviewing job logs). Let's see what our dev team can say after looking through your debug information.
mcwill
Enthusiast
Posts: 64
Liked: 10 times
Joined: Jan 16, 2010 9:47 am
Full Name: Iain McWilliams
Contact:

Re: Hardware sensors collection failure after V8 upgrade

Post by mcwill » 3 people like this post

Yes, on our system the cause was the affected vSphere hosts; Configuration -> DNS and Routing -> Host Identification -> Name = localhost

Changing the above so it matched the DNS entry solved the problem.

Regards,
Iain
macgyver
Influencer
Posts: 15
Liked: 2 times
Joined: Jan 20, 2011 6:03 pm
Contact:

Re: Hardware sensors collection failure after V8 upgrade

Post by macgyver »

Hi,

issue resolved with veeam Support

thanks

Regards

Mac
northrundicandus
Service Provider
Posts: 14
Liked: 3 times
Joined: Aug 11, 2011 9:01 pm
Full Name: Joe Ross
Contact:

Re: Hardware sensors collection failure after V8 upgrade

Post by northrundicandus » 1 person likes this post

It appears that the new CIM-SMASH protocol method of Hardware Sensors collection now requires DNS resolution of the individual ESX hosts to work correctly. Previously with v7 I only needed the vCenter server to be resolvable, and not for the individual ESX hosts managed by that vCenter.

Does this mean that customers who add their ESX servers to vCenter by IP address are out of luck when it comes to this kind of monitoring?
macgyver
Influencer
Posts: 15
Liked: 2 times
Joined: Jan 20, 2011 6:03 pm
Contact:

Re: Hardware sensors collection failure after V8 upgrade

Post by macgyver »

hello,

no your are not out of luck,

we have the same setup ( IP )

just make sure the veeamone server is able to resolve ( reverse lookup ) the dns names ( FQDN ) of the esxi host(s) in your vcenter(s) !

you can do this by static dns entries in your dns server ( more flexible ) or just use your local hosts file on the veeam one server = manual adding ip addresses and FQDN,
if later IP addresses or names change please don't forget to change them here as well.

This solved our problems with the hardware tab / display in Veeam One 8 ( and Veeam One 7 as well )

Regards

Mac
Vitaliy S.
VP, Product Management
Posts: 27375
Liked: 2799 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: Hardware sensors collection failure after V8 upgrade

Post by Vitaliy S. »

northrundicandus wrote:Previously with v7 I only needed the vCenter server to be resolvable, and not for the individual ESX hosts managed by that vCenter.

Does this mean that customers who add their ESX servers to vCenter by IP address are out of luck when it comes to this kind of monitoring?
Previously we were using vSphere API to monitor hardware sensors, but when we have raised one issue with VMware regarding this method, they have advised us not to use this method and use CIM-SMASH.

The case with ESXi servers added by IP address, should not be an issue. Let me check it with the dev team tomorrow.
northrundicandus
Service Provider
Posts: 14
Liked: 3 times
Joined: Aug 11, 2011 9:01 pm
Full Name: Joe Ross
Contact:

Re: Hardware sensors collection failure after V8 upgrade

Post by northrundicandus » 1 person likes this post

Vitaliy S. wrote: The case with ESXi servers added by IP address, should not be an issue. Let me check it with the dev team tomorrow.
I determined this is an issue if the customer defined a hostname and DNS domain in the ESX install, but added the ESX server to vCenter with an IP address instead. Adding a hosts file entry on the Veeam Monitor server is a quick fix for this until the customer can remove and add the server back to vCenter using a hostname instead.
Vitaliy S.
VP, Product Management
Posts: 27375
Liked: 2799 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: Hardware sensors collection failure after V8 upgrade

Post by Vitaliy S. »

Is there any reason to add ESXi host via IP address instead of its hostname? Just trying to confirm that I didn't miss any use case.
northrundicandus
Service Provider
Posts: 14
Liked: 3 times
Joined: Aug 11, 2011 9:01 pm
Full Name: Joe Ross
Contact:

Re: Hardware sensors collection failure after V8 upgrade

Post by northrundicandus »

I've seen this a lot in the field with small vSphere deployoments. Customers don't know they need to add a DNS record for their ESX hosts to their AD integrated DNS servers, so they end up adding their ESX hosts to vCenter by IP address instead.
Vitaliy S.
VP, Product Management
Posts: 27375
Liked: 2799 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: Hardware sensors collection failure after V8 upgrade

Post by Vitaliy S. »

I never remember my host IP address, but I do know its hostname all the time. :)

Ok, we will discuss this scenario internally and I will update the thread with the requirements for CIM-smash monitoring feature.
macgyver
Influencer
Posts: 15
Liked: 2 times
Joined: Jan 20, 2011 6:03 pm
Contact:

Re: Hardware sensors collection failure after V8 upgrade

Post by macgyver »

Hi Vitaliy,

this would be great !

i never thought of the solution veeamsupport came up with.

on the one hand Veeamone/CIM-Smash can use ip-addresses for all servers but on the other hand for backwards communication to the infrastructure it must be able to resolve the fqdn of the esxi servers for hardware monitoring

no chance of working with ip addresses !

maybe you can explain this to me ;-)

it's by design ? :-)

Regards

Mac
firmdale
Novice
Posts: 9
Liked: 1 time
Joined: Sep 09, 2010 3:32 pm
Full Name: Firmdale Hotels IT Department
Contact:

Re: Hardware sensors collection failure after V8 upgrade

Post by firmdale »

Case # 00675767

Same issue here. Adding DNS records resolved issue with ONE connecting to hosts but still seeing below:

Description
Sensor "Memory" equal Warning
Sensor "PROC 2 DIMM 4: Physical Memory Module Socket=04 on ProcessorBoard=2" equal Warning
Sensor "System Board 8 Memory: Uncorrectable ECC" equal Alert
Sensor "System Chassis 1 UID Light" equal Unknown
Sensor "System Chassis 2 Sys. Health LED" equal Unknown
Sensor "Proc 1 Level-1 Cache" equal Unknown
Sensor "Proc 1 Level-2 Cache" equal Unknown
Sensor "Proc 1 Level-3 Cache" equal Unknown
Sensor "Proc 2 Level-1 Cache" equal Unknown
Sensor "Proc 2 Level-2 Cache" equal Unknown
Sensor "Proc 2 Level-3 Cache" equal Unknown
Sensor "Disk or Disk Bay 1 C1 P1I Bay 1: Drive Present" equal Unknown
Sensor "Disk or Disk Bay 1 C1 P1I Bay 1: Hot Spare" equal Unknown
Sensor "Disk or Disk Bay 2 C1 P1I Bay 2: Drive Present" equal Unknown
Sensor "Disk or Disk Bay 2 C1 P1I Bay 2: Hot Spare" equal Unknown
Sensor "Disk or Disk Bay 3 C1 P1I Bay 3: Drive Present" equal Unknown
Sensor "Disk or Disk Bay 3 C1 P1I Bay 3: Hot Spare" equal Unknown
Sensor "Disk or Disk Bay 4 C1 P1I Bay 4: Drive Present" equal Unknown
Sensor "Disk or Disk Bay 4 C1 P1I Bay 4: Hot Spare" equal Unknown
Sensor "Disk or Disk Bay 5 C1 P2I Bay 5: Drive Present" equal Unknown
Sensor "Disk or Disk Bay 5 C1 P2I Bay 5: Hot Spare" equal Unknown
Sensor "Disk or Disk Bay 6 C1 P2I Bay 6: Drive Present" equal Unknown
Sensor "Disk or Disk Bay 6 C1 P2I Bay 6: Hot Spare" equal Unknown
Sensor "Disk or Disk Bay 7 C1 P2I Bay 7: Drive Present" equal Unknown
Sensor "Disk or Disk Bay 7 C1 P2I Bay 7: Hot Spare" equal Unknown
Sensor "Disk or Disk Bay 8 C1 P2I Bay 8: Drive Present" equal Unknown
Sensor "Disk or Disk Bay 8 C1 P2I Bay 8: Hot Spare" equal Unknown
Sensor "Power Supply 1 Power Supply 1: Presence detected" equal Unknown
Sensor "Power Supply 2 Power Supply 2: Presence detected" equal Unknown
Vitaliy S.
VP, Product Management
Posts: 27375
Liked: 2799 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: Hardware sensors collection failure after V8 upgrade

Post by Vitaliy S. »

These sensors do not health state in the vSphere Client (this column) for them is empty, that is why they are reported as unknown. We are working on the workaround for this behavior. Thanks!
Vitaliy S.
VP, Product Management
Posts: 27375
Liked: 2799 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: Hardware sensors collection failure after V8 upgrade

Post by Vitaliy S. » 1 person likes this post

macgyver wrote:i never thought of the solution veeamsupport came up with.

on the one hand Veeamone/CIM-Smash can use ip-addresses for all servers but on the other hand for backwards communication to the infrastructure it must be able to resolve the fqdn of the esxi servers for hardware monitoring

no chance of working with ip addresses !

maybe you can explain this to me ;-)

it's by design ? :-)
If you have ESXi hosts added by IP address, you can still make it work by following either of these ways:

1. adjust "hosts" files on the Veeam ONE server in order to resolve host name
2. do the required changes in the vCenter Server config (Configuration -> DNS and Routing -> Host Identification)

The explanation of this requirement is pretty straight forward. When the API call is sent to the host to get the data for hardware sensors, ESXi host returns it's name (configured in the option 2 above) and then this name is used to establish the connection tunnel for data retrieval. If Veeam ONE cannot resolve the name of the host, then it cannot connect to it, thus hardware sensors data is not available.

In version 7 we were using different mechanism of getting this data, and when we have discovered an issue with the API, have decided to switch to a more reliable way of monitoring hardware senors.

Hope this makes sense!
macgyver
Influencer
Posts: 15
Liked: 2 times
Joined: Jan 20, 2011 6:03 pm
Contact:

Re: Hardware sensors collection failure after V8 upgrade

Post by macgyver »

Hi Vitaliy,

all clear as day ;-)
If I'd known about this earlier ;-)

Thanks

Regards

Mac
Post Reply

Who is online

Users browsing this forum: No registered users and 7 guests