Real-time performance monitoring and troubleshooting
Post Reply
nmace
Enthusiast
Posts: 99
Liked: 12 times
Joined: Jul 23, 2012 3:48 pm
Contact:

VeeamOne & Disk Latency Alarms

Post by nmace »

I've been getting a LOT of alarm emails from Veeam regarding Host Datastore latency and VM disk latency, most reads but sometimes writes. We have it monitoring a Nutanix vSphere 5.5 cluster. Nutanix takes a somewhat non-traditional approach to the shared filesystem. It's presented as NFS shares, but under the hood it's some got dedup/compression/replication/SSD tiering going on. Also, the Nutanix console doesn't report any IO alarms or anything that resebles what VeeamOne is reporting. Nor do the VMs themselves seem to experiencing any sluggish-ness.

I've tried to increasing both the number of milliseconds that has to be hit before VeeamOne alarms as well as the amount of time. Changing this helped, but I still got 300+ emails over the weekend. Is there anything else I can do besides just disable those 2 alarms?
Vitaliy S.
VP, Product Management
Posts: 27055
Liked: 2710 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: VeeamOne & Disk Latency Alarms

Post by Vitaliy S. »

Hello,

What are you current alarm thresholds and latency performance graphs on the hosts? Are these alarms triggered at a particular time or it can happens just randomly?

Thank you!
nmace
Enthusiast
Posts: 99
Liked: 12 times
Joined: Jul 23, 2012 3:48 pm
Contact:

Re: VeeamOne & Disk Latency Alarms

Post by nmace »

I have Host Disk Read Latency (currently disabled) set to warn @ 100 ms, error @ 250 ms and time period 30 mins.

I have Virtual Machine VM Total Disk Latency (disabled) set to warn 60 ms and error for 85 ms.

Not all of the triggered alerts (but most) happen at night during the backups (using Veeam B&R), so yes these alerts are happening during a period of increased IO....but by the same token I can't believe some of the readings I'm getting. A datastore with 100+ ms latency? Seems like if that were really happening....lots of stuff would be broken.
Vitaliy S.
VP, Product Management
Posts: 27055
Liked: 2710 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: VeeamOne & Disk Latency Alarms

Post by Vitaliy S. »

Well...we are getting this information through VMware API and it must be something with how Nutanix reports on its performance. Anyway, if this happens during backup windows I would recommend setting a suppress period for the entire window. Should address the massive email notifications issue.
nmace
Enthusiast
Posts: 99
Liked: 12 times
Joined: Jul 23, 2012 3:48 pm
Contact:

Re: VeeamOne & Disk Latency Alarms

Post by nmace »

Nice! I didn't even notice that suppress tab, AND there is the option to suppress it during a Veeam Backup. VeeamOne being aware of Veeam B&R backups for the win!
Vitaliy S.
VP, Product Management
Posts: 27055
Liked: 2710 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: VeeamOne & Disk Latency Alarms

Post by Vitaliy S. »

Of course we are aware about Veeam B&R, moreover you can monitor/report on your Veeam backups with Veeam ONE.
cstemaly
Enthusiast
Posts: 50
Liked: 6 times
Joined: Aug 17, 2012 12:31 pm
Contact:

Re: VeeamOne & Disk Latency Alarms

Post by cstemaly »

My latency occurs on a single datastore during SureBackup verification jobs. Does the suppress for "Veeam Backup Activity" also apply to SureBackup jobs?
Vitaliy S.
VP, Product Management
Posts: 27055
Liked: 2710 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: VeeamOne & Disk Latency Alarms

Post by Vitaliy S. »

No it does not, it only tracks backup job activity. BTW, do you schedule your SureBackup jobs? It might be worth trying to suppress this alarm during SB job via advanced alarm settings.
cstemaly
Enthusiast
Posts: 50
Liked: 6 times
Joined: Aug 17, 2012 12:31 pm
Contact:

Re: VeeamOne & Disk Latency Alarms

Post by cstemaly »

Yes they're scheduled - I guess I'll go that route, thanks for the reply!
danas
Influencer
Posts: 19
Liked: 1 time
Joined: Jun 09, 2014 10:57 pm
Full Name: Dana
Contact:

Re: VeeamOne & Disk Latency Alarms

Post by danas »

I am seeing this issue with replication, which is ongoing. I would like to be able to monitor the latency, but it is reporting in such a away that changing thresholds wont help. Our SAN is a Dell MD3220i and we haven't had any issues with it. Somehow i don't think the latency below is accurate, is this a known issue that can be addressed? Thanks!

"Datastore Highest Latency" (3552916809269 ms) is above a defined threshold (75 ms)
Vitaliy S.
VP, Product Management
Posts: 27055
Liked: 2710 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: VeeamOne & Disk Latency Alarms

Post by Vitaliy S. »

Hello Dana,

Yes, you're correct that these values are inaccurately reported by VMware. Please check out this topic for the solution > V7 - VMware Issue - False alerts

Thank you!
danas
Influencer
Posts: 19
Liked: 1 time
Joined: Jun 09, 2014 10:57 pm
Full Name: Dana
Contact:

Re: VeeamOne & Disk Latency Alarms

Post by danas » 1 person likes this post

Ah, thanks. I JUST downloaded in installed VeeamONE about 6 days ago, didn't realize i was missing an update! I love it by the way, fantastic access to stats. I do have a question but i'll start a new thread. Thanks!!

Dana
Post Reply

Who is online

Users browsing this forum: No registered users and 6 guests