VeeamOne & Disk Latency Alarms

Real-time performance monitoring and troubleshooting

VeeamOne & Disk Latency Alarms

Veeam Logoby nmace » Mon Feb 17, 2014 7:37 pm

I've been getting a LOT of alarm emails from Veeam regarding Host Datastore latency and VM disk latency, most reads but sometimes writes. We have it monitoring a Nutanix vSphere 5.5 cluster. Nutanix takes a somewhat non-traditional approach to the shared filesystem. It's presented as NFS shares, but under the hood it's some got dedup/compression/replication/SSD tiering going on. Also, the Nutanix console doesn't report any IO alarms or anything that resebles what VeeamOne is reporting. Nor do the VMs themselves seem to experiencing any sluggish-ness.

I've tried to increasing both the number of milliseconds that has to be hit before VeeamOne alarms as well as the amount of time. Changing this helped, but I still got 300+ emails over the weekend. Is there anything else I can do besides just disable those 2 alarms?
nmace
Enthusiast
 
Posts: 93
Liked: 12 times
Joined: Mon Jul 23, 2012 3:48 pm

Re: VeeamOne & Disk Latency Alarms

Veeam Logoby Vitaliy S. » Mon Feb 17, 2014 9:28 pm

Hello,

What are you current alarm thresholds and latency performance graphs on the hosts? Are these alarms triggered at a particular time or it can happens just randomly?

Thank you!
Vitaliy S.
Veeam Software
 
Posts: 19558
Liked: 1102 times
Joined: Mon Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov

Re: VeeamOne & Disk Latency Alarms

Veeam Logoby nmace » Mon Feb 17, 2014 10:03 pm

I have Host Disk Read Latency (currently disabled) set to warn @ 100 ms, error @ 250 ms and time period 30 mins.

I have Virtual Machine VM Total Disk Latency (disabled) set to warn 60 ms and error for 85 ms.

Not all of the triggered alerts (but most) happen at night during the backups (using Veeam B&R), so yes these alerts are happening during a period of increased IO....but by the same token I can't believe some of the readings I'm getting. A datastore with 100+ ms latency? Seems like if that were really happening....lots of stuff would be broken.
nmace
Enthusiast
 
Posts: 93
Liked: 12 times
Joined: Mon Jul 23, 2012 3:48 pm

Re: VeeamOne & Disk Latency Alarms

Veeam Logoby Vitaliy S. » Tue Feb 18, 2014 9:56 am

Well...we are getting this information through VMware API and it must be something with how Nutanix reports on its performance. Anyway, if this happens during backup windows I would recommend setting a suppress period for the entire window. Should address the massive email notifications issue.
Vitaliy S.
Veeam Software
 
Posts: 19558
Liked: 1102 times
Joined: Mon Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov

Re: VeeamOne & Disk Latency Alarms

Veeam Logoby nmace » Tue Feb 18, 2014 3:48 pm

Nice! I didn't even notice that suppress tab, AND there is the option to suppress it during a Veeam Backup. VeeamOne being aware of Veeam B&R backups for the win!
nmace
Enthusiast
 
Posts: 93
Liked: 12 times
Joined: Mon Jul 23, 2012 3:48 pm

Re: VeeamOne & Disk Latency Alarms

Veeam Logoby Vitaliy S. » Tue Feb 18, 2014 4:01 pm

Of course we are aware about Veeam B&R, moreover you can monitor/report on your Veeam backups with Veeam ONE.
Vitaliy S.
Veeam Software
 
Posts: 19558
Liked: 1102 times
Joined: Mon Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov

Re: VeeamOne & Disk Latency Alarms

Veeam Logoby cstemaly » Wed Apr 02, 2014 12:34 pm

My latency occurs on a single datastore during SureBackup verification jobs. Does the suppress for "Veeam Backup Activity" also apply to SureBackup jobs?
cstemaly
Enthusiast
 
Posts: 50
Liked: 6 times
Joined: Fri Aug 17, 2012 12:31 pm

Re: VeeamOne & Disk Latency Alarms

Veeam Logoby Vitaliy S. » Wed Apr 02, 2014 1:22 pm

No it does not, it only tracks backup job activity. BTW, do you schedule your SureBackup jobs? It might be worth trying to suppress this alarm during SB job via advanced alarm settings.
Vitaliy S.
Veeam Software
 
Posts: 19558
Liked: 1102 times
Joined: Mon Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov

Re: VeeamOne & Disk Latency Alarms

Veeam Logoby cstemaly » Wed Apr 02, 2014 2:06 pm

Yes they're scheduled - I guess I'll go that route, thanks for the reply!
cstemaly
Enthusiast
 
Posts: 50
Liked: 6 times
Joined: Fri Aug 17, 2012 12:31 pm

Re: VeeamOne & Disk Latency Alarms

Veeam Logoby danas » Mon Jul 21, 2014 2:51 pm

I am seeing this issue with replication, which is ongoing. I would like to be able to monitor the latency, but it is reporting in such a away that changing thresholds wont help. Our SAN is a Dell MD3220i and we haven't had any issues with it. Somehow i don't think the latency below is accurate, is this a known issue that can be addressed? Thanks!

"Datastore Highest Latency" (3552916809269 ms) is above a defined threshold (75 ms)
danas
Influencer
 
Posts: 19
Liked: 1 time
Joined: Mon Jun 09, 2014 10:57 pm
Full Name: Dana

Re: VeeamOne & Disk Latency Alarms

Veeam Logoby Vitaliy S. » Mon Jul 21, 2014 3:31 pm

Hello Dana,

Yes, you're correct that these values are inaccurately reported by VMware. Please check out this topic for the solution > V7 - VMware Issue - False alerts

Thank you!
Vitaliy S.
Veeam Software
 
Posts: 19558
Liked: 1102 times
Joined: Mon Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov

Re: VeeamOne & Disk Latency Alarms

Veeam Logoby danas » Mon Jul 21, 2014 5:50 pm 1 person likes this post

Ah, thanks. I JUST downloaded in installed VeeamONE about 6 days ago, didn't realize i was missing an update! I love it by the way, fantastic access to stats. I do have a question but i'll start a new thread. Thanks!!

Dana
danas
Influencer
 
Posts: 19
Liked: 1 time
Joined: Mon Jun 09, 2014 10:57 pm
Full Name: Dana


Return to Monitoring



Who is online

Users browsing this forum: No registered users and 2 guests