-
- Enthusiast
- Posts: 99
- Liked: 12 times
- Joined: Jul 23, 2012 3:48 pm
- Contact:
VeeamOne & Disk Latency Alarms
I've been getting a LOT of alarm emails from Veeam regarding Host Datastore latency and VM disk latency, most reads but sometimes writes. We have it monitoring a Nutanix vSphere 5.5 cluster. Nutanix takes a somewhat non-traditional approach to the shared filesystem. It's presented as NFS shares, but under the hood it's some got dedup/compression/replication/SSD tiering going on. Also, the Nutanix console doesn't report any IO alarms or anything that resebles what VeeamOne is reporting. Nor do the VMs themselves seem to experiencing any sluggish-ness.
I've tried to increasing both the number of milliseconds that has to be hit before VeeamOne alarms as well as the amount of time. Changing this helped, but I still got 300+ emails over the weekend. Is there anything else I can do besides just disable those 2 alarms?
I've tried to increasing both the number of milliseconds that has to be hit before VeeamOne alarms as well as the amount of time. Changing this helped, but I still got 300+ emails over the weekend. Is there anything else I can do besides just disable those 2 alarms?
-
- VP, Product Management
- Posts: 27377
- Liked: 2800 times
- Joined: Mar 30, 2009 9:13 am
- Full Name: Vitaliy Safarov
- Contact:
Re: VeeamOne & Disk Latency Alarms
Hello,
What are you current alarm thresholds and latency performance graphs on the hosts? Are these alarms triggered at a particular time or it can happens just randomly?
Thank you!
What are you current alarm thresholds and latency performance graphs on the hosts? Are these alarms triggered at a particular time or it can happens just randomly?
Thank you!
-
- Enthusiast
- Posts: 99
- Liked: 12 times
- Joined: Jul 23, 2012 3:48 pm
- Contact:
Re: VeeamOne & Disk Latency Alarms
I have Host Disk Read Latency (currently disabled) set to warn @ 100 ms, error @ 250 ms and time period 30 mins.
I have Virtual Machine VM Total Disk Latency (disabled) set to warn 60 ms and error for 85 ms.
Not all of the triggered alerts (but most) happen at night during the backups (using Veeam B&R), so yes these alerts are happening during a period of increased IO....but by the same token I can't believe some of the readings I'm getting. A datastore with 100+ ms latency? Seems like if that were really happening....lots of stuff would be broken.
I have Virtual Machine VM Total Disk Latency (disabled) set to warn 60 ms and error for 85 ms.
Not all of the triggered alerts (but most) happen at night during the backups (using Veeam B&R), so yes these alerts are happening during a period of increased IO....but by the same token I can't believe some of the readings I'm getting. A datastore with 100+ ms latency? Seems like if that were really happening....lots of stuff would be broken.
-
- VP, Product Management
- Posts: 27377
- Liked: 2800 times
- Joined: Mar 30, 2009 9:13 am
- Full Name: Vitaliy Safarov
- Contact:
Re: VeeamOne & Disk Latency Alarms
Well...we are getting this information through VMware API and it must be something with how Nutanix reports on its performance. Anyway, if this happens during backup windows I would recommend setting a suppress period for the entire window. Should address the massive email notifications issue.
-
- Enthusiast
- Posts: 99
- Liked: 12 times
- Joined: Jul 23, 2012 3:48 pm
- Contact:
Re: VeeamOne & Disk Latency Alarms
Nice! I didn't even notice that suppress tab, AND there is the option to suppress it during a Veeam Backup. VeeamOne being aware of Veeam B&R backups for the win!
-
- VP, Product Management
- Posts: 27377
- Liked: 2800 times
- Joined: Mar 30, 2009 9:13 am
- Full Name: Vitaliy Safarov
- Contact:
Re: VeeamOne & Disk Latency Alarms
Of course we are aware about Veeam B&R, moreover you can monitor/report on your Veeam backups with Veeam ONE.
-
- Enthusiast
- Posts: 50
- Liked: 6 times
- Joined: Aug 17, 2012 12:31 pm
- Contact:
Re: VeeamOne & Disk Latency Alarms
My latency occurs on a single datastore during SureBackup verification jobs. Does the suppress for "Veeam Backup Activity" also apply to SureBackup jobs?
-
- VP, Product Management
- Posts: 27377
- Liked: 2800 times
- Joined: Mar 30, 2009 9:13 am
- Full Name: Vitaliy Safarov
- Contact:
Re: VeeamOne & Disk Latency Alarms
No it does not, it only tracks backup job activity. BTW, do you schedule your SureBackup jobs? It might be worth trying to suppress this alarm during SB job via advanced alarm settings.
-
- Enthusiast
- Posts: 50
- Liked: 6 times
- Joined: Aug 17, 2012 12:31 pm
- Contact:
Re: VeeamOne & Disk Latency Alarms
Yes they're scheduled - I guess I'll go that route, thanks for the reply!
-
- Influencer
- Posts: 19
- Liked: 1 time
- Joined: Jun 09, 2014 10:57 pm
- Full Name: Dana
- Contact:
Re: VeeamOne & Disk Latency Alarms
I am seeing this issue with replication, which is ongoing. I would like to be able to monitor the latency, but it is reporting in such a away that changing thresholds wont help. Our SAN is a Dell MD3220i and we haven't had any issues with it. Somehow i don't think the latency below is accurate, is this a known issue that can be addressed? Thanks!
"Datastore Highest Latency" (3552916809269 ms) is above a defined threshold (75 ms)
"Datastore Highest Latency" (3552916809269 ms) is above a defined threshold (75 ms)
-
- VP, Product Management
- Posts: 27377
- Liked: 2800 times
- Joined: Mar 30, 2009 9:13 am
- Full Name: Vitaliy Safarov
- Contact:
Re: VeeamOne & Disk Latency Alarms
Hello Dana,
Yes, you're correct that these values are inaccurately reported by VMware. Please check out this topic for the solution > V7 - VMware Issue - False alerts
Thank you!
Yes, you're correct that these values are inaccurately reported by VMware. Please check out this topic for the solution > V7 - VMware Issue - False alerts
Thank you!
-
- Influencer
- Posts: 19
- Liked: 1 time
- Joined: Jun 09, 2014 10:57 pm
- Full Name: Dana
- Contact:
Re: VeeamOne & Disk Latency Alarms
Ah, thanks. I JUST downloaded in installed VeeamONE about 6 days ago, didn't realize i was missing an update! I love it by the way, fantastic access to stats. I do have a question but i'll start a new thread. Thanks!!
Dana
Dana
Who is online
Users browsing this forum: No registered users and 4 guests