Datastore Latency Alarms

Monitoring and reporting for Veeam Backup & Replication, VMware vSphere and Microsoft Hyper-V

Datastore Latency Alarms

Veeam Logoby Barker » Wed Nov 08, 2017 4:25 pm

I've been getting random datastore latency alarms (both read and write) from various VM's.

I've seen a number of datastore latency threads regarding the issue where the aggregation was by default set to "Max" but I've already changed that to "Avg" and it hasn't helped at all. What's strange is when I check datastore performance reports on the latency both in VMware AND Veeam ONE, I'm not seeing number anywhere near what the alarms are getting triggered at.

Here is an example of what I'm seeing:
VM Datastore at 4:00AM, triggers with the following error:
"Disk/Datastore: Datastore Read Latency" (125.0 Milliseconds) is above a defined threshold (100.0 Milliseconds)

However when I run a report in Veeam ONE for that same timeframe and Datastore, below are the details:
Disk/Datastore: Datastore Read Latency (Millisecond)
Average - 5.00
Minimum - 0.00
Maximum - 40.00
Std. Deviation - 12.03
Trend - Decreasing

And specifically, at that time there was a spike, but it only maxed at about 17milliseconds, so I'm unsure why it would even trigger an alarm stating 125 milliseconds?

Am I missing something?
Barker
Novice
 
Posts: 7
Liked: never
Joined: Sat Aug 26, 2017 1:59 pm

Re: Datastore Latency Alarms

Veeam Logoby Shestakov » Thu Nov 09, 2017 2:56 pm

That`s possible if you observe historical performance values. The intervals are aggregated from shorter values which could trigger the alarm.
Please review this topic.
What "Time Period" do you have set in the alarm, 15minutes? Have you generated the report same day the alarm was triggered?
Thanks!
Shestakov
Veeam Software
 
Posts: 5210
Liked: 433 times
Joined: Wed May 21, 2014 11:03 am
Location: Saint Petersburg
Full Name: Nikita Shestakov

Re: Datastore Latency Alarms

Veeam Logoby Barker » Thu Nov 09, 2017 4:55 pm

Yes, I generated the report a few hours after the alert occurred. We get them pretty regularly (a few times a day) from multiple datastores.

Yes, 15 minutes (the default) is where I left it. Should I increase that?

I ran the "modeling" troubleshooter and increased it to 10 days as a test and it finally brought the error down to a warning, but I'm assuming I don't want to go that high.

What would be a good timeframe?
Barker
Novice
 
Posts: 7
Liked: never
Joined: Sat Aug 26, 2017 1:59 pm

Re: Datastore Latency Alarms

Veeam Logoby Barker » Thu Nov 09, 2017 5:50 pm

Also, in that link you posted, you had mentioned back then that there was a known bug in in that version for that particular alarm that triggers an alert when a spike happens, regardless of whether it is set to AVG or MAX. I'm assuming that's been fixed since then? We are currently on version 9.5.0.3254.
Barker
Novice
 
Posts: 7
Liked: never
Joined: Sat Aug 26, 2017 1:59 pm

Re: Datastore Latency Alarms

Veeam Logoby Shestakov » Thu Nov 09, 2017 10:41 pm

15 minutes is a recommended interval. I just checked if you not decreased it to 1 min so 1 spike can fire the alarm.
The mentioned bug is fixed, but behavior in your case looks unexpected, so I would ask to contact Veeam support.
Thanks!
Shestakov
Veeam Software
 
Posts: 5210
Liked: 433 times
Joined: Wed May 21, 2014 11:03 am
Location: Saint Petersburg
Full Name: Nikita Shestakov


Return to Veeam ONE



Who is online

Users browsing this forum: Bing [Bot] and 1 guest