Datastore Latency Alarms

[Barker]

I've been getting random datastore latency alarms (both read and write) from various VM's.

I've seen a number of datastore latency threads regarding the issue where the aggregation was by default set to "Max" but I've already changed that to "Avg" and it hasn't helped at all. What's strange is when I check datastore performance reports on the latency both in VMware AND Veeam ONE, I'm not seeing number anywhere near what the alarms are getting triggered at.

Here is an example of what I'm seeing:
VM Datastore at 4:00AM, triggers with the following error:
"Disk/Datastore: Datastore Read Latency" (125.0 Milliseconds) is above a defined threshold (100.0 Milliseconds)

However when I run a report in Veeam ONE for that same timeframe and Datastore, below are the details:
Disk/Datastore: Datastore Read Latency (Millisecond)
Average - 5.00
Minimum - 0.00
Maximum - 40.00
Std. Deviation - 12.03
Trend - Decreasing

And specifically, at that time there was a spike, but it only maxed at about 17milliseconds, so I'm unsure why it would even trigger an alarm stating 125 milliseconds?

Am I missing something?

[Shestakov]

That`s possible if you observe historical performance values. The intervals are aggregated from shorter values which could trigger the alarm.
Please review this topic.
What "Time Period" do you have set in the alarm, 15minutes? Have you generated the report same day the alarm was triggered?
Thanks!

[Barker]

Yes, I generated the report a few hours after the alert occurred. We get them pretty regularly (a few times a day) from multiple datastores.

Yes, 15 minutes (the default) is where I left it. Should I increase that?

I ran the "modeling" troubleshooter and increased it to 10 days as a test and it finally brought the error down to a warning, but I'm assuming I don't want to go that high.

What would be a good timeframe?

[Barker]

Also, in that link you posted, you had mentioned back then that there was a known bug in in that version for that particular alarm that triggers an alert when a spike happens, regardless of whether it is set to AVG or MAX. I'm assuming that's been fixed since then? We are currently on version 9.5.0.3254.

[Shestakov]

15 minutes is a recommended interval. I just checked if you not decreased it to 1 min so 1 spike can fire the alarm.
The mentioned bug is fixed, but behavior in your case looks unexpected, so I would ask to contact Veeam support.
Thanks!