Monitoring and reporting for Veeam Backup & Replication, VMware vSphere and Microsoft Hyper-V
Post Reply
Barker
Novice
Posts: 7
Liked: never
Joined: Aug 26, 2017 1:59 pm
Contact:

Datastore Latency Alarms

Post by Barker » Nov 08, 2017 4:25 pm

I've been getting random datastore latency alarms (both read and write) from various VM's.

I've seen a number of datastore latency threads regarding the issue where the aggregation was by default set to "Max" but I've already changed that to "Avg" and it hasn't helped at all. What's strange is when I check datastore performance reports on the latency both in VMware AND Veeam ONE, I'm not seeing number anywhere near what the alarms are getting triggered at.

Here is an example of what I'm seeing:
VM Datastore at 4:00AM, triggers with the following error:
"Disk/Datastore: Datastore Read Latency" (125.0 Milliseconds) is above a defined threshold (100.0 Milliseconds)

However when I run a report in Veeam ONE for that same timeframe and Datastore, below are the details:
Disk/Datastore: Datastore Read Latency (Millisecond)
Average - 5.00
Minimum - 0.00
Maximum - 40.00
Std. Deviation - 12.03
Trend - Decreasing

And specifically, at that time there was a spike, but it only maxed at about 17milliseconds, so I'm unsure why it would even trigger an alarm stating 125 milliseconds?

Am I missing something?

Shestakov
Veeam Software
Posts: 5968
Liked: 519 times
Joined: May 21, 2014 11:03 am
Full Name: Nikita Shestakov
Location: Prague
Contact:

Re: Datastore Latency Alarms

Post by Shestakov » Nov 09, 2017 2:56 pm

That`s possible if you observe historical performance values. The intervals are aggregated from shorter values which could trigger the alarm.
Please review this topic.
What "Time Period" do you have set in the alarm, 15minutes? Have you generated the report same day the alarm was triggered?
Thanks!

Barker
Novice
Posts: 7
Liked: never
Joined: Aug 26, 2017 1:59 pm
Contact:

Re: Datastore Latency Alarms

Post by Barker » Nov 09, 2017 4:55 pm

Yes, I generated the report a few hours after the alert occurred. We get them pretty regularly (a few times a day) from multiple datastores.

Yes, 15 minutes (the default) is where I left it. Should I increase that?

I ran the "modeling" troubleshooter and increased it to 10 days as a test and it finally brought the error down to a warning, but I'm assuming I don't want to go that high.

What would be a good timeframe?

Barker
Novice
Posts: 7
Liked: never
Joined: Aug 26, 2017 1:59 pm
Contact:

Re: Datastore Latency Alarms

Post by Barker » Nov 09, 2017 5:50 pm

Also, in that link you posted, you had mentioned back then that there was a known bug in in that version for that particular alarm that triggers an alert when a spike happens, regardless of whether it is set to AVG or MAX. I'm assuming that's been fixed since then? We are currently on version 9.5.0.3254.

Shestakov
Veeam Software
Posts: 5968
Liked: 519 times
Joined: May 21, 2014 11:03 am
Full Name: Nikita Shestakov
Location: Prague
Contact:

Re: Datastore Latency Alarms

Post by Shestakov » Nov 09, 2017 10:41 pm

15 minutes is a recommended interval. I just checked if you not decreased it to 1 min so 1 spike can fire the alarm.
The mentioned bug is fixed, but behavior in your case looks unexpected, so I would ask to contact Veeam support.
Thanks!

Post Reply

Who is online

Users browsing this forum: Baidu [Spider] and 2 guests