VM total disk latency

Real-time performance monitoring and troubleshooting

VM total disk latency

Veeam Logoby Pat490 » Thu Jun 22, 2017 9:04 am

Since a while I do get constantly warnings from alarm "VM total disk latency" for 1 VM with "Datastore Highest Latency" in details > 200+ milliseconds

but when I check vCenter graphs I can not find those high numbers. Also in V1 Monitor virtual disks graph I do not see those number. ESXTOP is also low.

So I am wondering why the alarm is triggered?

Image
Pat490
Expert
 
Posts: 141
Liked: 24 times
Joined: Tue Apr 28, 2015 7:18 am
Location: Germany
Full Name: Patrick

Re: VM total disk latency

Veeam Logoby Shestakov » Thu Jun 22, 2017 11:57 am

Hello Patrick,
That may be caused by performance metrics averaging. How much time past between the alarm was triggered and you checked Veeam ONE Monitor graphs? What data collection mode are you at (for typical infrastructures or for advanced ones)?
Thanks!
Shestakov
Veeam Software
 
Posts: 5194
Liked: 433 times
Joined: Wed May 21, 2014 11:03 am
Location: Saint Petersburg
Full Name: Nikita Shestakov

Re: VM total disk latency

Veeam Logoby Pat490 » Thu Jun 22, 2017 1:09 pm

time between alarms varies... for example today it was 8:29, 9:18, 9:50, 10:25, 11:14, 11:49, 12:45, 14:51

screenshot in last post was from VeeamOne Monitor. Or did I missunderstand?

What data collection mode? what do you mean? Here are the alarm settings:
Image
Pat490
Expert
 
Posts: 141
Liked: 24 times
Joined: Tue Apr 28, 2015 7:18 am
Location: Germany
Full Name: Patrick

Re: VM total disk latency

Veeam Logoby Shestakov » Thu Jun 22, 2017 2:35 pm

What I meant that depending on data collection mode, Veeam ONE collects 20-second samples from ESXi and shows it during the day or so(depends on the settings), then it aggregates 20-second values to 5-minutes intervals and shows the average value and so on up to 2-hour averages. These numbers are for Typical data collection mode.
According to your alarm settings, it will trigger if 15 min average Latency is higher than 75msec. If you check the performance graph a day after the alarm was triggered you may not find spikes as they are flattered.

What data collection mode? what do you mean?

I meant this setting. The one you choose during setup.
Shestakov
Veeam Software
 
Posts: 5194
Liked: 433 times
Joined: Wed May 21, 2014 11:03 am
Location: Saint Petersburg
Full Name: Nikita Shestakov

Re: VM total disk latency

Veeam Logoby Pat490 » Thu Jun 22, 2017 5:08 pm

It should be set to typical as we have small environment with only 3 hosts.
I am looking at graphs for a few days now and never saw peaks as high as written in the alarms with 200ms or more
Pat490
Expert
 
Posts: 141
Liked: 24 times
Joined: Tue Apr 28, 2015 7:18 am
Location: Germany
Full Name: Patrick

Re: VM total disk latency

Veeam Logoby Shestakov » Thu Jun 22, 2017 5:24 pm

Do you observe same issue with other alarms?
I would also recommend contact support for closer investigation.
Thanks!
Shestakov
Veeam Software
 
Posts: 5194
Liked: 433 times
Joined: Wed May 21, 2014 11:03 am
Location: Saint Petersburg
Full Name: Nikita Shestakov

Re: VM total disk latency

Veeam Logoby Pat490 » Fri Jun 23, 2017 5:19 am

rarely I get a "Datastore read latency" alarm for 1 datastore which I also can not see in vSphere graphs
Pat490
Expert
 
Posts: 141
Liked: 24 times
Joined: Tue Apr 28, 2015 7:18 am
Location: Germany
Full Name: Patrick

Re: VM total disk latency

Veeam Logoby Shestakov » Fri Jun 23, 2017 4:50 pm

Got it, thanks!
If you decide to contact support team, please post your case number.
Shestakov
Veeam Software
 
Posts: 5194
Liked: 433 times
Joined: Wed May 21, 2014 11:03 am
Location: Saint Petersburg
Full Name: Nikita Shestakov

Re: VM total disk latency

Veeam Logoby veit » Tue Sep 05, 2017 9:48 am

Hello,

sorry for pushing up this topic again. Since we face the exact same issue, i wanted to avoid opening a dublicated thread.
We receive rathe often datastore alarms for read or wirte latencies. They fire up for different datastores at random times.
Our infrastructure consists of 2 hosts, each connected to 20 datastores. In Average we find every day 5 to 10 latency warnings for all datastores. So only a few datastores each day, but changing from day to day.
My aim is it to correlate the warnings with perfmormance data collected for the datastore. Warning / Alarm is set as 15min max above 100ms / 250ms.
The tiniest interval i can find in I/O Graph or in the raw-data report is a 5 Minute average. Those latency peaks never appear in the 5min average. And i find it impossible to get IOP/s information for the exact time when latency warning comes up.

Example. Datastore "vol03" fired up warning read latency on 31.08.2017 at 12:53 with 153ms. Raw Data shows "Value(Millisecond)" for Datastore read latency in interval 31.08.2017 12:55: 0ms. Next interval (31.08.2017 13:00) shows 47ms, but would be wrong interval regarding the time of the warning. Raw Data IOPs : 31.08.2017 12:50 = 43 and on 31.08.2017 IOPs are at 93. Which is rather low i suppose.

Is there a way to get more fine grained raw Data / Performance Data. IOPs max would be interesting. Or 1 minute average for latency.
veit
Lurker
 
Posts: 1
Liked: never
Joined: Tue Sep 05, 2017 8:28 am
Full Name: veit

Re: VM total disk latency

Veeam Logoby Shestakov » Fri Sep 08, 2017 11:52 am

Hello veit,
You can change performance data granularity via support team. Note that it will increase database size and may slightly slow down application.
As for raw data, check Raw Performance Data report.
Thanks!
Shestakov
Veeam Software
 
Posts: 5194
Liked: 433 times
Joined: Wed May 21, 2014 11:03 am
Location: Saint Petersburg
Full Name: Nikita Shestakov


Return to Monitoring



Who is online

Users browsing this forum: No registered users and 1 guest