Monitoring and reporting for Veeam Data Platform
Post Reply
athome
Service Provider
Posts: 63
Liked: 1 time
Joined: Jul 16, 2010 9:26 am
Full Name: Anders Thome
Contact:

Datastore Latency

Post by athome »

Hi!

A general question regarding the datastore latency-tab in Veeam ONE:

What is the difference between the counters:

"Datastore Latency Observed by VMs" and "Datastore Read Latency"? or

if one selects a VM and goes to the "Virtual Disks Latency", and have a look at read and write latency there.

A
Vitaliy S.
VP, Product Management
Posts: 27055
Liked: 2710 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: Datastore Latency

Post by Vitaliy S. » 1 person likes this post

Hi Anders,

The first counter belongs to hosts subsystem and shows the average datastore latency as observed by all virtual machines in the host. This counter is not available for individual VMs when you navigate to the datastore tab of the performance charts.

Thanks!
athome
Service Provider
Posts: 63
Liked: 1 time
Joined: Jul 16, 2010 9:26 am
Full Name: Anders Thome
Contact:

Re: Datastore Latency

Post by athome »

Hi!

Ok, but what is the "reallife difference" between the two?
Does that mean that as long as datastore latency observed by vms is low/zero there is no latency issue?

The reason I'm asking is I have a customer with a brand new storage system.. and they see spikes in latency.. Some extremely high as noted in another thread here.. I will try R2 and see if that changes, while some are "just" 80-200 ms high.. but the datastore latency ombserved by vms in that environment is 0 on all datastores(all are on Enterprise-class FC-disk).

While in my lab witch has a 3 year old FC-storage .. I see very little spikes in latency.. but Datastore latency observed by vms is quite high..: https://dl.dropboxusercontent.com/u/112 ... atency.png

A
Vitaliy S.
VP, Product Management
Posts: 27055
Liked: 2710 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: Datastore Latency

Post by Vitaliy S. »

Hi Anders,

Extreme values of datastore latency should not be observed once you upgrade to Veeam ONE v7 R2, though these values will still be seen in the vSphere Client. Yes, I also see 0 value for my VMs in my lab for this counter, while read/write latency is above zero. Do you have SIOC enabled in your environment?

Thanks!
athome
Service Provider
Posts: 63
Liked: 1 time
Joined: Jul 16, 2010 9:26 am
Full Name: Anders Thome
Contact:

Re: Datastore Latency

Post by athome »

Well In my lab I run a datastorecluster, so yes.. SIOC might be enabled there(but I've not tuned it) While in the customers environment there is no SIOC enabled.
Vitaliy S.
VP, Product Management
Posts: 27055
Liked: 2710 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: Datastore Latency

Post by Vitaliy S. »

In my lab SIOC is not enabled and "datastore latency observed by VMs" is zero all the time, I think this is also the case with your customer.
Vitaliy S.
VP, Product Management
Posts: 27055
Liked: 2710 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: Datastore Latency

Post by Vitaliy S. » 1 person likes this post

BTW, here is a good article on "datastore latency observed by VMs" > http://frankdenneman.nl/tag/vmobservedlatency/
speterson
Lurker
Posts: 1
Liked: never
Joined: Nov 04, 2014 5:10 pm
Full Name: Stephen Peterson
Contact:

Re: Datastore Latency

Post by speterson »

Im still not quite sure what thhe Datastore Latency Observed by VMs represents.

For instance I have an object 10.99.8.1/Datastore6 that shows
Write latency of 15 ms as the latest, 5 ms minimum 11.83 avg.
Read latency of 22 ms as the latest, 0 minimum, 14.35 avg
Latency observed by VMS 10082 as the latest, 3948 as the minimum, 11069 as the average.


What does the 10082 represent? does that mean during this 30 minute time frame that there were 10082 occurrences of latency greater than 50ms for that object?
I don't quite understand.

thanks
albertwt
Veeam Legend
Posts: 879
Liked: 46 times
Joined: Nov 05, 2009 12:24 pm
Location: Sydney, NSW
Contact:

Re: Datastore Latency

Post by albertwt »

Yes, it is quite confusing, because when I read the alarm email generated during the night for this type of alarm, I don't know how to interpret this error.
--
/* Veeam software enthusiast user & supporter ! */
Vitaliy S.
VP, Product Management
Posts: 27055
Liked: 2710 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: Datastore Latency

Post by Vitaliy S. » 1 person likes this post

Albert, what other latency values do you have for the same period? What activity do you have scheduled for the same time when this alarm occurs?
albertwt
Veeam Legend
Posts: 879
Liked: 46 times
Joined: Nov 05, 2009 12:24 pm
Location: Sydney, NSW
Contact:

Re: Datastore Latency

Post by albertwt »

Vitaliy S. wrote:Albert, what other latency values do you have for the same period? What activity do you have scheduled for the same time when this alarm occurs?
See the screenshot below:

Image

I have already enabled the option to Suppress the alarm during the Veeam backup job runs but somehow I still get bombarded by the latency alerting.
--
/* Veeam software enthusiast user & supporter ! */
Vitaliy S.
VP, Product Management
Posts: 27055
Liked: 2710 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: Datastore Latency

Post by Vitaliy S. »

What suppress options have you selected for Datastore and Host disk latency alarms?
albertwt
Veeam Legend
Posts: 879
Liked: 46 times
Joined: Nov 05, 2009 12:24 pm
Location: Sydney, NSW
Contact:

Re: Datastore Latency

Post by albertwt »

Vitaliy S. wrote:What suppress options have you selected for Datastore and Host disk latency alarms?
Vitaliy,

The only suppress option that I selected just the 1 hour as it is the maximum threshold value. is it possible to edit it to be like 2 or 4 hours instead ?
--
/* Veeam software enthusiast user & supporter ! */
Vitaliy S.
VP, Product Management
Posts: 27055
Liked: 2710 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: Datastore Latency

Post by Vitaliy S. »

Yes, that's possible to do in version 8, we have extended the list of available intervals. On the other hand, you may want to consider using time intervals to suppress the entire alarm during backup activity.
Post Reply

Who is online

Users browsing this forum: No registered users and 11 guests