Datastore Latency

Monitoring and reporting for Veeam Backup & Replication, VMware vSphere and Microsoft Hyper-V

Datastore Latency

Veeam Logoby athome » Fri Dec 06, 2013 1:34 pm

Hi!

A general question regarding the datastore latency-tab in Veeam ONE:

What is the difference between the counters:

"Datastore Latency Observed by VMs" and "Datastore Read Latency"? or

if one selects a VM and goes to the "Virtual Disks Latency", and have a look at read and write latency there.

A
athome
Enthusiast
 
Posts: 56
Liked: 1 time
Joined: Fri Jul 16, 2010 9:26 am
Full Name: Anders Thome

Re: Datastore Latency

Veeam Logoby Vitaliy S. » Fri Dec 06, 2013 4:02 pm 1 person likes this post

Hi Anders,

The first counter belongs to hosts subsystem and shows the average datastore latency as observed by all virtual machines in the host. This counter is not available for individual VMs when you navigate to the datastore tab of the performance charts.

Thanks!
Vitaliy S.
Veeam Software
 
Posts: 19564
Liked: 1104 times
Joined: Mon Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov

Re: Datastore Latency

Veeam Logoby athome » Tue Dec 10, 2013 6:13 am

Hi!

Ok, but what is the "reallife difference" between the two?
Does that mean that as long as datastore latency observed by vms is low/zero there is no latency issue?

The reason I'm asking is I have a customer with a brand new storage system.. and they see spikes in latency.. Some extremely high as noted in another thread here.. I will try R2 and see if that changes, while some are "just" 80-200 ms high.. but the datastore latency ombserved by vms in that environment is 0 on all datastores(all are on Enterprise-class FC-disk).

While in my lab witch has a 3 year old FC-storage .. I see very little spikes in latency.. but Datastore latency observed by vms is quite high..: https://dl.dropboxusercontent.com/u/112 ... atency.png

A
athome
Enthusiast
 
Posts: 56
Liked: 1 time
Joined: Fri Jul 16, 2010 9:26 am
Full Name: Anders Thome

Re: Datastore Latency

Veeam Logoby Vitaliy S. » Tue Dec 10, 2013 1:27 pm

Hi Anders,

Extreme values of datastore latency should not be observed once you upgrade to Veeam ONE v7 R2, though these values will still be seen in the vSphere Client. Yes, I also see 0 value for my VMs in my lab for this counter, while read/write latency is above zero. Do you have SIOC enabled in your environment?

Thanks!
Vitaliy S.
Veeam Software
 
Posts: 19564
Liked: 1104 times
Joined: Mon Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov

Re: Datastore Latency

Veeam Logoby athome » Wed Dec 11, 2013 6:09 am

Well In my lab I run a datastorecluster, so yes.. SIOC might be enabled there(but I've not tuned it) While in the customers environment there is no SIOC enabled.
athome
Enthusiast
 
Posts: 56
Liked: 1 time
Joined: Fri Jul 16, 2010 9:26 am
Full Name: Anders Thome

Re: Datastore Latency

Veeam Logoby Vitaliy S. » Wed Dec 11, 2013 9:52 am

In my lab SIOC is not enabled and "datastore latency observed by VMs" is zero all the time, I think this is also the case with your customer.
Vitaliy S.
Veeam Software
 
Posts: 19564
Liked: 1104 times
Joined: Mon Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov

Re: Datastore Latency

Veeam Logoby Vitaliy S. » Wed Dec 11, 2013 10:23 am 1 person likes this post

BTW, here is a good article on "datastore latency observed by VMs" > http://frankdenneman.nl/tag/vmobservedlatency/
Vitaliy S.
Veeam Software
 
Posts: 19564
Liked: 1104 times
Joined: Mon Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov

Re: Datastore Latency

Veeam Logoby speterson » Tue Nov 04, 2014 5:20 pm

Im still not quite sure what thhe Datastore Latency Observed by VMs represents.

For instance I have an object 10.99.8.1/Datastore6 that shows
Write latency of 15 ms as the latest, 5 ms minimum 11.83 avg.
Read latency of 22 ms as the latest, 0 minimum, 14.35 avg
Latency observed by VMS 10082 as the latest, 3948 as the minimum, 11069 as the average.


What does the 10082 represent? does that mean during this 30 minute time frame that there were 10082 occurrences of latency greater than 50ms for that object?
I don't quite understand.

thanks
speterson
Lurker
 
Posts: 1
Liked: never
Joined: Tue Nov 04, 2014 5:10 pm
Full Name: Stephen Peterson

Re: Datastore Latency

Veeam Logoby albertwt » Mon Nov 10, 2014 12:44 pm

Yes, it is quite confusing, because when I read the alarm email generated during the night for this type of alarm, I don't know how to interpret this error.
--
/* Veeam software enthusiast user & supporter ! */
albertwt
Expert
 
Posts: 607
Liked: 19 times
Joined: Thu Nov 05, 2009 12:24 pm
Location: Sydney, NSW

Re: Datastore Latency

Veeam Logoby Vitaliy S. » Mon Nov 10, 2014 12:47 pm 1 person likes this post

Albert, what other latency values do you have for the same period? What activity do you have scheduled for the same time when this alarm occurs?
Vitaliy S.
Veeam Software
 
Posts: 19564
Liked: 1104 times
Joined: Mon Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov

Re: Datastore Latency

Veeam Logoby albertwt » Mon Nov 10, 2014 10:24 pm

Vitaliy S. wrote:Albert, what other latency values do you have for the same period? What activity do you have scheduled for the same time when this alarm occurs?


See the screenshot below:

Image

I have already enabled the option to Suppress the alarm during the Veeam backup job runs but somehow I still get bombarded by the latency alerting.
--
/* Veeam software enthusiast user & supporter ! */
albertwt
Expert
 
Posts: 607
Liked: 19 times
Joined: Thu Nov 05, 2009 12:24 pm
Location: Sydney, NSW

Re: Datastore Latency

Veeam Logoby Vitaliy S. » Tue Nov 11, 2014 10:13 am

What suppress options have you selected for Datastore and Host disk latency alarms?
Vitaliy S.
Veeam Software
 
Posts: 19564
Liked: 1104 times
Joined: Mon Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov

Re: Datastore Latency

Veeam Logoby albertwt » Tue Dec 09, 2014 12:30 am

Vitaliy S. wrote:What suppress options have you selected for Datastore and Host disk latency alarms?


Vitaliy,

The only suppress option that I selected just the 1 hour as it is the maximum threshold value. is it possible to edit it to be like 2 or 4 hours instead ?
--
/* Veeam software enthusiast user & supporter ! */
albertwt
Expert
 
Posts: 607
Liked: 19 times
Joined: Thu Nov 05, 2009 12:24 pm
Location: Sydney, NSW

Re: Datastore Latency

Veeam Logoby Vitaliy S. » Tue Dec 09, 2014 10:33 am

Yes, that's possible to do in version 8, we have extended the list of available intervals. On the other hand, you may want to consider using time intervals to suppress the entire alarm during backup activity.
Vitaliy S.
Veeam Software
 
Posts: 19564
Liked: 1104 times
Joined: Mon Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov


Return to Veeam ONE



Who is online

Users browsing this forum: Regnor and 7 guests