Monitoring and reporting for Veeam Data Platform
Post Reply
DonnieBrasco
Novice
Posts: 7
Liked: never
Joined: Mar 16, 2016 2:11 am
Full Name: Navpreet Rakwal
Contact:

Question about VM total disk latency

Post by DonnieBrasco »

Hi

I just installed VEEAM one in our setup and all the alarms are configured with default settings.

I have been getting VM total disk latency warning and errors for many VMs. When I go to the vCenter server performance charts and look at vmdisk latency stats, I can see that there is latency spike at the time when warning/alert was generated in VEEAM one.

My question is do we have any threshold before we raise warning or alert in VEEAM one or VEEAM one will keep generating warning/alerts even if there is single I/O latency spike observed in vCenter server for that particular vmdisk.

-NP
Shestakov
Veteran
Posts: 7328
Liked: 781 times
Joined: May 21, 2014 11:03 am
Full Name: Nikita Shestakov
Location: Prague
Contact:

Re: Question about VM total disk latency

Post by Shestakov »

Hi,
Yes, there is a Time period option to set the interval for the alarm. So the alarm warns you if average performance during the interval is above the threshold.
Thanks!
Vitaliy S.
VP, Product Management
Posts: 27377
Liked: 2800 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: Question about VM total disk latency

Post by Vitaliy S. »

The default for most alarms is 15 minutes, so you don't need to tweak anything and will only be alerted when there is a high load for an extended period of time.
DonnieBrasco
Novice
Posts: 7
Liked: never
Joined: Mar 16, 2016 2:11 am
Full Name: Navpreet Rakwal
Contact:

Re: Question about VM total disk latency

Post by DonnieBrasco »

Thanks for your response.

If I go to alarm settings, I can see that there under rules tab, we have value time period setting and default value is 15 min.

Now, does that mean that alarm will be generated when average latency for that VM goes beyond 50ms for last 15 mins or latency for the VM was above 50ms for 15 mins ?

-NP
Shestakov
Veteran
Posts: 7328
Liked: 781 times
Joined: May 21, 2014 11:03 am
Full Name: Nikita Shestakov
Location: Prague
Contact:

Re: Question about VM total disk latency

Post by Shestakov »

The alarm will be generated when average latency for that VM goes beyond 50ms for last 15 mins
DonnieBrasco
Novice
Posts: 7
Liked: never
Joined: Mar 16, 2016 2:11 am
Full Name: Navpreet Rakwal
Contact:

Re: Question about VM total disk latency

Post by DonnieBrasco »

Thanks Nikita for your response.

If VEEAM one checks for the average latency for last 15 mins, they why when the alarm is triggered the latency no. shown for the alarm is not the average latency no. but it is the latest latency spike seen in the vCenter performance graph for a vmdisk ?

Also, as per my understanding, VEEAM one polls data from vCenter server on realtime basis and vCenter real time performance charts are updated every 20 secs so does that mean that VEEAM one does the analytic and computes 15 min average latency and then shows the alert ?

-NP
Shestakov
Veteran
Posts: 7328
Liked: 781 times
Joined: May 21, 2014 11:03 am
Full Name: Nikita Shestakov
Location: Prague
Contact:

Re: Question about VM total disk latency

Post by Shestakov » 1 person likes this post

Veeam ONE collects 20-second samples from ESXi and shows it during the day or so(depends on the settings), then it aggregates 20-second values to 5-minutes intervals and shows the average value and so on up to 2-hour averages.
For 15-minutes interval Veeam ONE takes 20 sec samples over 15 minutes and calculates the average value.
DonnieBrasco
Novice
Posts: 7
Liked: never
Joined: Mar 16, 2016 2:11 am
Full Name: Navpreet Rakwal
Contact:

Re: Question about VM total disk latency

Post by DonnieBrasco »

Just one more thing I need to understand

For example - In vCenter server real time performance graph there was a latency spike of 100ms and after that it comes back to normal.
Now in VEEAM one, in case there is a warning generated for the same VM, I don't see the warning generated with average latency no. It will generate alert

"Datastore Highest Latency" (100.0 Milliseconds) is above a defined threshold (75.0 Milliseconds)"

Can someone explain me why we are reporting highest spike which was observed by the VM and not the average latency ?

-NP
v.tom
Enthusiast
Posts: 28
Liked: 5 times
Joined: May 06, 2014 2:28 pm
Full Name: Tom
Contact:

Re: Question about VM total disk latency

Post by v.tom »

We are observing this problem also in our lab.
It triggers the alarm on a one time spike, not the average in the specified time period.

vCenter 6.0 build 3018523
ESXi 5.5 build 3116895
Veeam ONE 9.0.0.2062
VMCE certified
Shestakov
Veteran
Posts: 7328
Liked: 781 times
Joined: May 21, 2014 11:03 am
Full Name: Nikita Shestakov
Location: Prague
Contact:

Re: Question about VM total disk latency

Post by Shestakov »

That`s a known issue for that particular alarm. Our R&D team is working on a fix.
You may contact technical support to get the fix as soon as it`s available.
Thanks!
DonnieBrasco
Novice
Posts: 7
Liked: never
Joined: Mar 16, 2016 2:11 am
Full Name: Navpreet Rakwal
Contact:

Re: Question about VM total disk latency

Post by DonnieBrasco »

Hello Nikita,

Do you have any Bug ID for the reference and also the exact problem description ?

-NP
Shestakov
Veteran
Posts: 7328
Liked: 781 times
Joined: May 21, 2014 11:03 am
Full Name: Nikita Shestakov
Location: Prague
Contact:

Re: Question about VM total disk latency

Post by Shestakov »

They should know the issue.
You can link them to this forum topic.
Thanks!
DonnieBrasco
Novice
Posts: 7
Liked: never
Joined: Mar 16, 2016 2:11 am
Full Name: Navpreet Rakwal
Contact:

Re: Question about VM total disk latency

Post by DonnieBrasco »

I did. They asked me to upload the logs.
I don't think logs are relevant for this case.

Case #01732408
Shestakov
Veteran
Posts: 7328
Liked: 781 times
Joined: May 21, 2014 11:03 am
Full Name: Nikita Shestakov
Location: Prague
Contact:

Re: Question about VM total disk latency

Post by Shestakov »

Well, there is still a probability you have a different issue. So they need to make sure.
You may forward them the reference to an original support ticket 01686624 and the bug is tracked as Issue 68288
DonnieBrasco
Novice
Posts: 7
Liked: never
Joined: Mar 16, 2016 2:11 am
Full Name: Navpreet Rakwal
Contact:

Re: Question about VM total disk latency

Post by DonnieBrasco »

Hello Nikita,

Is it specific to some ESXi or vCenter server version ? Or it is a issue independent of vCenter/ESX version ?

NP
Shestakov
Veteran
Posts: 7328
Liked: 781 times
Joined: May 21, 2014 11:03 am
Full Name: Nikita Shestakov
Location: Prague
Contact:

Re: Question about VM total disk latency

Post by Shestakov »

I believe it`s an independent bug, since there is Veeam One v9 logic flaw for this particular alarm.
sonic23
Influencer
Posts: 23
Liked: 3 times
Joined: Sep 23, 2016 1:31 pm
Full Name: Jesus Martin
Contact:

Re: Question about VM total disk latency

Post by sonic23 »

Hi Nikita,

I also noticed this alert showing up so much in our environment and happened onl when we upgraded to VeeamOne v9. Is there a fix on this? I watched the VMs on esxtop using 2 seconds sampling and though there are times that latency went over the threshold it comes back right away. I changed it to 30 minutes from the default 15, still getting it..then tried to change the threshold to 1 hour just to test but still happening. Is there a bug on v9?

Thanks.
Shestakov
Veteran
Posts: 7328
Liked: 781 times
Joined: May 21, 2014 11:03 am
Full Name: Nikita Shestakov
Location: Prague
Contact:

Re: Question about VM total disk latency

Post by Shestakov »

Hi,
there was a bug with alarm notification frequency as you can read above.
Regarding the alarm trigger, if you doubt the alarm works correctly, please check Datastore Latency graph in Veeam ONE Monitor.
Thanks!
sonic23
Influencer
Posts: 23
Liked: 3 times
Joined: Sep 23, 2016 1:31 pm
Full Name: Jesus Martin
Contact:

Re: Question about VM total disk latency

Post by sonic23 »

Aside from checking the datastore latency in VeeamOne monitor, when will this bug be fixed?

Thanks.
Shestakov
Veteran
Posts: 7328
Liked: 781 times
Joined: May 21, 2014 11:03 am
Full Name: Nikita Shestakov
Location: Prague
Contact:

Re: Question about VM total disk latency

Post by Shestakov »

As far as I know it`s already fixed in the most recent version. What version are you at?
Do you experience the behavior when time period in alarm rule is 1 hour but alarms are still triggered every 15 minutes?
sonic23
Influencer
Posts: 23
Liked: 3 times
Joined: Sep 23, 2016 1:31 pm
Full Name: Jesus Martin
Contact:

Re: Question about VM total disk latency

Post by sonic23 »

That's weird. In VeeamOne7 we didn't have an issue like that, just happened when straight upgrade to v9. I applied also Update1 which version is 9.0.0.2088. The issue is why an alarm is getting triggered when the latency was not over the threshold in 15 minutes. Correct me if I'm wrong, my understanding is that if the latency goes over the threshold for steady 15 minutes the alarm will be triggered.
Shestakov
Veteran
Posts: 7328
Liked: 781 times
Joined: May 21, 2014 11:03 am
Full Name: Nikita Shestakov
Location: Prague
Contact:

Re: Question about VM total disk latency

Post by Shestakov »

sonic23 wrote:my understanding is that if the latency goes over the threshold for steady 15 minutes the alarm will be triggered
You are correct. That should work with no issues.

We are discussing 2 separate things:
1. Fairness of the alarm triggering
2. Frequency of the alarm triggering

The mentioned bug is related to the triggering frequency, while it there was no bug about its fairness.
sonic23
Influencer
Posts: 23
Liked: 3 times
Joined: Sep 23, 2016 1:31 pm
Full Name: Jesus Martin
Contact:

Re: Question about VM total disk latency

Post by sonic23 »

I have an alarm a while back, the alarm was triggered at 1:07 from one of the VM which is not being used that much, the Write latency was 122ms and just a spike and it went to almost 0ms until now. I thought the alarm will be triggered if the latency is steady above 75 ms but in this case its not. I'm not expecting an alarm but we did get it, this is the same scenario I noticed for the whole environment.
Shestakov
Veteran
Posts: 7328
Liked: 781 times
Joined: May 21, 2014 11:03 am
Full Name: Nikita Shestakov
Location: Prague
Contact:

Re: Question about VM total disk latency

Post by Shestakov »

Well, if had spike of 122ms, there is a probability average values was above the threshold =75 ms for 15 minutes. You can check it in Veeam ONE Monitor or Custom Performance Data report. To make sure Veeam ONE shoes correct results you can double check it with vSphere client.
AlexCanham
Lurker
Posts: 1
Liked: never
Joined: Dec 15, 2017 11:05 am
Full Name: Alex Canham
Contact:

Re: Question about VM total disk latency

Post by AlexCanham »

I am having this issue with Veeam One 9.5.0.3254. The alerts do not match metrics on VMWare VMs or host.
Does anyone know when this will be fixed?
Shestakov
Veteran
Posts: 7328
Liked: 781 times
Joined: May 21, 2014 11:03 am
Full Name: Nikita Shestakov
Location: Prague
Contact:

Re: Question about VM total disk latency

Post by Shestakov »

Hello Alex and welcome to the community!
Do you mean notification frequency bug or issues with alarm threshold?
Thanks!
Rascii
Influencer
Posts: 22
Liked: 1 time
Joined: Mar 14, 2014 7:22 pm
Contact:

Re: Question about VM total disk latency

Post by Rascii »

Would changing the Aggregation from Max to Avg solve his spike issue?


I think the default is Max. Can someone define what aggregation is for this metric or should be set at?

Thanks
Shestakov
Veteran
Posts: 7328
Liked: 781 times
Joined: May 21, 2014 11:03 am
Full Name: Nikita Shestakov
Location: Prague
Contact:

Re: Question about VM total disk latency

Post by Shestakov »

Yes, it should.
Max means you check spikes, avg. that you work with aggregated (average) performance values.
Post Reply

Who is online

Users browsing this forum: No registered users and 7 guests