-
- Novice
- Posts: 7
- Liked: never
- Joined: Mar 16, 2016 2:11 am
- Full Name: Navpreet Rakwal
- Contact:
Question about VM total disk latency
Hi
I just installed VEEAM one in our setup and all the alarms are configured with default settings.
I have been getting VM total disk latency warning and errors for many VMs. When I go to the vCenter server performance charts and look at vmdisk latency stats, I can see that there is latency spike at the time when warning/alert was generated in VEEAM one.
My question is do we have any threshold before we raise warning or alert in VEEAM one or VEEAM one will keep generating warning/alerts even if there is single I/O latency spike observed in vCenter server for that particular vmdisk.
-NP
I just installed VEEAM one in our setup and all the alarms are configured with default settings.
I have been getting VM total disk latency warning and errors for many VMs. When I go to the vCenter server performance charts and look at vmdisk latency stats, I can see that there is latency spike at the time when warning/alert was generated in VEEAM one.
My question is do we have any threshold before we raise warning or alert in VEEAM one or VEEAM one will keep generating warning/alerts even if there is single I/O latency spike observed in vCenter server for that particular vmdisk.
-NP
-
- Veteran
- Posts: 7328
- Liked: 781 times
- Joined: May 21, 2014 11:03 am
- Full Name: Nikita Shestakov
- Location: Prague
- Contact:
Re: Question about VM total disk latency
Hi,
Yes, there is a Time period option to set the interval for the alarm. So the alarm warns you if average performance during the interval is above the threshold.
Thanks!
Yes, there is a Time period option to set the interval for the alarm. So the alarm warns you if average performance during the interval is above the threshold.
Thanks!
-
- VP, Product Management
- Posts: 27377
- Liked: 2800 times
- Joined: Mar 30, 2009 9:13 am
- Full Name: Vitaliy Safarov
- Contact:
Re: Question about VM total disk latency
The default for most alarms is 15 minutes, so you don't need to tweak anything and will only be alerted when there is a high load for an extended period of time.
-
- Novice
- Posts: 7
- Liked: never
- Joined: Mar 16, 2016 2:11 am
- Full Name: Navpreet Rakwal
- Contact:
Re: Question about VM total disk latency
Thanks for your response.
If I go to alarm settings, I can see that there under rules tab, we have value time period setting and default value is 15 min.
Now, does that mean that alarm will be generated when average latency for that VM goes beyond 50ms for last 15 mins or latency for the VM was above 50ms for 15 mins ?
-NP
If I go to alarm settings, I can see that there under rules tab, we have value time period setting and default value is 15 min.
Now, does that mean that alarm will be generated when average latency for that VM goes beyond 50ms for last 15 mins or latency for the VM was above 50ms for 15 mins ?
-NP
-
- Veteran
- Posts: 7328
- Liked: 781 times
- Joined: May 21, 2014 11:03 am
- Full Name: Nikita Shestakov
- Location: Prague
- Contact:
Re: Question about VM total disk latency
The alarm will be generated when average latency for that VM goes beyond 50ms for last 15 mins
-
- Novice
- Posts: 7
- Liked: never
- Joined: Mar 16, 2016 2:11 am
- Full Name: Navpreet Rakwal
- Contact:
Re: Question about VM total disk latency
Thanks Nikita for your response.
If VEEAM one checks for the average latency for last 15 mins, they why when the alarm is triggered the latency no. shown for the alarm is not the average latency no. but it is the latest latency spike seen in the vCenter performance graph for a vmdisk ?
Also, as per my understanding, VEEAM one polls data from vCenter server on realtime basis and vCenter real time performance charts are updated every 20 secs so does that mean that VEEAM one does the analytic and computes 15 min average latency and then shows the alert ?
-NP
If VEEAM one checks for the average latency for last 15 mins, they why when the alarm is triggered the latency no. shown for the alarm is not the average latency no. but it is the latest latency spike seen in the vCenter performance graph for a vmdisk ?
Also, as per my understanding, VEEAM one polls data from vCenter server on realtime basis and vCenter real time performance charts are updated every 20 secs so does that mean that VEEAM one does the analytic and computes 15 min average latency and then shows the alert ?
-NP
-
- Veteran
- Posts: 7328
- Liked: 781 times
- Joined: May 21, 2014 11:03 am
- Full Name: Nikita Shestakov
- Location: Prague
- Contact:
Re: Question about VM total disk latency
Veeam ONE collects 20-second samples from ESXi and shows it during the day or so(depends on the settings), then it aggregates 20-second values to 5-minutes intervals and shows the average value and so on up to 2-hour averages.
For 15-minutes interval Veeam ONE takes 20 sec samples over 15 minutes and calculates the average value.
For 15-minutes interval Veeam ONE takes 20 sec samples over 15 minutes and calculates the average value.
-
- Novice
- Posts: 7
- Liked: never
- Joined: Mar 16, 2016 2:11 am
- Full Name: Navpreet Rakwal
- Contact:
Re: Question about VM total disk latency
Just one more thing I need to understand
For example - In vCenter server real time performance graph there was a latency spike of 100ms and after that it comes back to normal.
Now in VEEAM one, in case there is a warning generated for the same VM, I don't see the warning generated with average latency no. It will generate alert
"Datastore Highest Latency" (100.0 Milliseconds) is above a defined threshold (75.0 Milliseconds)"
Can someone explain me why we are reporting highest spike which was observed by the VM and not the average latency ?
-NP
For example - In vCenter server real time performance graph there was a latency spike of 100ms and after that it comes back to normal.
Now in VEEAM one, in case there is a warning generated for the same VM, I don't see the warning generated with average latency no. It will generate alert
"Datastore Highest Latency" (100.0 Milliseconds) is above a defined threshold (75.0 Milliseconds)"
Can someone explain me why we are reporting highest spike which was observed by the VM and not the average latency ?
-NP
-
- Enthusiast
- Posts: 28
- Liked: 5 times
- Joined: May 06, 2014 2:28 pm
- Full Name: Tom
- Contact:
Re: Question about VM total disk latency
We are observing this problem also in our lab.
It triggers the alarm on a one time spike, not the average in the specified time period.
vCenter 6.0 build 3018523
ESXi 5.5 build 3116895
Veeam ONE 9.0.0.2062
It triggers the alarm on a one time spike, not the average in the specified time period.
vCenter 6.0 build 3018523
ESXi 5.5 build 3116895
Veeam ONE 9.0.0.2062
VMCE certified
-
- Veteran
- Posts: 7328
- Liked: 781 times
- Joined: May 21, 2014 11:03 am
- Full Name: Nikita Shestakov
- Location: Prague
- Contact:
Re: Question about VM total disk latency
That`s a known issue for that particular alarm. Our R&D team is working on a fix.
You may contact technical support to get the fix as soon as it`s available.
Thanks!
You may contact technical support to get the fix as soon as it`s available.
Thanks!
-
- Novice
- Posts: 7
- Liked: never
- Joined: Mar 16, 2016 2:11 am
- Full Name: Navpreet Rakwal
- Contact:
Re: Question about VM total disk latency
Hello Nikita,
Do you have any Bug ID for the reference and also the exact problem description ?
-NP
Do you have any Bug ID for the reference and also the exact problem description ?
-NP
-
- Veteran
- Posts: 7328
- Liked: 781 times
- Joined: May 21, 2014 11:03 am
- Full Name: Nikita Shestakov
- Location: Prague
- Contact:
Re: Question about VM total disk latency
They should know the issue.
You can link them to this forum topic.
Thanks!
You can link them to this forum topic.
Thanks!
-
- Novice
- Posts: 7
- Liked: never
- Joined: Mar 16, 2016 2:11 am
- Full Name: Navpreet Rakwal
- Contact:
Re: Question about VM total disk latency
I did. They asked me to upload the logs.
I don't think logs are relevant for this case.
Case #01732408
I don't think logs are relevant for this case.
Case #01732408
-
- Veteran
- Posts: 7328
- Liked: 781 times
- Joined: May 21, 2014 11:03 am
- Full Name: Nikita Shestakov
- Location: Prague
- Contact:
Re: Question about VM total disk latency
Well, there is still a probability you have a different issue. So they need to make sure.
You may forward them the reference to an original support ticket 01686624 and the bug is tracked as Issue 68288
You may forward them the reference to an original support ticket 01686624 and the bug is tracked as Issue 68288
-
- Novice
- Posts: 7
- Liked: never
- Joined: Mar 16, 2016 2:11 am
- Full Name: Navpreet Rakwal
- Contact:
Re: Question about VM total disk latency
Hello Nikita,
Is it specific to some ESXi or vCenter server version ? Or it is a issue independent of vCenter/ESX version ?
NP
Is it specific to some ESXi or vCenter server version ? Or it is a issue independent of vCenter/ESX version ?
NP
-
- Veteran
- Posts: 7328
- Liked: 781 times
- Joined: May 21, 2014 11:03 am
- Full Name: Nikita Shestakov
- Location: Prague
- Contact:
Re: Question about VM total disk latency
I believe it`s an independent bug, since there is Veeam One v9 logic flaw for this particular alarm.
-
- Influencer
- Posts: 23
- Liked: 3 times
- Joined: Sep 23, 2016 1:31 pm
- Full Name: Jesus Martin
- Contact:
Re: Question about VM total disk latency
Hi Nikita,
I also noticed this alert showing up so much in our environment and happened onl when we upgraded to VeeamOne v9. Is there a fix on this? I watched the VMs on esxtop using 2 seconds sampling and though there are times that latency went over the threshold it comes back right away. I changed it to 30 minutes from the default 15, still getting it..then tried to change the threshold to 1 hour just to test but still happening. Is there a bug on v9?
Thanks.
I also noticed this alert showing up so much in our environment and happened onl when we upgraded to VeeamOne v9. Is there a fix on this? I watched the VMs on esxtop using 2 seconds sampling and though there are times that latency went over the threshold it comes back right away. I changed it to 30 minutes from the default 15, still getting it..then tried to change the threshold to 1 hour just to test but still happening. Is there a bug on v9?
Thanks.
-
- Veteran
- Posts: 7328
- Liked: 781 times
- Joined: May 21, 2014 11:03 am
- Full Name: Nikita Shestakov
- Location: Prague
- Contact:
Re: Question about VM total disk latency
Hi,
there was a bug with alarm notification frequency as you can read above.
Regarding the alarm trigger, if you doubt the alarm works correctly, please check Datastore Latency graph in Veeam ONE Monitor.
Thanks!
there was a bug with alarm notification frequency as you can read above.
Regarding the alarm trigger, if you doubt the alarm works correctly, please check Datastore Latency graph in Veeam ONE Monitor.
Thanks!
-
- Influencer
- Posts: 23
- Liked: 3 times
- Joined: Sep 23, 2016 1:31 pm
- Full Name: Jesus Martin
- Contact:
Re: Question about VM total disk latency
Aside from checking the datastore latency in VeeamOne monitor, when will this bug be fixed?
Thanks.
Thanks.
-
- Veteran
- Posts: 7328
- Liked: 781 times
- Joined: May 21, 2014 11:03 am
- Full Name: Nikita Shestakov
- Location: Prague
- Contact:
Re: Question about VM total disk latency
As far as I know it`s already fixed in the most recent version. What version are you at?
Do you experience the behavior when time period in alarm rule is 1 hour but alarms are still triggered every 15 minutes?
Do you experience the behavior when time period in alarm rule is 1 hour but alarms are still triggered every 15 minutes?
-
- Influencer
- Posts: 23
- Liked: 3 times
- Joined: Sep 23, 2016 1:31 pm
- Full Name: Jesus Martin
- Contact:
Re: Question about VM total disk latency
That's weird. In VeeamOne7 we didn't have an issue like that, just happened when straight upgrade to v9. I applied also Update1 which version is 9.0.0.2088. The issue is why an alarm is getting triggered when the latency was not over the threshold in 15 minutes. Correct me if I'm wrong, my understanding is that if the latency goes over the threshold for steady 15 minutes the alarm will be triggered.
-
- Veteran
- Posts: 7328
- Liked: 781 times
- Joined: May 21, 2014 11:03 am
- Full Name: Nikita Shestakov
- Location: Prague
- Contact:
Re: Question about VM total disk latency
You are correct. That should work with no issues.sonic23 wrote:my understanding is that if the latency goes over the threshold for steady 15 minutes the alarm will be triggered
We are discussing 2 separate things:
1. Fairness of the alarm triggering
2. Frequency of the alarm triggering
The mentioned bug is related to the triggering frequency, while it there was no bug about its fairness.
-
- Influencer
- Posts: 23
- Liked: 3 times
- Joined: Sep 23, 2016 1:31 pm
- Full Name: Jesus Martin
- Contact:
Re: Question about VM total disk latency
I have an alarm a while back, the alarm was triggered at 1:07 from one of the VM which is not being used that much, the Write latency was 122ms and just a spike and it went to almost 0ms until now. I thought the alarm will be triggered if the latency is steady above 75 ms but in this case its not. I'm not expecting an alarm but we did get it, this is the same scenario I noticed for the whole environment.
-
- Veteran
- Posts: 7328
- Liked: 781 times
- Joined: May 21, 2014 11:03 am
- Full Name: Nikita Shestakov
- Location: Prague
- Contact:
Re: Question about VM total disk latency
Well, if had spike of 122ms, there is a probability average values was above the threshold =75 ms for 15 minutes. You can check it in Veeam ONE Monitor or Custom Performance Data report. To make sure Veeam ONE shoes correct results you can double check it with vSphere client.
-
- Lurker
- Posts: 1
- Liked: never
- Joined: Dec 15, 2017 11:05 am
- Full Name: Alex Canham
- Contact:
Re: Question about VM total disk latency
I am having this issue with Veeam One 9.5.0.3254. The alerts do not match metrics on VMWare VMs or host.
Does anyone know when this will be fixed?
Does anyone know when this will be fixed?
-
- Veteran
- Posts: 7328
- Liked: 781 times
- Joined: May 21, 2014 11:03 am
- Full Name: Nikita Shestakov
- Location: Prague
- Contact:
Re: Question about VM total disk latency
Hello Alex and welcome to the community!
Do you mean notification frequency bug or issues with alarm threshold?
Thanks!
Do you mean notification frequency bug or issues with alarm threshold?
Thanks!
-
- Influencer
- Posts: 22
- Liked: 1 time
- Joined: Mar 14, 2014 7:22 pm
- Contact:
Re: Question about VM total disk latency
Would changing the Aggregation from Max to Avg solve his spike issue?
I think the default is Max. Can someone define what aggregation is for this metric or should be set at?
Thanks
I think the default is Max. Can someone define what aggregation is for this metric or should be set at?
Thanks
-
- Veteran
- Posts: 7328
- Liked: 781 times
- Joined: May 21, 2014 11:03 am
- Full Name: Nikita Shestakov
- Location: Prague
- Contact:
Re: Question about VM total disk latency
Yes, it should.
Max means you check spikes, avg. that you work with aggregated (average) performance values.
Max means you check spikes, avg. that you work with aggregated (average) performance values.
Who is online
Users browsing this forum: No registered users and 7 guests