Question about VM total disk latency

[DonnieBrasco]

Hi

I just installed VEEAM one in our setup and all the alarms are configured with default settings.

I have been getting VM total disk latency warning and errors for many VMs. When I go to the vCenter server performance charts and look at vmdisk latency stats, I can see that there is latency spike at the time when warning/alert was generated in VEEAM one.

My question is do we have any threshold before we raise warning or alert in VEEAM one or VEEAM one will keep generating warning/alerts even if there is single I/O latency spike observed in vCenter server for that particular vmdisk.

-NP

[Shestakov]

Hi,
Yes, there is a Time period option to set the interval for the alarm. So the alarm warns you if average performance during the interval is above the threshold.
Thanks!

[Vitaliy S.]

The default for most alarms is 15 minutes, so you don't need to tweak anything and will only be alerted when there is a high load for an extended period of time.

[DonnieBrasco]

Thanks for your response.

If I go to alarm settings, I can see that there under rules tab, we have value time period setting and default value is 15 min.

Now, does that mean that alarm will be generated when average latency for that VM goes beyond 50ms for last 15 mins or latency for the VM was above 50ms for 15 mins ?

-NP

[Shestakov]

The alarm will be generated when average latency for that VM goes beyond 50ms for last 15 mins

[DonnieBrasco]

Thanks Nikita for your response.

If VEEAM one checks for the average latency for last 15 mins, they why when the alarm is triggered the latency no. shown for the alarm is not the average latency no. but it is the latest latency spike seen in the vCenter performance graph for a vmdisk ?

Also, as per my understanding, VEEAM one polls data from vCenter server on realtime basis and vCenter real time performance charts are updated every 20 secs so does that mean that VEEAM one does the analytic and computes 15 min average latency and then shows the alert ?

-NP

[Shestakov]

Veeam ONE collects 20-second samples from ESXi and shows it during the day or so(depends on the settings), then it aggregates 20-second values to 5-minutes intervals and shows the average value and so on up to 2-hour averages.
For 15-minutes interval Veeam ONE takes 20 sec samples over 15 minutes and calculates the average value.

[DonnieBrasco]

Just one more thing I need to understand

For example - In vCenter server real time performance graph there was a latency spike of 100ms and after that it comes back to normal.
Now in VEEAM one, in case there is a warning generated for the same VM, I don't see the warning generated with average latency no. It will generate alert

"Datastore Highest Latency" (100.0 Milliseconds) is above a defined threshold (75.0 Milliseconds)"

Can someone explain me why we are reporting highest spike which was observed by the VM and not the average latency ?

-NP

[v.tom]

We are observing this problem also in our lab.
It triggers the alarm on a one time spike, not the average in the specified time period.

vCenter 6.0 build 3018523
ESXi 5.5 build 3116895
Veeam ONE 9.0.0.2062

[Shestakov]

That`s a known issue for that particular alarm. Our R&D team is working on a fix.
You may contact technical support to get the fix as soon as it`s available.
Thanks!

[DonnieBrasco]

Hello Nikita,

Do you have any Bug ID for the reference and also the exact problem description ?

-NP

[Shestakov]

They should know the issue.
You can link them to this forum topic.
Thanks!

[DonnieBrasco]

I did. They asked me to upload the logs.
I don't think logs are relevant for this case.

Case #01732408

[Shestakov]

Well, there is still a probability you have a different issue. So they need to make sure.
You may forward them the reference to an original support ticket 01686624 and the bug is tracked as Issue 68288

[DonnieBrasco]

Hello Nikita,

Is it specific to some ESXi or vCenter server version ? Or it is a issue independent of vCenter/ESX version ?

NP

[Shestakov]

I believe it`s an independent bug, since there is Veeam One v9 logic flaw for this particular alarm.

[sonic23]

Hi Nikita,

I also noticed this alert showing up so much in our environment and happened onl when we upgraded to VeeamOne v9. Is there a fix on this? I watched the VMs on esxtop using 2 seconds sampling and though there are times that latency went over the threshold it comes back right away. I changed it to 30 minutes from the default 15, still getting it..then tried to change the threshold to 1 hour just to test but still happening. Is there a bug on v9?

Thanks.

[Shestakov]

Hi,
there was a bug with alarm notification frequency as you can read above.
Regarding the alarm trigger, if you doubt the alarm works correctly, please check Datastore Latency graph in Veeam ONE Monitor.
Thanks!

[sonic23]

Aside from checking the datastore latency in VeeamOne monitor, when will this bug be fixed?

Thanks.

[Shestakov]

As far as I know it`s already fixed in the most recent version. What version are you at?
Do you experience the behavior when time period in alarm rule is 1 hour but alarms are still triggered every 15 minutes?

[sonic23]

That's weird. In VeeamOne7 we didn't have an issue like that, just happened when straight upgrade to v9. I applied also Update1 which version is 9.0.0.2088. The issue is why an alarm is getting triggered when the latency was not over the threshold in 15 minutes. Correct me if I'm wrong, my understanding is that if the latency goes over the threshold for steady 15 minutes the alarm will be triggered.

[Shestakov]

my understanding is that if the latency goes over the threshold for steady 15 minutes the alarm will be triggered

You are correct. That should work with no issues.

We are discussing 2 separate things:
1. Fairness of the alarm triggering
2. Frequency of the alarm triggering

The mentioned bug is related to the triggering frequency, while it there was no bug about its fairness.

[sonic23]

I have an alarm a while back, the alarm was triggered at 1:07 from one of the VM which is not being used that much, the Write latency was 122ms and just a spike and it went to almost 0ms until now. I thought the alarm will be triggered if the latency is steady above 75 ms but in this case its not. I'm not expecting an alarm but we did get it, this is the same scenario I noticed for the whole environment.

[Shestakov]

Well, if had spike of 122ms, there is a probability average values was above the threshold =75 ms for 15 minutes. You can check it in Veeam ONE Monitor or Custom Performance Data report. To make sure Veeam ONE shoes correct results you can double check it with vSphere client.

[AlexCanham]

I am having this issue with Veeam One 9.5.0.3254. The alerts do not match metrics on VMWare VMs or host.
Does anyone know when this will be fixed?

[Shestakov]

Hello Alex and welcome to the community!
Do you mean notification frequency bug or issues with alarm threshold?
Thanks!

[Rascii]

Would changing the Aggregation from Max to Avg solve his spike issue?


I think the default is Max. Can someone define what aggregation is for this metric or should be set at?

Thanks

[Shestakov]

Yes, it should.
Max means you check spikes, avg. that you work with aggregated (average) performance values.