Odd latency reporting

[mlange]

I've been getting a lot of high datastore latency notifications recently, often reporting into the 100's of milliseconds. However we don't actually seem to be experiencing any latency of that nature. Looking at the datastores themselves the highest peaks I see are 20 ms, and those aren't even happening during the times Veeam One is reporting. We're running Veeam One 9.5 Update 3, ESXi 6.5 and the datastores are on a Nimble array. Is this a known issue or is there a patch or something I'm missing?

[Vitaliy S.]

Hi Matt,

This is not a known issue, any metric reported by Veeam ONE is retrieved from a vCenter Server. Can you please post screenshots of these latency peaks taken from the vSphere Client and Monitor Client?

Thanks!

[mlange]

Ok, so maybe I don't understand how any of this is supposed to work, I guess we'll find out.

Here is a notification email I received this morning:


This is what VSphere shows:


This is what the Nimble shows:


This is what the Monitor shows for the VM in question:


And what the Monitor shows for that datastore in general:


I'm new to using VeeamOne, and I didn't set ours up so this could just be me not understanding how it supposed to work the way it is configured, or it could be misconfigured and I just don't understand where or why.

[Shestakov]

Hello Matt,
Do first two graphs represent data from 27dec?

[mlange]

Yes, all of the images are from the 27th.

[Shestakov]

That doesn't look expected. I would suggest to contact Veeam support.
The only reason why alarm says 57msec, while Veeam ONE Monitor shows around 8msec is the data aggregation.

[ViciousXUSMC]

I think I am having the same issue. When I first installed Veeam One I was inundated with latency notices.
I never had vCenter tell me anything was wrong, no users see anything wrong. I just figured I had not tuned my monitor yet and the default values were too strict.

I turned that alert off actually but may turn it back on again and see if I get the same crazy numbers.

[Vitaliy S.]

Hi Patrick,

Turning off the alarm is not recommended, however, you can indeed tune the threshold to match your environment behavior. Go to the host or datastore tab and select the latency counter to be displayed on the chart. Once you do this, verify what is the average latency, add 10%/20% and set the new threshold in the alarm settings.

Thanks!

[Shestakov]

Also notice that the "normal latency" depends on hardware. For older datastores higher latency may be expected.
So you may want to set different thresholds for different datastores.

[KPS]

Hi!

I have the same problem with VeeamONE and vSphere 6.7U1:
Latency is shown as too high - but inconsistent:

- No latency on virtual disks
- No latency on datastore
- HIGH latency on VM-datastore-read-latency


although, it is a large, local full-flash datastore with only ONE (idle) VM.


Strange...

[Shestakov]

Hi KPS,
as was suggested in a parallel topic, please contact Veeam technical support for the issue investigation.
Thanks!