Datastore Write Latency much too High from VeeamOne !!!

[WinstonWolf]

Hello ,
I have every Morning 60 Mails with Datastore Write Latency Warning or Error Messages from Veeam One Site .
Like this :


But on the other Side we use the Netapp VA OCPM ( OnCommand Performance Manager ) & the OCUM ( On Command Unified Manager )
This are special VAs from Netapp which Monitor the Datastores and Volumes also. And sorry Veeam, i trust this Netapp own tools more then Veeam One

And the Output from this Tools is for the same Datastore RZ1_vmware_ds10 this :


As you can see the Disk write latency is only on the highest peaks and only for an short period 20.0 ms .

What Veeam One see there ??!?!?

Any Statement for this ?

[Shestakov]

Hello,
Could you check what type of aggregation do you have set up in the triggered alarm(Avg/Max)?
I believe that`s why there are different values.

Usually monitoring tools have to aggregate data in 5 or 20 minutes averages and products may have different time frames.
Have you checked this metric in Veeam ONE Monitor?

Thanks!

[Mike Resseler]

Hi Winston,

Could you compare the data from Veeam ONE also with the data you see in vCenter? Because that is where we grab our information. I am not sure (but don't know either so ) that we are comparing apples with apples here. What metric are you looking at with OCMP? (I don't know what MS / OP means and it might be a different metric.)

Thanks
Mike

[Vitaliy S.]

Yes, Mike is correct. Veeam ONE cannot make up data, we collect it via vSphere API, so if there is a spike reported in Veeam ONE, it will also be reflected in the vSphere Client.

[WinstonWolf]

Hello again ,

Here is the Screenshot from my Alarm Rules from "Datastore Write Latency"


Can it be better with Aggregation : "Avg" ?

Here is an Screenshot from Write Latency from the Vcenter Server , as you can see there are some Spikes but not for long time ( they can produced by Storage Snapshots maybe )



That means for me to Switch the default Setting from Aggregation max to Aggregation Avg . or ?

Thanks
Michael

Thanks
Michael

[WinstonWolf]

Hello ,

Now i think Veeam One runs totaly wrong .
At the Moment i become this Warning ( Error ) Mail from Veeam One which told me as follows :



But when i look on my Vcenter Server the the Performance Graph from this Datastore shows this :



Please Veeam , can u explain it for me .

Thanks

[Shestakov]

Hello Michael,

The alarm has been triggered because you`ve set aggregation=max. So once you had a spike higher than the threshold, alarm worked.
Regarding the last screenshot, could you also check values in Veeam ONE Monitor for the same time frame? If you think that Veeam ONE doesn`t work correctly, please contact Veeam support team for the detailed investigation. The will check if aggregation, object and metric are chosen correctly. Once you open a case, please let us know your case ID.

Thanks!

[WinstonWolf]

My First Stepp now is to completely uninstall and reinstall Veeam One . I also delete the Veeam One Database .
Then i observe if the Events comes again . Its not so much work

[Shestakov]

Well... Sounds more time-consuming than a call to the support team

[Vitaliy S.]

And re-installing the data collection engine cannot fix this (if the issue exists)

[bsylvain]

I have the same "inconvenience" from Veeam One.

MAX seems to be the default parameter for this alarm (read/write latency).
I tried to switch to avg but it doesn't seems to keep the setting (if I go back in the alarm definition it's back to MAX), I have to create a custom alarm and disable the defaults ones. Is this normal?

Thanks.
Sylvain

[Vitaliy S.]

Hi Sylvain,

If the new setting is not preserved in the alarms configuration, then please open a support ticket and let our team take a look at this. Based on your description, it sound like a bug.

EDIT: Just checked it with our support team and they have a fix for this problem. Please ask for Update 1 RC to fix this issue.

Thanks!

[bsylvain]

Hi Vitaliy,

Thanks, I will wait for official release then, no hurry.

[Vitaliy S.]

Ok, it should be a long wait. We are aiming to make it GA soon.

[bsylvain]

Hi

I forgot to follow up, but of course the update 1 fixed the problem. I have been able to set the alarms to AVG, no more annoyance.

Thanks

Sylvain

[mattiamigliorati]

Hi

I know this is an old post, but after 1 year I'm digging into the same issue..
I have Veeam 9.5 Up 3 and vSphere 6.5 Up1. The storage is a Netapp and I have OCUM and OCPM working.
I'm monitoring datastore latency and another problematic parameter "VM total disk latency".
What I saw is that during the night are latencies going high, but this is acceptable as backup job are working.
Sometimes during working hours there are latency spikes and I'm troubleshooting this.
I know that Veeam ONE is collecting everything from vCenter.

What I'm searching for is:
Can I completely trust what I see in Veeam ONE or should I taking into account others thing to have irrefutable value ?

Best regards

Mattia

[Shestakov]

Hi Mattia,
Yes you can.
You are right that Veeam ONE collects performance data from vSphere. That's the way to make sure it's correct.

Thanks!

[mattiamigliorati]

Thank you Shestakov for the quick response.

One more question.
As I found into this guide https://docs.vmware.com/en/VMware-vSphe ... -guide.pdf , on page 98, this statement:
The value for the deviceLatency data counter is greater than 15ms indicates there are probably problems with the storage array

As you know 15ms is by far more less than what Veeam ONE and other monitoring programs set a limit for performance issue.

Isn't it true?

Regards

Mattia

[Shestakov]

I don't remember the defaults we have, but you are probably correct. With development of better performing hardware our defaults may become outdated.
I'll research it, thanks for the heads up!

[TheQ]

After migrating from vSphere 5.5 to 6.5 and changing ESX hardware from HP ProLiant Gen8 to Gen10 we receive daily alarm messages about VM total disk latency. As described in the early posts of this thread these spikes can not be retrieved or matched in vCenter performance monitoring. As we did not change our VEEAM One installation during migration we are wondering why everything was fine when running vSphere 5.5??? Of course we have updated all hard- and software components to the latest software and firmware releases and changed aggregation in alarm settings from MAX to AVG for VM total disk latency.

Is this issue still under investigation? Anyone seeing same or similar problems?

[Shestakov]

Hi Frank and welcome to the community!
Do you have support case opened? If yes, please provide its number.
Thanks

[TheQ]

Not yet. Will let you know asap

[Shestakov]

By the way, what product version are you at?

[TheQ]

currrent version 9.5.0.3801 includes Update from end of 2017

[TheQ]

We now have a case number #03184720

[Shestakov]

Thanks Frank, as far as I remember there was a hotfix for such a case, but it became a part of v9.5 update 1. So it's strange you have an issue in update 3.
Please keep working with the support team as they will make the investigation.

[TheQ]

Quite impressive how quick support and community is reacting on this issue. GOOD JOB everyone!

Today we received an confirmation that we have an abnormal behaviour. A Workaround has been offered by support team. Their idea is to change the alarm trigger config in database.

We already applied this workaround. Let's wait and see if this helps.

[Shestakov]

Great to know, thanks!
I've shared your positive feedback with the support engineers.

[TheQ]

Morning folks,

It seems as if the workaround is fine for us.

We did not receive any alarms about disk latency since 4 days although we had some IO intensive workloads with small latency spikes.

[Shestakov]

Thanks for the update, Frank!