Real-time performance monitoring and troubleshooting
WinstonWolf
Veteran
Posts: 284
Liked: 11 times
Joined: Jan 06, 2011 8:33 am
Contact:

Datastore Write Latency much too High from VeeamOne !!!

Post by WinstonWolf »

Hello ,
I have every Morning 60 Mails with Datastore Write Latency Warning or Error Messages from Veeam One Site .
Like this :
Image

But on the other Side we use the Netapp VA OCPM ( OnCommand Performance Manager ) & the OCUM ( On Command Unified Manager )
This are special VAs from Netapp which Monitor the Datastores and Volumes also. And sorry Veeam, i trust this Netapp own tools more then Veeam One ;-)

And the Output from this Tools is for the same Datastore RZ1_vmware_ds10 this :
Image

As you can see the Disk write latency is only on the highest peaks and only for an short period 20.0 ms .

What Veeam One see there ??!?!?

Any Statement for this ?
Shestakov
Veteran
Posts: 7328
Liked: 781 times
Joined: May 21, 2014 11:03 am
Full Name: Nikita Shestakov
Location: Prague
Contact:

Re: Datastore Write Latency much too High from VeeamOne !!!

Post by Shestakov »

Hello,
Could you check what type of aggregation do you have set up in the triggered alarm(Avg/Max)?
I believe that`s why there are different values.

Usually monitoring tools have to aggregate data in 5 or 20 minutes averages and products may have different time frames.
Have you checked this metric in Veeam ONE Monitor?

Thanks!
Mike Resseler
Product Manager
Posts: 8044
Liked: 1263 times
Joined: Feb 08, 2013 3:08 pm
Full Name: Mike Resseler
Location: Belgium
Contact:

Re: Datastore Write Latency much too High from VeeamOne !!!

Post by Mike Resseler »

Hi Winston,

Could you compare the data from Veeam ONE also with the data you see in vCenter? Because that is where we grab our information. I am not sure (but don't know either so ;-)) that we are comparing apples with apples here. What metric are you looking at with OCMP? (I don't know what MS / OP means and it might be a different metric.)

Thanks
Mike
Vitaliy S.
VP, Product Management
Posts: 27055
Liked: 2710 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: Datastore Write Latency much too High from VeeamOne !!!

Post by Vitaliy S. »

Yes, Mike is correct. Veeam ONE cannot make up data, we collect it via vSphere API, so if there is a spike reported in Veeam ONE, it will also be reflected in the vSphere Client.
WinstonWolf
Veteran
Posts: 284
Liked: 11 times
Joined: Jan 06, 2011 8:33 am
Contact:

Re: Datastore Write Latency much too High from VeeamOne !!!

Post by WinstonWolf »

Hello again ,

Here is the Screenshot from my Alarm Rules from "Datastore Write Latency"
Image

Can it be better with Aggregation : "Avg" ?

Here is an Screenshot from Write Latency from the Vcenter Server , as you can see there are some Spikes but not for long time ( they can produced by Storage Snapshots maybe )

Image

That means for me to Switch the default Setting from Aggregation max to Aggregation Avg . or ?

Thanks
Michael

Thanks
Michael
WinstonWolf
Veteran
Posts: 284
Liked: 11 times
Joined: Jan 06, 2011 8:33 am
Contact:

Re: Datastore Write Latency much too High from VeeamOne !!!

Post by WinstonWolf »

Hello ,

Now i think Veeam One runs totaly wrong .
At the Moment i become this Warning ( Error ) Mail from Veeam One which told me as follows :

Image

But when i look on my Vcenter Server the the Performance Graph from this Datastore shows this :

Image

Please Veeam , can u explain it for me .

Thanks
Shestakov
Veteran
Posts: 7328
Liked: 781 times
Joined: May 21, 2014 11:03 am
Full Name: Nikita Shestakov
Location: Prague
Contact:

Re: Datastore Write Latency much too High from VeeamOne !!!

Post by Shestakov »

Hello Michael,

The alarm has been triggered because you`ve set aggregation=max. So once you had a spike higher than the threshold, alarm worked.
Regarding the last screenshot, could you also check values in Veeam ONE Monitor for the same time frame? If you think that Veeam ONE doesn`t work correctly, please contact Veeam support team for the detailed investigation. The will check if aggregation, object and metric are chosen correctly. Once you open a case, please let us know your case ID.

Thanks!
WinstonWolf
Veteran
Posts: 284
Liked: 11 times
Joined: Jan 06, 2011 8:33 am
Contact:

Re: Datastore Write Latency much too High from VeeamOne !!!

Post by WinstonWolf »

My First Stepp now is to completely uninstall and reinstall Veeam One . I also delete the Veeam One Database .
Then i observe if the Events comes again . Its not so much work ;-)
Shestakov
Veteran
Posts: 7328
Liked: 781 times
Joined: May 21, 2014 11:03 am
Full Name: Nikita Shestakov
Location: Prague
Contact:

Re: Datastore Write Latency much too High from VeeamOne !!!

Post by Shestakov »

Well... Sounds more time-consuming than a call to the support team :)
Vitaliy S.
VP, Product Management
Posts: 27055
Liked: 2710 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: Datastore Write Latency much too High from VeeamOne !!!

Post by Vitaliy S. »

And re-installing the data collection engine cannot fix this (if the issue exists) ;)
bsylvain
Novice
Posts: 5
Liked: 3 times
Joined: Sep 18, 2014 11:28 am
Contact:

Re: Datastore Write Latency much too High from VeeamOne !!!

Post by bsylvain »

I have the same "inconvenience" from Veeam One.

MAX seems to be the default parameter for this alarm (read/write latency).
I tried to switch to avg but it doesn't seems to keep the setting (if I go back in the alarm definition it's back to MAX), I have to create a custom alarm and disable the defaults ones. Is this normal?

Thanks.
Sylvain
Vitaliy S.
VP, Product Management
Posts: 27055
Liked: 2710 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: Datastore Write Latency much too High from VeeamOne !!!

Post by Vitaliy S. » 1 person likes this post

Hi Sylvain,

If the new setting is not preserved in the alarms configuration, then please open a support ticket and let our team take a look at this. Based on your description, it sound like a bug.

EDIT: Just checked it with our support team and they have a fix for this problem. Please ask for Update 1 RC to fix this issue.

Thanks!
bsylvain
Novice
Posts: 5
Liked: 3 times
Joined: Sep 18, 2014 11:28 am
Contact:

Re: Datastore Write Latency much too High from VeeamOne !!!

Post by bsylvain »

Hi Vitaliy,

Thanks, I will wait for official release then, no hurry.
Vitaliy S.
VP, Product Management
Posts: 27055
Liked: 2710 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: Datastore Write Latency much too High from VeeamOne !!!

Post by Vitaliy S. »

Ok, it should be a long wait. We are aiming to make it GA soon.
bsylvain
Novice
Posts: 5
Liked: 3 times
Joined: Sep 18, 2014 11:28 am
Contact:

Re: Datastore Write Latency much too High from VeeamOne !!!

Post by bsylvain » 3 people like this post

Hi

I forgot to follow up, but of course the update 1 fixed the problem. I have been able to set the alarms to AVG, no more annoyance.

Thanks

Sylvain
mattiamigliorati
Influencer
Posts: 12
Liked: 2 times
Joined: Jul 27, 2015 12:10 pm
Full Name: mattia
Contact:

Re: Datastore Write Latency much too High from VeeamOne !!!

Post by mattiamigliorati »

Hi

I know this is an old post, but after 1 year I'm digging into the same issue..
I have Veeam 9.5 Up 3 and vSphere 6.5 Up1. The storage is a Netapp and I have OCUM and OCPM working.
I'm monitoring datastore latency and another problematic parameter "VM total disk latency".
What I saw is that during the night are latencies going high, but this is acceptable as backup job are working.
Sometimes during working hours there are latency spikes and I'm troubleshooting this.
I know that Veeam ONE is collecting everything from vCenter.

What I'm searching for is:
Can I completely trust what I see in Veeam ONE or should I taking into account others thing to have irrefutable value ?

Best regards

Mattia
Shestakov
Veteran
Posts: 7328
Liked: 781 times
Joined: May 21, 2014 11:03 am
Full Name: Nikita Shestakov
Location: Prague
Contact:

Re: Datastore Write Latency much too High from VeeamOne !!!

Post by Shestakov »

Hi Mattia,
Yes you can.
You are right that Veeam ONE collects performance data from vSphere. That's the way to make sure it's correct.

Thanks!
mattiamigliorati
Influencer
Posts: 12
Liked: 2 times
Joined: Jul 27, 2015 12:10 pm
Full Name: mattia
Contact:

Re: Datastore Write Latency much too High from VeeamOne !!!

Post by mattiamigliorati »

Thank you Shestakov for the quick response.

One more question.
As I found into this guide https://docs.vmware.com/en/VMware-vSphe ... -guide.pdf , on page 98, this statement:
The value for the deviceLatency data counter is greater than 15ms indicates there are probably problems with the storage array

As you know 15ms is by far more less than what Veeam ONE and other monitoring programs set a limit for performance issue.

Isn't it true?

Regards

Mattia
Shestakov
Veteran
Posts: 7328
Liked: 781 times
Joined: May 21, 2014 11:03 am
Full Name: Nikita Shestakov
Location: Prague
Contact:

Re: Datastore Write Latency much too High from VeeamOne !!!

Post by Shestakov »

I don't remember the defaults we have, but you are probably correct. With development of better performing hardware our defaults may become outdated.
I'll research it, thanks for the heads up!
TheQ
Novice
Posts: 6
Liked: 4 times
Joined: Sep 05, 2018 6:28 am
Full Name: Frank Queisser
Contact:

Re: Datastore Write Latency much too High from VeeamOne !!!

Post by TheQ »

After migrating from vSphere 5.5 to 6.5 and changing ESX hardware from HP ProLiant Gen8 to Gen10 we receive daily alarm messages about VM total disk latency. As described in the early posts of this thread these spikes can not be retrieved or matched in vCenter performance monitoring. As we did not change our VEEAM One installation during migration we are wondering why everything was fine when running vSphere 5.5??? Of course we have updated all hard- and software components to the latest software and firmware releases and changed aggregation in alarm settings from MAX to AVG for VM total disk latency.

Is this issue still under investigation? Anyone seeing same or similar problems?
Shestakov
Veteran
Posts: 7328
Liked: 781 times
Joined: May 21, 2014 11:03 am
Full Name: Nikita Shestakov
Location: Prague
Contact:

Re: Datastore Write Latency much too High from VeeamOne !!!

Post by Shestakov »

Hi Frank and welcome to the community!
Do you have support case opened? If yes, please provide its number.
Thanks
TheQ
Novice
Posts: 6
Liked: 4 times
Joined: Sep 05, 2018 6:28 am
Full Name: Frank Queisser
Contact:

Re: Datastore Write Latency much too High from VeeamOne !!!

Post by TheQ »

Not yet. Will let you know asap
Shestakov
Veteran
Posts: 7328
Liked: 781 times
Joined: May 21, 2014 11:03 am
Full Name: Nikita Shestakov
Location: Prague
Contact:

Re: Datastore Write Latency much too High from VeeamOne !!!

Post by Shestakov »

By the way, what product version are you at?
TheQ
Novice
Posts: 6
Liked: 4 times
Joined: Sep 05, 2018 6:28 am
Full Name: Frank Queisser
Contact:

Re: Datastore Write Latency much too High from VeeamOne !!!

Post by TheQ »

currrent version 9.5.0.3801 includes Update from end of 2017
TheQ
Novice
Posts: 6
Liked: 4 times
Joined: Sep 05, 2018 6:28 am
Full Name: Frank Queisser
Contact:

Re: Datastore Write Latency much too High from VeeamOne !!!

Post by TheQ »

We now have a case number #03184720
Shestakov
Veteran
Posts: 7328
Liked: 781 times
Joined: May 21, 2014 11:03 am
Full Name: Nikita Shestakov
Location: Prague
Contact:

Re: Datastore Write Latency much too High from VeeamOne !!!

Post by Shestakov »

Thanks Frank, as far as I remember there was a hotfix for such a case, but it became a part of v9.5 update 1. So it's strange you have an issue in update 3.
Please keep working with the support team as they will make the investigation.
TheQ
Novice
Posts: 6
Liked: 4 times
Joined: Sep 05, 2018 6:28 am
Full Name: Frank Queisser
Contact:

Re: Datastore Write Latency much too High from VeeamOne !!!

Post by TheQ » 2 people like this post

Quite impressive how quick support and community is reacting on this issue. GOOD JOB everyone!

Today we received an confirmation that we have an abnormal behaviour. A Workaround has been offered by support team. Their idea is to change the alarm trigger config in database.

We already applied this workaround. Let's wait and see if this helps.
Shestakov
Veteran
Posts: 7328
Liked: 781 times
Joined: May 21, 2014 11:03 am
Full Name: Nikita Shestakov
Location: Prague
Contact:

Re: Datastore Write Latency much too High from VeeamOne !!!

Post by Shestakov »

Great to know, thanks!
I've shared your positive feedback with the support engineers.
TheQ
Novice
Posts: 6
Liked: 4 times
Joined: Sep 05, 2018 6:28 am
Full Name: Frank Queisser
Contact:

Re: Datastore Write Latency much too High from VeeamOne !!!

Post by TheQ » 2 people like this post

Morning folks,

It seems as if the workaround is fine for us. :D

We did not receive any alarms about disk latency since 4 days although we had some IO intensive workloads with small latency spikes. :!:
Shestakov
Veteran
Posts: 7328
Liked: 781 times
Joined: May 21, 2014 11:03 am
Full Name: Nikita Shestakov
Location: Prague
Contact:

Re: Datastore Write Latency much too High from VeeamOne !!!

Post by Shestakov »

Thanks for the update, Frank!
Post Reply

Who is online

Users browsing this forum: No registered users and 4 guests