Real-time performance monitoring and troubleshooting
WinstonWolf
Expert
Posts: 187
Liked: 4 times
Joined: Jan 06, 2011 8:33 am
Contact:

Datastore Write Latency much too High from VeeamOne !!!

Post by WinstonWolf » Dec 06, 2016 8:27 am

Hello ,
I have every Morning 60 Mails with Datastore Write Latency Warning or Error Messages from Veeam One Site .
Like this :
Image

But on the other Side we use the Netapp VA OCPM ( OnCommand Performance Manager ) & the OCUM ( On Command Unified Manager )
This are special VAs from Netapp which Monitor the Datastores and Volumes also. And sorry Veeam, i trust this Netapp own tools more then Veeam One ;-)

And the Output from this Tools is for the same Datastore RZ1_vmware_ds10 this :
Image

As you can see the Disk write latency is only on the highest peaks and only for an short period 20.0 ms .

What Veeam One see there ??!?!?

Any Statement for this ?

Shestakov
Veeam Software
Posts: 6742
Liked: 681 times
Joined: May 21, 2014 11:03 am
Full Name: Nikita Shestakov
Location: Prague
Contact:

Re: Datastore Write Latency much too High from VeeamOne !!!

Post by Shestakov » Dec 06, 2016 8:54 am

Hello,
Could you check what type of aggregation do you have set up in the triggered alarm(Avg/Max)?
I believe that`s why there are different values.

Usually monitoring tools have to aggregate data in 5 or 20 minutes averages and products may have different time frames.
Have you checked this metric in Veeam ONE Monitor?

Thanks!

Mike Resseler
Product Manager
Posts: 5662
Liked: 593 times
Joined: Feb 08, 2013 3:08 pm
Full Name: Mike Resseler
Location: Belgium
Contact:

Re: Datastore Write Latency much too High from VeeamOne !!!

Post by Mike Resseler » Dec 06, 2016 9:37 am

Hi Winston,

Could you compare the data from Veeam ONE also with the data you see in vCenter? Because that is where we grab our information. I am not sure (but don't know either so ;-)) that we are comparing apples with apples here. What metric are you looking at with OCMP? (I don't know what MS / OP means and it might be a different metric.)

Thanks
Mike

Vitaliy S.
Product Manager
Posts: 22697
Liked: 1498 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: Datastore Write Latency much too High from VeeamOne !!!

Post by Vitaliy S. » Dec 07, 2016 4:44 pm

Yes, Mike is correct. Veeam ONE cannot make up data, we collect it via vSphere API, so if there is a spike reported in Veeam ONE, it will also be reflected in the vSphere Client.

WinstonWolf
Expert
Posts: 187
Liked: 4 times
Joined: Jan 06, 2011 8:33 am
Contact:

Re: Datastore Write Latency much too High from VeeamOne !!!

Post by WinstonWolf » Dec 08, 2016 8:39 am

Hello again ,

Here is the Screenshot from my Alarm Rules from "Datastore Write Latency"
Image

Can it be better with Aggregation : "Avg" ?

Here is an Screenshot from Write Latency from the Vcenter Server , as you can see there are some Spikes but not for long time ( they can produced by Storage Snapshots maybe )

Image

That means for me to Switch the default Setting from Aggregation max to Aggregation Avg . or ?

Thanks
Michael

Thanks
Michael

WinstonWolf
Expert
Posts: 187
Liked: 4 times
Joined: Jan 06, 2011 8:33 am
Contact:

Re: Datastore Write Latency much too High from VeeamOne !!!

Post by WinstonWolf » Dec 08, 2016 9:13 am

Hello ,

Now i think Veeam One runs totaly wrong .
At the Moment i become this Warning ( Error ) Mail from Veeam One which told me as follows :

Image

But when i look on my Vcenter Server the the Performance Graph from this Datastore shows this :

Image

Please Veeam , can u explain it for me .

Thanks

Shestakov
Veeam Software
Posts: 6742
Liked: 681 times
Joined: May 21, 2014 11:03 am
Full Name: Nikita Shestakov
Location: Prague
Contact:

Re: Datastore Write Latency much too High from VeeamOne !!!

Post by Shestakov » Dec 08, 2016 11:07 am

Hello Michael,

The alarm has been triggered because you`ve set aggregation=max. So once you had a spike higher than the threshold, alarm worked.
Regarding the last screenshot, could you also check values in Veeam ONE Monitor for the same time frame? If you think that Veeam ONE doesn`t work correctly, please contact Veeam support team for the detailed investigation. The will check if aggregation, object and metric are chosen correctly. Once you open a case, please let us know your case ID.

Thanks!

WinstonWolf
Expert
Posts: 187
Liked: 4 times
Joined: Jan 06, 2011 8:33 am
Contact:

Re: Datastore Write Latency much too High from VeeamOne !!!

Post by WinstonWolf » Dec 08, 2016 12:09 pm

My First Stepp now is to completely uninstall and reinstall Veeam One . I also delete the Veeam One Database .
Then i observe if the Events comes again . Its not so much work ;-)

Shestakov
Veeam Software
Posts: 6742
Liked: 681 times
Joined: May 21, 2014 11:03 am
Full Name: Nikita Shestakov
Location: Prague
Contact:

Re: Datastore Write Latency much too High from VeeamOne !!!

Post by Shestakov » Dec 08, 2016 12:35 pm

Well... Sounds more time-consuming than a call to the support team :)

Vitaliy S.
Product Manager
Posts: 22697
Liked: 1498 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: Datastore Write Latency much too High from VeeamOne !!!

Post by Vitaliy S. » Dec 08, 2016 12:37 pm

And re-installing the data collection engine cannot fix this (if the issue exists) ;)

bsylvain
Novice
Posts: 5
Liked: 3 times
Joined: Sep 18, 2014 11:28 am
Contact:

Re: Datastore Write Latency much too High from VeeamOne !!!

Post by bsylvain » Jan 06, 2017 3:03 pm

I have the same "inconvenience" from Veeam One.

MAX seems to be the default parameter for this alarm (read/write latency).
I tried to switch to avg but it doesn't seems to keep the setting (if I go back in the alarm definition it's back to MAX), I have to create a custom alarm and disable the defaults ones. Is this normal?

Thanks.
Sylvain

Vitaliy S.
Product Manager
Posts: 22697
Liked: 1498 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: Datastore Write Latency much too High from VeeamOne !!!

Post by Vitaliy S. » Jan 06, 2017 3:05 pm 1 person likes this post

Hi Sylvain,

If the new setting is not preserved in the alarms configuration, then please open a support ticket and let our team take a look at this. Based on your description, it sound like a bug.

EDIT: Just checked it with our support team and they have a fix for this problem. Please ask for Update 1 RC to fix this issue.

Thanks!

bsylvain
Novice
Posts: 5
Liked: 3 times
Joined: Sep 18, 2014 11:28 am
Contact:

Re: Datastore Write Latency much too High from VeeamOne !!!

Post by bsylvain » Jan 06, 2017 3:14 pm

Hi Vitaliy,

Thanks, I will wait for official release then, no hurry.

Vitaliy S.
Product Manager
Posts: 22697
Liked: 1498 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: Datastore Write Latency much too High from VeeamOne !!!

Post by Vitaliy S. » Jan 06, 2017 3:15 pm

Ok, it should be a long wait. We are aiming to make it GA soon.

bsylvain
Novice
Posts: 5
Liked: 3 times
Joined: Sep 18, 2014 11:28 am
Contact:

Re: Datastore Write Latency much too High from VeeamOne !!!

Post by bsylvain » Feb 17, 2017 2:20 pm 3 people like this post

Hi

I forgot to follow up, but of course the update 1 fixed the problem. I have been able to set the alarms to AVG, no more annoyance.

Thanks

Sylvain

mattiamigliorati
Influencer
Posts: 11
Liked: never
Joined: Jul 27, 2015 12:10 pm
Full Name: mattia
Contact:

Re: Datastore Write Latency much too High from VeeamOne !!!

Post by mattiamigliorati » May 08, 2018 3:17 pm

Hi

I know this is an old post, but after 1 year I'm digging into the same issue..
I have Veeam 9.5 Up 3 and vSphere 6.5 Up1. The storage is a Netapp and I have OCUM and OCPM working.
I'm monitoring datastore latency and another problematic parameter "VM total disk latency".
What I saw is that during the night are latencies going high, but this is acceptable as backup job are working.
Sometimes during working hours there are latency spikes and I'm troubleshooting this.
I know that Veeam ONE is collecting everything from vCenter.

What I'm searching for is:
Can I completely trust what I see in Veeam ONE or should I taking into account others thing to have irrefutable value ?

Best regards

Mattia

Shestakov
Veeam Software
Posts: 6742
Liked: 681 times
Joined: May 21, 2014 11:03 am
Full Name: Nikita Shestakov
Location: Prague
Contact:

Re: Datastore Write Latency much too High from VeeamOne !!!

Post by Shestakov » May 08, 2018 3:55 pm

Hi Mattia,
Yes you can.
You are right that Veeam ONE collects performance data from vSphere. That's the way to make sure it's correct.

Thanks!

mattiamigliorati
Influencer
Posts: 11
Liked: never
Joined: Jul 27, 2015 12:10 pm
Full Name: mattia
Contact:

Re: Datastore Write Latency much too High from VeeamOne !!!

Post by mattiamigliorati » May 09, 2018 1:02 pm

Thank you Shestakov for the quick response.

One more question.
As I found into this guide https://docs.vmware.com/en/VMware-vSphe ... -guide.pdf , on page 98, this statement:
The value for the deviceLatency data counter is greater than 15ms indicates there are probably problems with the storage array

As you know 15ms is by far more less than what Veeam ONE and other monitoring programs set a limit for performance issue.

Isn't it true?

Regards

Mattia

Shestakov
Veeam Software
Posts: 6742
Liked: 681 times
Joined: May 21, 2014 11:03 am
Full Name: Nikita Shestakov
Location: Prague
Contact:

Re: Datastore Write Latency much too High from VeeamOne !!!

Post by Shestakov » May 13, 2018 2:42 pm

I don't remember the defaults we have, but you are probably correct. With development of better performing hardware our defaults may become outdated.
I'll research it, thanks for the heads up!

TheQ
Novice
Posts: 6
Liked: 4 times
Joined: Sep 05, 2018 6:28 am
Full Name: Frank Queisser
Contact:

Re: Datastore Write Latency much too High from VeeamOne !!!

Post by TheQ » Sep 05, 2018 6:40 am

After migrating from vSphere 5.5 to 6.5 and changing ESX hardware from HP ProLiant Gen8 to Gen10 we receive daily alarm messages about VM total disk latency. As described in the early posts of this thread these spikes can not be retrieved or matched in vCenter performance monitoring. As we did not change our VEEAM One installation during migration we are wondering why everything was fine when running vSphere 5.5??? Of course we have updated all hard- and software components to the latest software and firmware releases and changed aggregation in alarm settings from MAX to AVG for VM total disk latency.

Is this issue still under investigation? Anyone seeing same or similar problems?

Shestakov
Veeam Software
Posts: 6742
Liked: 681 times
Joined: May 21, 2014 11:03 am
Full Name: Nikita Shestakov
Location: Prague
Contact:

Re: Datastore Write Latency much too High from VeeamOne !!!

Post by Shestakov » Sep 06, 2018 6:29 pm

Hi Frank and welcome to the community!
Do you have support case opened? If yes, please provide its number.
Thanks

TheQ
Novice
Posts: 6
Liked: 4 times
Joined: Sep 05, 2018 6:28 am
Full Name: Frank Queisser
Contact:

Re: Datastore Write Latency much too High from VeeamOne !!!

Post by TheQ » Sep 07, 2018 8:43 am

Not yet. Will let you know asap

Shestakov
Veeam Software
Posts: 6742
Liked: 681 times
Joined: May 21, 2014 11:03 am
Full Name: Nikita Shestakov
Location: Prague
Contact:

Re: Datastore Write Latency much too High from VeeamOne !!!

Post by Shestakov » Sep 07, 2018 11:50 am

By the way, what product version are you at?

TheQ
Novice
Posts: 6
Liked: 4 times
Joined: Sep 05, 2018 6:28 am
Full Name: Frank Queisser
Contact:

Re: Datastore Write Latency much too High from VeeamOne !!!

Post by TheQ » Sep 10, 2018 5:41 am

currrent version 9.5.0.3801 includes Update from end of 2017

TheQ
Novice
Posts: 6
Liked: 4 times
Joined: Sep 05, 2018 6:28 am
Full Name: Frank Queisser
Contact:

Re: Datastore Write Latency much too High from VeeamOne !!!

Post by TheQ » Sep 10, 2018 8:11 am

We now have a case number #03184720

Shestakov
Veeam Software
Posts: 6742
Liked: 681 times
Joined: May 21, 2014 11:03 am
Full Name: Nikita Shestakov
Location: Prague
Contact:

Re: Datastore Write Latency much too High from VeeamOne !!!

Post by Shestakov » Sep 10, 2018 10:34 am

Thanks Frank, as far as I remember there was a hotfix for such a case, but it became a part of v9.5 update 1. So it's strange you have an issue in update 3.
Please keep working with the support team as they will make the investigation.

TheQ
Novice
Posts: 6
Liked: 4 times
Joined: Sep 05, 2018 6:28 am
Full Name: Frank Queisser
Contact:

Re: Datastore Write Latency much too High from VeeamOne !!!

Post by TheQ » Sep 11, 2018 11:38 am 2 people like this post

Quite impressive how quick support and community is reacting on this issue. GOOD JOB everyone!

Today we received an confirmation that we have an abnormal behaviour. A Workaround has been offered by support team. Their idea is to change the alarm trigger config in database.

We already applied this workaround. Let's wait and see if this helps.

Shestakov
Veeam Software
Posts: 6742
Liked: 681 times
Joined: May 21, 2014 11:03 am
Full Name: Nikita Shestakov
Location: Prague
Contact:

Re: Datastore Write Latency much too High from VeeamOne !!!

Post by Shestakov » Sep 11, 2018 2:20 pm

Great to know, thanks!
I've shared your positive feedback with the support engineers.

TheQ
Novice
Posts: 6
Liked: 4 times
Joined: Sep 05, 2018 6:28 am
Full Name: Frank Queisser
Contact:

Re: Datastore Write Latency much too High from VeeamOne !!!

Post by TheQ » Sep 17, 2018 5:44 am 2 people like this post

Morning folks,

It seems as if the workaround is fine for us. :D

We did not receive any alarms about disk latency since 4 days although we had some IO intensive workloads with small latency spikes. :!:

Shestakov
Veeam Software
Posts: 6742
Liked: 681 times
Joined: May 21, 2014 11:03 am
Full Name: Nikita Shestakov
Location: Prague
Contact:

Re: Datastore Write Latency much too High from VeeamOne !!!

Post by Shestakov » Sep 18, 2018 6:25 pm

Thanks for the update, Frank!

Post Reply

Who is online

Users browsing this forum: No registered users and 1 guest