Real-time performance monitoring and troubleshooting
mattiamigliorati
Influencer
Posts: 10
Liked: never
Joined: Jul 27, 2015 12:10 pm
Full Name: mattia
Contact:

Re: Datastore Write Latency much too High from VeeamOne !!!

Post by mattiamigliorati » May 08, 2018 3:17 pm

Hi

I know this is an old post, but after 1 year I'm digging into the same issue..
I have Veeam 9.5 Up 3 and vSphere 6.5 Up1. The storage is a Netapp and I have OCUM and OCPM working.
I'm monitoring datastore latency and another problematic parameter "VM total disk latency".
What I saw is that during the night are latencies going high, but this is acceptable as backup job are working.
Sometimes during working hours there are latency spikes and I'm troubleshooting this.
I know that Veeam ONE is collecting everything from vCenter.

What I'm searching for is:
Can I completely trust what I see in Veeam ONE or should I taking into account others thing to have irrefutable value ?

Best regards

Mattia

Shestakov
Veeam Software
Posts: 5968
Liked: 519 times
Joined: May 21, 2014 11:03 am
Full Name: Nikita Shestakov
Location: Prague
Contact:

Re: Datastore Write Latency much too High from VeeamOne !!!

Post by Shestakov » May 08, 2018 3:55 pm

Hi Mattia,
Yes you can.
You are right that Veeam ONE collects performance data from vSphere. That's the way to make sure it's correct.

Thanks!

mattiamigliorati
Influencer
Posts: 10
Liked: never
Joined: Jul 27, 2015 12:10 pm
Full Name: mattia
Contact:

Re: Datastore Write Latency much too High from VeeamOne !!!

Post by mattiamigliorati » May 09, 2018 1:02 pm

Thank you Shestakov for the quick response.

One more question.
As I found into this guide https://docs.vmware.com/en/VMware-vSphe ... -guide.pdf , on page 98, this statement:
The value for the deviceLatency data counter is greater than 15ms indicates there are probably problems with the storage array

As you know 15ms is by far more less than what Veeam ONE and other monitoring programs set a limit for performance issue.

Isn't it true?

Regards

Mattia

Shestakov
Veeam Software
Posts: 5968
Liked: 519 times
Joined: May 21, 2014 11:03 am
Full Name: Nikita Shestakov
Location: Prague
Contact:

Re: Datastore Write Latency much too High from VeeamOne !!!

Post by Shestakov » May 13, 2018 2:42 pm

I don't remember the defaults we have, but you are probably correct. With development of better performing hardware our defaults may become outdated.
I'll research it, thanks for the heads up!

TheQ
Novice
Posts: 6
Liked: 4 times
Joined: Sep 05, 2018 6:28 am
Full Name: Frank Queisser
Contact:

Re: Datastore Write Latency much too High from VeeamOne !!!

Post by TheQ » Sep 05, 2018 6:40 am

After migrating from vSphere 5.5 to 6.5 and changing ESX hardware from HP ProLiant Gen8 to Gen10 we receive daily alarm messages about VM total disk latency. As described in the early posts of this thread these spikes can not be retrieved or matched in vCenter performance monitoring. As we did not change our VEEAM One installation during migration we are wondering why everything was fine when running vSphere 5.5??? Of course we have updated all hard- and software components to the latest software and firmware releases and changed aggregation in alarm settings from MAX to AVG for VM total disk latency.

Is this issue still under investigation? Anyone seeing same or similar problems?

Shestakov
Veeam Software
Posts: 5968
Liked: 519 times
Joined: May 21, 2014 11:03 am
Full Name: Nikita Shestakov
Location: Prague
Contact:

Re: Datastore Write Latency much too High from VeeamOne !!!

Post by Shestakov » Sep 06, 2018 6:29 pm

Hi Frank and welcome to the community!
Do you have support case opened? If yes, please provide its number.
Thanks

TheQ
Novice
Posts: 6
Liked: 4 times
Joined: Sep 05, 2018 6:28 am
Full Name: Frank Queisser
Contact:

Re: Datastore Write Latency much too High from VeeamOne !!!

Post by TheQ » Sep 07, 2018 8:43 am

Not yet. Will let you know asap

Shestakov
Veeam Software
Posts: 5968
Liked: 519 times
Joined: May 21, 2014 11:03 am
Full Name: Nikita Shestakov
Location: Prague
Contact:

Re: Datastore Write Latency much too High from VeeamOne !!!

Post by Shestakov » Sep 07, 2018 11:50 am

By the way, what product version are you at?

TheQ
Novice
Posts: 6
Liked: 4 times
Joined: Sep 05, 2018 6:28 am
Full Name: Frank Queisser
Contact:

Re: Datastore Write Latency much too High from VeeamOne !!!

Post by TheQ » Sep 10, 2018 5:41 am

currrent version 9.5.0.3801 includes Update from end of 2017

TheQ
Novice
Posts: 6
Liked: 4 times
Joined: Sep 05, 2018 6:28 am
Full Name: Frank Queisser
Contact:

Re: Datastore Write Latency much too High from VeeamOne !!!

Post by TheQ » Sep 10, 2018 8:11 am

We now have a case number #03184720

Shestakov
Veeam Software
Posts: 5968
Liked: 519 times
Joined: May 21, 2014 11:03 am
Full Name: Nikita Shestakov
Location: Prague
Contact:

Re: Datastore Write Latency much too High from VeeamOne !!!

Post by Shestakov » Sep 10, 2018 10:34 am

Thanks Frank, as far as I remember there was a hotfix for such a case, but it became a part of v9.5 update 1. So it's strange you have an issue in update 3.
Please keep working with the support team as they will make the investigation.

TheQ
Novice
Posts: 6
Liked: 4 times
Joined: Sep 05, 2018 6:28 am
Full Name: Frank Queisser
Contact:

Re: Datastore Write Latency much too High from VeeamOne !!!

Post by TheQ » Sep 11, 2018 11:38 am 2 people like this post

Quite impressive how quick support and community is reacting on this issue. GOOD JOB everyone!

Today we received an confirmation that we have an abnormal behaviour. A Workaround has been offered by support team. Their idea is to change the alarm trigger config in database.

We already applied this workaround. Let's wait and see if this helps.

Shestakov
Veeam Software
Posts: 5968
Liked: 519 times
Joined: May 21, 2014 11:03 am
Full Name: Nikita Shestakov
Location: Prague
Contact:

Re: Datastore Write Latency much too High from VeeamOne !!!

Post by Shestakov » Sep 11, 2018 2:20 pm

Great to know, thanks!
I've shared your positive feedback with the support engineers.

TheQ
Novice
Posts: 6
Liked: 4 times
Joined: Sep 05, 2018 6:28 am
Full Name: Frank Queisser
Contact:

Re: Datastore Write Latency much too High from VeeamOne !!!

Post by TheQ » Sep 17, 2018 5:44 am 2 people like this post

Morning folks,

It seems as if the workaround is fine for us. :D

We did not receive any alarms about disk latency since 4 days although we had some IO intensive workloads with small latency spikes. :!:

Shestakov
Veeam Software
Posts: 5968
Liked: 519 times
Joined: May 21, 2014 11:03 am
Full Name: Nikita Shestakov
Location: Prague
Contact:

Re: Datastore Write Latency much too High from VeeamOne !!!

Post by Shestakov » Sep 18, 2018 6:25 pm

Thanks for the update, Frank!

Post Reply

Who is online

Users browsing this forum: No registered users and 6 guests