Real-time performance monitoring and troubleshooting
Post Reply
tom.depoorter
Influencer
Posts: 21
Liked: 3 times
Joined: Oct 02, 2013 2:06 pm
Full Name: Tom Depoorter
Contact:

Interpretation 'Average CPU Wait'

Post by tom.depoorter » 2 people like this post

Dear

I'm not sure how I need to interpret the metric 'average CPU wait'

The help says: "CPU time the VM spent waiting for hardware or VMkernel lock thread locks."
So I would say that a high percentage would mean no good, since the VM is waiting for hardware.

My VMs have a high average CPU wait,
but I don't know why.
Image

CPU-, memory-, network- en datastore-usage looks OK.

Can any one help?

Best regards

Tom
Vitaliy S.
VP, Product Management
Posts: 27377
Liked: 2800 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: Interpretation 'Average CPU Wait'

Post by Vitaliy S. » 1 person likes this post

Hello Tom,

Be aware that high CPU Wait does not always result in bad VM performance. The total time of waiting CPU resources on the VMkernel level can be calculated using this formula %WAIT - % IDLE. So if %IDLE is equal to %WAIT, then everything is fine.

Thank you!
tom.depoorter
Influencer
Posts: 21
Liked: 3 times
Joined: Oct 02, 2013 2:06 pm
Full Name: Tom Depoorter
Contact:

Re: Interpretation 'Average CPU Wait'

Post by tom.depoorter »

Hey Vitaliy S.

Your formula is minus WAIT minus IDLE ?

Where can I find the value of %IDLE ?

Best regards

Tom
Vitaliy S.
VP, Product Management
Posts: 27377
Liked: 2800 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: Interpretation 'Average CPU Wait'

Post by Vitaliy S. » 1 person likes this post

I have corrected my post above, it is just WAIT minus IDLE. Idle performance metric is not used very often in troubleshooting, so please check it directly in vSphere Client.
tom.depoorter
Influencer
Posts: 21
Liked: 3 times
Joined: Oct 02, 2013 2:06 pm
Full Name: Tom Depoorter
Contact:

Re: Interpretation 'Average CPU Wait'

Post by tom.depoorter »

Ok Vitaliy
thx for you correction.

But when %IDLE is equal to %WAIT
the CPU WAIT percentage should be 0.

I've taken a look to the CPU-counters in the chart options in vSPhere Client,
but I didn't fid an IDLE-counter
Vitaliy S.
VP, Product Management
Posts: 27377
Liked: 2800 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: Interpretation 'Average CPU Wait'

Post by Vitaliy S. »

CPU IDLE can be found if you navigate to the VM in question -> Performance -> Advanced -> CPU realtime counters -> IDLE. Hope this helps!
tom.depoorter
Influencer
Posts: 21
Liked: 3 times
Joined: Oct 02, 2013 2:06 pm
Full Name: Tom Depoorter
Contact:

Re: Interpretation 'Average CPU Wait'

Post by tom.depoorter »

I only have this counters: wait, usage, swap wait, usage in mhz, used, ready, system.
no idle.
Vitaliy S.
VP, Product Management
Posts: 27377
Liked: 2800 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: Interpretation 'Average CPU Wait'

Post by Vitaliy S. »

Hmm...what vSphere version are you at?
tom.depoorter
Influencer
Posts: 21
Liked: 3 times
Joined: Oct 02, 2013 2:06 pm
Full Name: Tom Depoorter
Contact:

Re: Interpretation 'Average CPU Wait'

Post by tom.depoorter »

4.1.0
Vitaliy S.
VP, Product Management
Posts: 27377
Liked: 2800 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: Interpretation 'Average CPU Wait'

Post by Vitaliy S. »

Looks like this metric is only available via ESXTOP in vSphere 4.1.
tom.depoorter
Influencer
Posts: 21
Liked: 3 times
Joined: Oct 02, 2013 2:06 pm
Full Name: Tom Depoorter
Contact:

Re: Interpretation 'Average CPU Wait'

Post by tom.depoorter »

ok Vitaliy

I've never used that,
but I'll give it a try.

But what about this remark?
"But when %IDLE is equal to %WAIT
the CPU WAIT percentage should be 0."

I'm obviously still interpreting something differently then you.
Vitaliy S.
VP, Product Management
Posts: 27377
Liked: 2800 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: Interpretation 'Average CPU Wait'

Post by Vitaliy S. »

%WAIT includes %IDLE by default, in other words %WAIT- %IDLE = X, where "X" is the time spent waiting for the resources. Hope this makes sense.
tom.depoorter
Influencer
Posts: 21
Liked: 3 times
Joined: Oct 02, 2013 2:06 pm
Full Name: Tom Depoorter
Contact:

Re: Interpretation 'Average CPU Wait'

Post by tom.depoorter »

Dear Vitaliy

Sorry, it took me some time to discover (r)esxtop.
(I had to work on another project)

This is what I see:

Image

There is no IDLE?
Is it possible that that metric doesn't exist in this version of Vsphere?

Best regards

Tom
Vitaliy S.
VP, Product Management
Posts: 27377
Liked: 2800 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: Interpretation 'Average CPU Wait'

Post by Vitaliy S. »

Hmm... based on this link, it should be: http://pubs.vmware.com/vsphere-51/index ... nters.html

Can you please try following this blog post to get this value?
tom.depoorter
Influencer
Posts: 21
Liked: 3 times
Joined: Oct 02, 2013 2:06 pm
Full Name: Tom Depoorter
Contact:

Re: Interpretation 'Average CPU Wait'

Post by tom.depoorter »

Thanks for your post Vitaliy

It's strange: it seems to me that I'm not able to add those "Additional fields".
Could this be a difference between running esxtop for ESX and resxtop for ESXi?
(I keep searching)

If this value isn't available in ESXi
how veeam monitor gets this value?

Tom
Vitaliy S.
VP, Product Management
Posts: 27377
Liked: 2800 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: Interpretation 'Average CPU Wait'

Post by Vitaliy S. »

Veeam ONE collects all counters from the vCenter Server/ESXi hosts via VMware API, in other words we rely on the counter list that is exposed by the vCenter Server/ESXi host. Veeam ONE does not collect esxtop data.

I believe the counter set for esxtop command should be the same for all hosts, no matter if it is ESXi of ESX host.

Anyway, getting back to your original question, I don't think that you're experiencing performance issues (otherwise you would have noticed that already), so most likely IDLE counter is what makes WAIT metric so high.
tom.depoorter
Influencer
Posts: 21
Liked: 3 times
Joined: Oct 02, 2013 2:06 pm
Full Name: Tom Depoorter
Contact:

Re: Interpretation 'Average CPU Wait'

Post by tom.depoorter » 1 person likes this post

hey Vitaliy

You're probably right, but I was just curious to discover those counters.

Any how I've find out how it can be displayed.
I post it, because it can help some one else who has real performance issues :D

I think it's a problem with the number of counters that can be displayed simultaneously.
By hitting f I've disabled all columns except %STATE TIMES
Then I see %IDLE.

%IDLE and %WAIT are close.
So no issue.

Thanks a lot for your time.
But the way, you've got a great product 8)
Vitaliy S.
VP, Product Management
Posts: 27377
Liked: 2800 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: Interpretation 'Average CPU Wait'

Post by Vitaliy S. »

Thanks for the update and thank you for your kind words, our dev team will appreciate it.
cffit
Veteran
Posts: 338
Liked: 35 times
Joined: Jan 20, 2012 2:36 pm
Full Name: Christensen Farms
Contact:

Re: Interpretation 'Average CPU Wait'

Post by cffit »

I am trying to figure out if a VM that I have performance issues with could be caused by having 4 virtual CPUs allocated to it. I think the counter I need to look at is the CPU Co-stop counter to figure this out. Is that correct? That counter is very low. However, the Average CPU Wait time is 373% and the CPU usage averages around 50% all day, so it still seems fishy to me. Is the Idle time reported in version 5.1 in the VIC anywhere? Or do I still have to use esxtop? Are there other counters that can tell me if I have too many virtual CPUs to the point where it's causing an issue?
Vitaliy S.
VP, Product Management
Posts: 27377
Liked: 2800 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: Interpretation 'Average CPU Wait'

Post by Vitaliy S. »

Hi Christensen,

You need to review CPU Co-Stop and CPU Ready counters to review potential CPU bottlenecks. What are the average values for this VM?
cffit wrote:Is the Idle time reported in version 5.1 in the VIC anywhere?
Yes, it should be under CPU -> Real Time counters.

Thanks!
shocko
Novice
Posts: 7
Liked: never
Joined: Jan 04, 2014 10:16 am
Full Name: shocko
Contact:

[MERGED] CPU Wait and Bottleneck Graph

Post by shocko »

Guys, just wondering what use can be made of the CPU wait displayed in Veeam One? It tells us nothing of the health of a VM as you have to go and find the %IDLE time manually in vSphere and subtract from CPU Wait to find the true amount of time the vm was blocked waiting for hardware. Why have Veeam implemented this in this manner? Am I missing something?
Shestakov
Veteran
Posts: 7328
Liked: 781 times
Joined: May 21, 2014 11:03 am
Full Name: Nikita Shestakov
Location: Prague
Contact:

Re: Interpretation 'Average CPU Wait'

Post by Shestakov »

Hello Darragh,
What you are saying makes total sense. We are planning to improve cpu bottleneck analysis chart in concordance with your feedback.
Thank you for the participation!
shocko
Novice
Posts: 7
Liked: never
Joined: Jan 04, 2014 10:16 am
Full Name: shocko
Contact:

Re: Interpretation 'Average CPU Wait'

Post by shocko »

Good to hear. I was doing a POC and found that the 'bottlenecks' graphs confused users as things appeared to be running fine but it looked like Veeam One was saying otherwise. Any time frame for this to be implemented?
Shestakov
Veteran
Posts: 7328
Liked: 781 times
Joined: May 21, 2014 11:03 am
Full Name: Nikita Shestakov
Location: Prague
Contact:

Re: Interpretation 'Average CPU Wait'

Post by Shestakov » 1 person likes this post

Darragh,
That`s great to achieve valuable advice like yours. Thanks again!
The change is going to be included to the next minor updates.
shocko
Novice
Posts: 7
Liked: never
Joined: Jan 04, 2014 10:16 am
Full Name: shocko
Contact:

Re: Interpretation 'Average CPU Wait'

Post by shocko »

Did this make it to veeam one 8 ??
Vitaliy S.
VP, Product Management
Posts: 27377
Liked: 2800 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: Interpretation 'Average CPU Wait'

Post by Vitaliy S. »

If you can't see it in all counters list by selecting a custom view option (can't check it myself now), then, unfortunately, it didn't make it to v8.
Shestakov
Veteran
Posts: 7328
Liked: 781 times
Joined: May 21, 2014 11:03 am
Full Name: Nikita Shestakov
Location: Prague
Contact:

Re: Interpretation 'Average CPU Wait'

Post by Shestakov »

Hello Darragh,
Since our conversation was just 2 weeks before the actual release of Veeam ONE v8, we scheduled the change to our next (v9) release.
Thanks!
Post Reply

Who is online

Users browsing this forum: No registered users and 1 guest