Interpretation 'Average CPU Wait'

[tom.depoorter]

Dear

I'm not sure how I need to interpret the metric 'average CPU wait'

The help says: "CPU time the VM spent waiting for hardware or VMkernel lock thread locks."
So I would say that a high percentage would mean no good, since the VM is waiting for hardware.

My VMs have a high average CPU wait,
but I don't know why.


CPU-, memory-, network- en datastore-usage looks OK.

Can any one help?

Best regards

Tom

[Vitaliy S.]

Hello Tom,

Be aware that high CPU Wait does not always result in bad VM performance. The total time of waiting CPU resources on the VMkernel level can be calculated using this formula %WAIT - % IDLE. So if %IDLE is equal to %WAIT, then everything is fine.

Thank you!

[tom.depoorter]

Hey Vitaliy S.

Your formula is minus WAIT minus IDLE ?

Where can I find the value of %IDLE ?

Best regards

Tom

[Vitaliy S.]

I have corrected my post above, it is just WAIT minus IDLE. Idle performance metric is not used very often in troubleshooting, so please check it directly in vSphere Client.

[tom.depoorter]

Ok Vitaliy
thx for you correction.

But when %IDLE is equal to %WAIT
the CPU WAIT percentage should be 0.

I've taken a look to the CPU-counters in the chart options in vSPhere Client,
but I didn't fid an IDLE-counter

[Vitaliy S.]

CPU IDLE can be found if you navigate to the VM in question -> Performance -> Advanced -> CPU realtime counters -> IDLE. Hope this helps!

[tom.depoorter]

I only have this counters: wait, usage, swap wait, usage in mhz, used, ready, system.
no idle.

[Vitaliy S.]

Hmm...what vSphere version are you at?

[tom.depoorter]

4.1.0

[Vitaliy S.]

Looks like this metric is only available via ESXTOP in vSphere 4.1.

[tom.depoorter]

ok Vitaliy

I've never used that,
but I'll give it a try.

But what about this remark?
"But when %IDLE is equal to %WAIT
the CPU WAIT percentage should be 0."

I'm obviously still interpreting something differently then you.

[Vitaliy S.]

%WAIT includes %IDLE by default, in other words %WAIT- %IDLE = X, where "X" is the time spent waiting for the resources. Hope this makes sense.

[tom.depoorter]

Dear Vitaliy

Sorry, it took me some time to discover (r)esxtop.
(I had to work on another project)

This is what I see:



There is no IDLE?
Is it possible that that metric doesn't exist in this version of Vsphere?

Best regards

Tom

[Vitaliy S.]

Hmm... based on this link, it should be: http://pubs.vmware.com/vsphere-51/index ... nters.html

Can you please try following this blog post to get this value?

[tom.depoorter]

Thanks for your post Vitaliy

It's strange: it seems to me that I'm not able to add those "Additional fields".
Could this be a difference between running esxtop for ESX and resxtop for ESXi?
(I keep searching)

If this value isn't available in ESXi
how veeam monitor gets this value?

Tom

[Vitaliy S.]

Veeam ONE collects all counters from the vCenter Server/ESXi hosts via VMware API, in other words we rely on the counter list that is exposed by the vCenter Server/ESXi host. Veeam ONE does not collect esxtop data.

I believe the counter set for esxtop command should be the same for all hosts, no matter if it is ESXi of ESX host.

Anyway, getting back to your original question, I don't think that you're experiencing performance issues (otherwise you would have noticed that already), so most likely IDLE counter is what makes WAIT metric so high.

[tom.depoorter]

hey Vitaliy

You're probably right, but I was just curious to discover those counters.

Any how I've find out how it can be displayed.
I post it, because it can help some one else who has real performance issues

I think it's a problem with the number of counters that can be displayed simultaneously.
By hitting f I've disabled all columns except %STATE TIMES
Then I see %IDLE.

%IDLE and %WAIT are close.
So no issue.

Thanks a lot for your time.
But the way, you've got a great product

[Vitaliy S.]

Thanks for the update and thank you for your kind words, our dev team will appreciate it.

[cffit]

I am trying to figure out if a VM that I have performance issues with could be caused by having 4 virtual CPUs allocated to it. I think the counter I need to look at is the CPU Co-stop counter to figure this out. Is that correct? That counter is very low. However, the Average CPU Wait time is 373% and the CPU usage averages around 50% all day, so it still seems fishy to me. Is the Idle time reported in version 5.1 in the VIC anywhere? Or do I still have to use esxtop? Are there other counters that can tell me if I have too many virtual CPUs to the point where it's causing an issue?

[Vitaliy S.]

Hi Christensen,

You need to review CPU Co-Stop and CPU Ready counters to review potential CPU bottlenecks. What are the average values for this VM?

Is the Idle time reported in version 5.1 in the VIC anywhere?

Yes, it should be under CPU -> Real Time counters.

Thanks!

[shocko]

Guys, just wondering what use can be made of the CPU wait displayed in Veeam One? It tells us nothing of the health of a VM as you have to go and find the %IDLE time manually in vSphere and subtract from CPU Wait to find the true amount of time the vm was blocked waiting for hardware. Why have Veeam implemented this in this manner? Am I missing something?

[Shestakov]

Hello Darragh,
What you are saying makes total sense. We are planning to improve cpu bottleneck analysis chart in concordance with your feedback.
Thank you for the participation!

[shocko]

Good to hear. I was doing a POC and found that the 'bottlenecks' graphs confused users as things appeared to be running fine but it looked like Veeam One was saying otherwise. Any time frame for this to be implemented?

[Shestakov]

Darragh,
That`s great to achieve valuable advice like yours. Thanks again!
The change is going to be included to the next minor updates.

[shocko]

Did this make it to veeam one 8 ??

[Vitaliy S.]

If you can't see it in all counters list by selecting a custom view option (can't check it myself now), then, unfortunately, it didn't make it to v8.

[Shestakov]

Hello Darragh,
Since our conversation was just 2 weeks before the actual release of Veeam ONE v8, we scheduled the change to our next (v9) release.
Thanks!