Undersized VM not showing any although they alert for CPU

[kevinnaughton]

I have several VMs that generate lots of alerts for CPU utilization. But when I run an undersized VM report (using day or week for the scope) I get zero recommendations. One in particular has had 100% utilization for a 20 minutes sustained period, but does not get reported as undersized. And when I view CPU, Past Week data, it shows max CPU Usage above 90%. Why aren't these being reported as undersized?

[Vitaliy S.]

Hi Kevin,

These VMs are not reported as undersized, because 90% of utilization could be "normal" for these VMs. Let me explain it further - if VM has 90% of CPU utilization and it stays on the same level and CPU Ready metric does not go above the threshold -it means that this VM does not need more CPU resources. Our engine not only looks at CPU average utilization, but also on other metrics such as CPU Ready to detect undersized VMs. Besides, if there was only 1 or 2 spikes, then it can also be fine. I would recommend to see the time stamp of these spikes and check with some maintenance tasks in the VMs, as it could be just an antivirus scan that causes this...

Thanks!

[kevinnaughton]

These are SQL servers, running SQL tasks and pegging the allocated CPUs at 100% for extended periods (20 minutes+). Adding more vCPUs results in tasks completing faster. My understanding is that CPU ready indicates that the VM cannot get it's vCPUs scheduled on physical CPU cores. Even though the physical resources exist, the allocation can still be a bottleneck. In fact, CPU ready would be more of an indicator to reduce cores to reduce pressure on the host. For these VMs, the bottleneck is the amount of processing power allocated to them and increasing that allocation relieves the bottleneck. To me, that's the definition of an undersized VM.

[Vitaliy S.]

You're correct, SQL Server will always consume the memory you allocate to this VM. If you want to see how SQL Server is using its memory, then please review memory active counter.

Regarding the CPU Ready counter, I agree with you that you can reduce the number of allocated CPU, but it would not mean that your VM will start to perform better. If you see high CPU values, then our usual recommendation would be to migrate the VM to the host with more CPU power.

[kevinnaughton]

I'm not sure that explains what I'm seeing. I'm not talking about memory. I'm talking about CPU usage. My VMs show extended periods of 100% CPU utilization. But when I run an undersized VM report, they are not listed as being undersized. Why wouldn't these be listed in the report as being undersized? What use is the undersized VM report if it doens't report that VMs using 100% of their allocated resources are undersized?

[Vitaliy S.]

Why wouldn't these be listed in the report as being undersized? What use is the undersized VM report if it doens't report that VMs using 100% of their allocated resources are undersized?

That is exactly what I was trying to explain, if VM is using 99% of its CPU resources it does not mean it needs more. What are the CPU ready values for this VM? Can you please double check?

[kevinnaughton]

It's 100% usage, for 20 minutes, all the while the peak CPU ready is 0.06.

[Vitaliy S.]

Thanks for sharing. In this example, if VM has just one spike of 100% for CPU Usage and CPU Ready is low, then this VM cannot be considered undersized. VM will considered undersized if it constantly lacks of CPU resources (its usage is more than 90%) and it has high CPU Ready values. Hope this makes sense.

[kevinnaughton]

Sorry, it's still not making sense to me. I don't understand why CPU ready is a factor for determining "undersized VMs". CPU ready would indicate too many CPUs allocated on the hypervisor therefore restricting the ability to schedule processor time for the VM. That's an indicator of too many vCPUs allocated to the VM making scheduling more difficult and/or an undersized hypervisor. High CPU utilization on the VM indicates the VM does not have enough CPU allocation for processes on the guest to run without maxing out the allocation. Here is the resolution text in the alert in your software:

"Check for high demand for CPU resources from application workloads inside the VM. Consider adding virtual CPUs to better distribute the processing across multiple cores. Note that multi-CPU VMs can impact the efficiency of the host CPU scheduler, as host must allocate all cores simultaneously in order to allow genuine multi-CPU activities within the VM."

[Shestakov]

Hello Kevin, let me jump into the conversation.
You are right, we can not say for sure that VM is undersized if it just has high CPU Ready value, that`s why we use both CPU Ready and CPU Usage metrics for the report.
And you can use either both of them or just one.

CPU ready would indicate too many CPUs allocated on the hypervisor therefore restricting the ability to schedule processor time for the VM. That's an indicator of too many vCPUs allocated to the VM making scheduling more difficult and/or an undersized hypervisor.

Not only CPU Over-subscription, but also Wrong Sizing, CPU Affinity, Fault Tolerance and several other things can lead to high CPU Ready. Again, that`s why we use both CPU Ready and CPU Usage metrics.
But, I think you are right and we need to change an advice part a bit.

Could you provide your point of view regarding CPU sizing in the report, should we use only CPU Usage or combine those metrics in any different manner?
Thank you for the valuable contribution!

[Vitaliy S.]

If VM has a single peak of CPU usage, it doesn't mean it needs more resources. For example, if Veeam proxy servers are utilizing 99% of the CPU during backup job run, then it is perfectly normal, they should do that. If the average CPU utilization over the extended period of time shows > 90% of CPU utilization, then this VM is good candidate for giving more CPU resources. Another example, every VM when it is rebooted consumes almost all CPU/Memory resources, but it doesn't mean it is undersized. The same refers for antivirus activity or some maintenance tasks scheduled inside the VM. If it happens on regular basis (let's assume it happens every last Friday of the month), but the average utilization of this VM is 5-10%, then this also does not indicate it is undersized. To address these cases I usually recommend, trying to find the pattern for these spikes and adjust VM configurations accordingly.

Thanks for your feedback, very interesting discussion!

[kevinnaughton]

Shestakov, I'd like to see that a VM with frequent and sustained CPU utilization of 100% is listed as an undersized VM.

Vitaliy, that's my point, these VMs have 100% utilization, for up to 20 minutes, several times per day. Yes, most of the time they are idle and not doing that and if you're using an average CPU I could see why it wouldn't be considered undersized. But these are generating alerts for high CPU utilization (over 90%) a dozen times per day with a couple of those lasting longer than 20 minutes each. These VMs, like many others, aren't used all the time. They have specific jobs that are kicked off sporadically throughout the day. But when these jobs are run, they are high priority and time sensitive. By increasing the number of vCPUs, I have been able to reduce the time these jobs take to run thereby increasing productivity. I would have assumed that with so many alerts about CPU, they should show up under undersized VM report. But it sounds like you guys are smoothing the data points out so that relatively short lived spikes don't get counted in that determination.

My suggestion would be to either add another report, or modify the existing and provide more info. Many of our most critical VMs are running infrequent but CPU intensive tasks and are time critical. It would be handy to see not only those VMs that are undersized in a more general sense, but also those that regularly experience exhaustion of a resource.

Thanks for your attention and responses. For now I will just have to rely on scouring logs and alerts for these. Would love to see it in a report, though.

Cheers.

[Shestakov]

Kevin, I find your reasoning fair.

What I see as a possible improvement for the report is adding Business hours field to make it possible for users to take into account utilization only within specific hours not counting, say night time when VMs are idle.

Another suggestion is to show time when CPU Usage was more than 95 or 97.5%, meaning VM experienced a lack of CPU resources.
I don`t see a huge value to make an additional report for Undersized VMs investigation.

Kevin, what do you think about above-mentioned intentions?
Thanks.

[kevinnaughton]

The first suggestions wouldn't help, as they occur throughout the 24 hour day. But yes, to the second point, a quick and easy way to get a list of VMs that pegged a finite resource (e.g. CPU, RAM) at 100% at any point for any duration would be handy.

Thanks again.

[Shestakov]

Thanks for the fast reply and your opinion.

Speaking in support of the first suggestion I would say it can be useful at if resource consumption depends on working ours a lot. In this case you can choose 9am-5pm time interval and CPU Usage > 60-70% what can help to find Undersized VMs. Since Veeam ONE collects average indices of 5 min time intervals, so for some time intervals CPU Usage can be around 95% what is less than 100 and wouldn`t be shown as a peak in spite of having short-term peaks.

Anyways, we will think how to improve the report based on your and other feedbacks. Thanks!