Oversized VM's Report Too Agressive

[deduplicat3d]

When I launch the Oversized Virtual Machines by Memory report is shows the VM's Memory usage as very low and recommend I dramatically decrease memory. The numbers that are presented look more like active memory rather than consumed. Am I doing something wrong?

[deduplicat3d]

PS in the Monitor tool when I click on a VM it shows active memory but in reporter when i run a report on the VM it shows consumed.

[Vitaliy S.]

Oversized Virtual Machines report is using "Memory Usage" counter which is calculated by dividing VM Memory Active by VM Memory Granted, we are not using memory consumed as this is a "false" metric to make sizing recommendations. For example, SQL Server VM will always report on 100% consumed memory, while only 20-30% might be actually actively used.

Be aware that while detecting oversized VMs, our engine searches for VMs with low average resource usage, detects the peak, and then adds 25% as a buffer to this peak usage, and finally makes the recommendations. Furthermore, if you want to verify the recommended configuration, you can drill-down to VM resource usage details and see what data our recommendation is based on.

Hope this helps!

[cyberswb]

I've run this report a couple of times and I'm always surprised at how aggressive the recommendations are for memory usage.

One client has a production Exchange 2007 VM servicing about 60 mailboxes. The existing VM is assigned 8 GB RAM but when I run the oversized VM report, it suggests cutting this down to 1.9 GB based on peak memory usage.

I guess part of me thinks would be way too low, despite what the active memory statistics would say -- you never see MS recommend running Windows Server 2008 x64 plus Exchange in a memory footprint that small. I've personally run Exchange 2010 (for 3 mailboxes..) in 2.5GB RAM VM but it was sloooowwww to manage, interactive logins were glacial, but the Exchange services seemed to run without issue.

Has anyone actually applied the recommendations as they come off the report without issue? Part of this is motivated by another client who "insists" that the 8 GB memory allocated to his SQL server is "too low" (he wants 32 to 64 GB) despite peak Memory Active being just under 2.25 GB.

Are there some cases where Memory Active isn't really accurate and the apparently inactive RAM is being used for local caching or something?

[Vitaliy S.]

Hi Shawn,

I guess part of me thinks would be way too low, despite what the active memory statistics would say

What is the average and peak memory usage for this VM over 1 month? What is the current memory usage?

Part of this is motivated by another client who "insists" that the 8 GB memory allocated to his SQL server is "too low" (he wants 32 to 64 GB) despite peak Memory Active being just under 2.25 GB.

What do other performance metrics (CPU Ready, Co-Stop, Disk Latency, the number of IOPs etc.) show for this VM? Do you have any performance issues with this SQL Server VM?

Thanks!

[cyberswb]

What is the average and peak memory usage for this VM over 1 month? What is the current memory usage?

Current memory usage is about 13% now, peak over the past month was 23%.

What I find surprising is that if the "memory active" or "memory usage" statistics actually represent what the VM is really using actively for memory, then nearly all the conventional wisdom, advice, system requirements, and so on for memory are pretty much totally wrong. MS recommends 2 GB for Windows 2008 R2 and another 4 GB for Exchange 2010, plus 5 MB per mailbox. In most VM implementations I've seen active is never over 3 GB, even peak.

Anyway, that's why I'm question it because it's such a departure from conventional wisdom and vendor recommendations, which historically (for better or for worse) have been routinely considered ridiculous minimums and doubled in actual implementations.

Either memory usage/memory active is the real deal and we've been fooled all this time into using increasingly large amounts of RAM and assuming they're necessary, or those stats are an incomplete picture and there's some portion of the non-active/unused memory that are being used in ways we don't know.

Bottom line, I'm not saying it's wrong it's just such a disconnect from what I and everyone else has taken for granted that it seems like a paradigm changer.

What do other performance metrics (CPU Ready, Co-Stop, Disk Latency, the number of IOPs etc.) show for this VM? Do you have any performance issues with this SQL Server VM?

Generally performance is good and there are no complaints.

The SQL server is used for a very large application with a lot of data; on more than one occasion they've complained about performance that we generally attributed to a SAN system that was in need of an upgrade, but most of the time these problems are fixed with some kind of sheepish response about "bad queries" or "missing indexes" or some other problem unrelated to the server environment.

But I'm convinced that SQL guys are all impossible to please with performance data; they still want their own rack in the data center with physical servers they admin. Virtualization has been a loss of stature for the SQL guys.

[Vitaliy S.]

Hi Shawn,

recommends 2 GB for Windows 2008 R2 and another 4 GB for Exchange 2010, plus 5 MB per mailbox. In most VM implementations I've seen active is never over 3 GB, even peak.

Yes, that's the trickiest part, especially when you try to size the Exchange/SQL Server VM. When you look at the active memory counter, it shows you the number of memory pages that have been recently accessed by the application, without taking into consideration pages that have been pre-allocated/reserved by the application, but not currently used. If you want to know the total amount of memory held by the VM, you need to review memory consumed metric. This metric is absolutely useless for SQL Servers/Exchange, as reading back through my response above these application will always consume all memory you've allocated to these VMs.

That said, in our oversized report we are using memory active (the peak usage) + 25% of buffer to give you the recommended amount of memory for your VMs. This should help you to avoid application performance problems after reducing the amount of memory allocated to the VM. Depending on the application and it's monthly workload you may want to add additional 10-15% of buffer to the recommended amount of RAM.

Anyway, that's why I'm question it because it's such a departure from conventional wisdom and vendor recommendations, which historically (for better or for worse) have been routinely considered ridiculous minimums and doubled in actual implementations.

Either memory usage/memory active is the real deal and we've been fooled all this time into using increasingly large amounts of RAM and assuming they're necessary, or those stats are an incomplete picture and there's some portion of the non-active/unused memory that are being used in ways we don't know.

Bottom line, I'm not saying it's wrong it's just such a disconnect from what I and everyone else has taken for granted that it seems like a paradigm changer.

I fully agree with you, but I don't think that we were fooled. Installing additional memory on the physical server is not an easy task simply because you might not have available slots for additional RAM . That is why vendors might have recommended a bit more than it was usually required (in order to support future growth of the IT infrastructure).Today with the tools that are currently available, you can determine the amount of memory this is actually used by the application in your particular environment, so keeping vendors recommendations and your actual usage in mind, you can size your VMs in a proper way.

Generally performance is good and there are no complaints.

The SQL server is used for a very large application with a lot of data; on more than one occasion they've complained about performance that we generally attributed to a SAN system that was in need of an upgrade, but most of the time these problems are fixed with some kind of sheepish response about "bad queries" or "missing indexes" or some other problem unrelated to the server environment.

But I'm convinced that SQL guys are all impossible to please with performance data; they still want their own rack in the data center with physical servers they admin. Virtualization has been a loss of stature for the SQL guys.

True Our report developers (SQL guys) think in the exactly same manner!

[Daveyd]

Along these same lines, I just had an incident this morning where we migrated from an Oracle DB on a physical server to one on a VM. The Oracle DB was installed on a 2008 R2 VM and allocated 4GB of RAM and 2 vCPUs. This morning when users logged into the DB their application was slow to respond whereas it was not when the DB was on a physical server with the same allocated RAM. Looking at the RAM metrics for the past few hours, consumed was 4GB, and active was 0.9GB (Task Manager showed 3.8GB used with 300MB cached) yet the clients were painfully slow and timing out their server connections. The vendor recommended we up the RAM to 8GB. I did and the issue has gone away.

I showed the vendor that the active RAM was minimal and RAM constraint in the VM did not seem apparent but as a I stated, adding addition RAM to the VM improved client response time 100%.

[Vitaliy S.]

Ok, thanks David for your input! Do you have any ideas on how to better handle these situations (VMs with database servers inside)?

[Daveyd]

I really wish I did especially when metrics are telling me everything is fine. Not blaming Veeam, of course.

Everything you read tells you do not rely at Task manager and rely on swap/ballooning/compression rates or reservations to determine if a VM is starved for memory and active memory/usage% should be a good indication of how well a VM is running in terms of RAM but in this case it was unreliable. The vendor stated he has seen this issue befire with Oracle DBs running in VMs and the issue has always been corrected by adding more RAM. Maybe it's something to do with the way Oracle utilizes RAM, I don't know. But it would have been nice to see a smoking gun and point it out.

[pufferdude]

I too noticed these crazy-low memory recommendation and came here looking for answers and found this thread. I'm interested to know if anyone has actually followed the recommended memory allocations, because it just seems that they (at least some of them) can't possibly work correctly. For example, I have an admittedly low-use AD box at a remote site with 4GB memory and Veeam wants me to lower that to 0.2GB! Will Windows 2008R2 even BOOT with 256 *MB*? Another example... an old Win2003 VM running iMail with 750MB allocated and Veeam is recommending it be lowered to 100 *M* bytes.

I just don't understand how these recommendations can be remotely accurate or even doable.

[Vitaliy S.]

Hi Jim,

The recommendations that are given in this report are based on the average active memory utilization for your VMs. You may want to add additional 25% of memory to the recommended size while reconfiguring the VMs.

For example, I have an admittedly low-use AD box at a remote site with 4GB memory and Veeam wants me to lower that to 0.2GB!

Yes, I've seen the same behavior in my labs too... what if we put, let's say 1GBs, as a minimum recommendation in order not to show smth like 0.2 GBs? What do you think on that?

Thanks for the feedback!

[deduplicat3d]

I don't think that you should put a minimum into Veeam. I like having Veeam give the recommendations that it feels are true and then the Sys Admins can use best judgement. I have some linux servers that I've decreased to those levels because of Veeam, I was skeptical at first, but now I'm a believer!

[chris352]

Can anyone shed some light on this? Each of our hosts has 64GB or ram and we're serving some 200+ users... Exchange 2010, SBS 2011, over a dozen Xenapp hosts etc and Veeam report is telling me to drop ram???
I thought we were barely scraping by with what we had...

[Vitaliy S.]

Hi Chris,

This report is based on active memory usage counter, which shows how much memory your VMs are actively using. As far as I can see your VMs are not using all memory you've allocated to them, that is why you have that amount of RAM in the recommendations section.

Thanks!

[chris352]

I think this report is wrong. the 11th VM down from the top has 8.00 GB ram allocated to it, is running Exchange server 2012 to over 200 people and I'm supposed to allocate only 1.9GBs of ram to it? How can that be correct?

[Vitaliy S.]

Unfortunately, this report doesn't have any insight into the applications running inside your VMs, it is calculating the recommended amount of memory based on the memory usage (memory active/memory granted). Can you please open Monitor Client and see what amount of RAM your VM is actively using? Also I guess only Exchange and SQL Server VMs are affected by this, right?

[murdocmk]

I wonder what the timer is for dropping pages from the active memory list? If it's aggressive, then I can see how that might result in misleading active memory counters.

I've looked but haven't found an answer to that yet.

[deduplicat3d]

I'm assuming it is Mem.SamplePeriod, which defaults to 60 seconds.

http://pubs.vmware.com/vsphere-4-esx-vc ... butes.html

I would be careful about changing it as it appears to be used for several pieces of memory resource management.

[cparker4486]

I came to the forum having the same concerns. The ideas being presented here are hard to stomach but it certainly makes sense: regardless of what Windows believes it's using, the virtualization host knows the truth.

I'm going to setup some new RD Session Hosts and I think I will begin their RAM life at the currently recommended amount by the oversized report (for my existing single host).

[ky-lab_rat]

I'm seeing the same thing as the others have with the Oversize Report. Veeam is reporting that I need to downsize one of my VM's on memory. I circled the vm in the image below and the next image is a grab from veeam performance page showing that I should probably add ram and not remove it.

[Vitaliy S.]

The sub-report shows you memory consumed, not memory active. Can you please tell me what type of server is this? Is it a SQL Server?

[ky-lab_rat]

This is a centos linux vm running mysql. If that's the metric veeam uses then shouldn't that be mentioned or listed in the report it gives to validate the results to the user.

[Vitaliy S.]

Yes, you're right. In the next update we will allow to choose which metric to use - memory active or memory consumed. As to the report you're looking at, then MS SQL servers as well MySQL servers are known to consume almost all chunks memory you allocate to these VMs, so seeing high value for memory consumed is expected. On the other hand, memory active shows how this memory was actively used and based on the report it doesn't seem that this VM was very busy for the reported time interval.

[cffit]

I came on here with the same exact feeling as everyone else. I do know for a fact that reducing our SQL servers to what VEEAM suggests will render them unusable, but maybe servers not running databases or Exchange would be ok.

I also wondered about the CPU suggestions. It suggests I lower all my VMs to use 1 single CPU. I would think the CPU debate would be much less than the memory debate given CPU isn't handled like memory in the case of SQL and Exchange. Has anyone experimented with lowering their 2 and 4 CPU machines to 1 CPU based on the suggestions from this report?

[Vitaliy S.]

Hi Christensen,

What is the average CPU usage for your VMs that have this recommendation of lowering the number of vCPUs? Do you have all 4 vCPUs used (you can check that via vSphere Client)?

Thanks!

[cffit]

We have about 18 VMs that have either 2 or 4 CPUs (virtual sockets) assigned, and VEEAM suggests that we change them all to 1 vCPU. Some are SQL servers, some are terminal servers using RDS, and some are application servers. I'm confused how the calculations are done, becuase if these suggestions are based on averages, and averages are based on CPU use divided by 24 hours in a day, but 16 of those hours the servers have little or no usage, doesn't that skew the "average"? I need the average to be considered the average during the 8-10 hour work day when it's in use. Taking the average over a full 24 hour day can really knock down the usage numbers.

Does Windows 2008 R2 handle removing CPUs well? I'd certainly be willing to try this out and see how it goes.

[Vitaliy S.]

Yes, it does sound reasonable to specify business hours for these calculations, however we do search for peak usage (based on the average usage) and then compare that to current configuration to calculate the difference.

[fredbloggs]

Has to be stated, active memory is a very dangerous counter to report on when sizing VMs

http://blogs.vmware.com/vsphere/2013/10 ... emory.html

The issue is, whilst Veeam checks what the peak active memory counter reported you need to be a little more aware on how the application works, here's an excerpt from the above blog

This counter provides you an estimated total of amount of memory that has been touched over the sampling period. This means that depending on where you are looking at this is counter, its sampling period may be different. In vCenter for example, real-time charts show Active memory samples every 20 seconds. This means that it’s showing the estimated amount of memory that is touched by the virtual machine in the last 20 seconds. However in esxtop, the display refresh is every 5 seconds. This means that it’s showing the estimated amount of memory that is touched by the virtual machine in the last 5 seconds.

BUT – Here’s the important part - There is no way to know if the memory pages that were touched in the last number of sampling period are unique or not.

Basically, active memory may never exceed 1GB (in vcentre) but over a period of 2 minutes it could use up to 6GB of unique RAM pages and therefore you could quite easily starve your VM of much needed memory.

I'm not sure why VMware make this the default counter, nor why they've allowed the active myth to be so pervasive or the best way to report on oversized VMs, perhaps you simply shouldn't or at least include quite a big disclaimer.

[Vitaliy S.]

Hello Fred,

Yes, I agree with you and that is why we have also included memory consumed counter into our calculations.

Thank you!