Host Swap memory

[risingflight]

Hi Everyone

I am using Veeam Monitor

I get warnings for 3 Esxi hosts.

1. Host 1 & 2
Warning : Host Hardware Status
Host hardware status for Memory subsystem has changed to Warning level

Memory Usage : 148725
Capacity : 262109

( I can see the warning in Vcenter Server)

2.Host 3
Warning : Host Swap memory
"Memory Swap Used" (70 MB) is above a defined threshold (64 MB)

Memory Usage : 102045
Capacity : 262109
( There is no warning for this in vcenter server)

[Shestakov]

Hi,
All alarms have description and resolution fields describing possible reasons and actions to resolve. Please take a look.

1. It was triggered by some of the host`s hardware sensors. Please check the "Hardware" tab for the alarmed host for more info. You can also review the related VMware KB.
It`s not related to memory though.

2. In second case applications inside some VMs demand much more memory than the VMs have, so overhead is swapped to the disk. Probably those VMs are undersized. I would consider running Undersized VMs report to check it and get statistics-based suggestion of how much memory to add.

Thanks!

[Vitaliy S.]

( There is no warning for this in vcenter server)

Just wanted to add that Veeam ONE has a lot more predefined alarms compared to the vCenter Server list.

[readie]

2.Host 3
Warning : Host Swap memory
"Memory Swap Used" (70 MB) is above a defined threshold (64 MB)

Memory Usage : 102045
Capacity : 262109
( There is no warning for this in vcenter server)

I too am troubled by 'Host Swap Memory' although I have tweaked ALL my VMs so that they now have no alarms at all (only just now, so perhaps it might take some time to filter up to the host).
Interested in your thresholds, as mine are set (by default, I haven't altered them) to 64KB warning and 128KB error.
My values do seem very low (I'm not too worried about KB of memory here or there, in hosts which have 96GB RAM) so should I just tweak thresholds up a bit?
My actual values (reported by Veeam One) are 106MB, 16MB and 8MB (all well over my threshold of 128KB).

[Shestakov]

Hello Bob,
Generally 106MB od memory swapped is too much. I would check the memory usage on those hosts with the triggered alarms. Also try to run Undersized VMs report.
You may have some vms being undresized. Thanks!

[readie]

Run 'Undersized VMs' report and nothing. Our infrastructure is generally over-provisioned, with RAM and CPU to spare.
An interesting 'feature' of the VEEAM ONE thresholds, is that for each VM the 'VM memory swap usage' thresholds are 64MB and 128MB (warning and error) but for the HOST 'Host swap memory' they are 64KB and 128KB - that's KB not MB.
I would have expected the HOST thresholds to be a few times the VM thresholds, as (if I understand it correctly) the Host Swap Memory is the sum of the VM swap memories???
Was this just a slip by someone setting up thresholds, or have I misunderstood?
I am really having difficulty 'getting rid of' these alarms (even given that our system is overprovisioned - but of course I might have something else setup incorrectly).

[Shestakov]

Let`s go a bit deeper here.
If you have memory swap alarm triggered, some VMs had a real lack of memory.
Here is how memory reclamation works:


Besides the limitation on the reclaimed memory size, both page sharing and ballooning take time to reclaim memory. The pagesharing speed depends on the page scan rate and the sharing opportunity. Ballooning speed relies on the guest operating system’s response time for memory allocation. In contrast, hypervisor swapping is a guaranteed technique to reclaim a specific amount of memory within a specific amount of time. However, hypervisor swapping may severely penalize guest performance. This occurs when the hypervisor has no knowledge about which guest physical pages should be swapped out, and the swapping may cause unintended interactions with the native memory management policies in the guest operating system.

Here is a great paper about Memory Resource Management in VMware to read more.

Have you also observed "High balloon memory utilization" and/or other memory-related alarms?

( There is no warning for this in vcenter server)

Check if there were any pikes of this metric in vcenter when the Veeam ONE alarm is triggered.

An interesting 'feature' of the VEEAM ONE thresholds, is that for each VM the 'VM memory swap usage' thresholds are 64MB and 128MB (warning and error) but for the HOST 'Host swap memory' they are 64KB and 128KB - that's KB not MB.
I would have expected the HOST thresholds to be a few times the VM thresholds, as (if I understand it correctly) the Host Swap Memory is the sum of the VM swap memories???
Was this just a slip by someone setting up thresholds, or have I misunderstood?

Looks like a bug. I`ve checked my lab, I have both measured in MB. What product version are you at?

Thanks!

[readie]

Thanks for detailed reply. Our system NEVER seems to do ballooning at all - I have always assumed because we are over-provisioned with more RAM in hosts than VMs require.
Will read the article, thanks.
I'm on verion 9 of Veeam ONE, but have upgraded and upgraded since about V7, or 6, or 5 . . . so I wonder whether new UPGRADES do not reset these thresholds. I am tempted to change mine to MB . . . but was waiting for your indication that is what you have.

[Shestakov]

How often did the alarm trigger? If you don`t have a lack of memory and ballooning is not used at all, there is no reason for hypervisor to swap the memory. So I wonder if you can see memory swap in vSphere client performance chart.

As for thresholds, you can definitely change them.
Upgrades themselves should not cause random threshold changes. Have you upgraded directly from v7 or from v8?

[readie]

The HOST alarms are permanent (but not now, I have changed KB to MB). Very occasionally a VM shows an alarm for 'VM swap memory usage' when it goes above 64M/128M . . . but a few have perhaps 8 or 10 MB of swap if I look in VM resource usage, even just after a reboot.
Should VM ballooning trigger before using swap? Since I am over provisioned, is there any way I can check whether ballooning is doing what it should do?
Upgrading - I am just guessing (since I've never changed them) that the thresholds were KB several versions ago, and upgrading (we generally go for the latest V, so have gone from 5 to 6, 6 to 7, 7 to 8 etc) has never reset them to MB??
You may gather than I am only just starting to use these alarms (generally have seen so many that I couldn't face trying to find what they all were and eliminate them). BUT I now think I'm on top of the last one, and any new alarms WILL indicate something is up.

[Shestakov]

Should VM ballooning trigger before using swap? Since I am over provisioned, is there any way I can check whether ballooning is doing what it should do?

In general yes, ballooning reclamation technique starts working before the memory swap, which is actually the last resort.
Ballooning may be turned off/on manually.

If it is turned on, the way to check if it actually works is to run a performance(memory) burner in the lab and inspect ballooning performance/alarms.