Host Swap memory

Monitoring and reporting for Veeam Backup & Replication, VMware vSphere and Microsoft Hyper-V

Host Swap memory

Veeam Logoby risingflight » Wed Dec 23, 2015 11:07 am

Hi Everyone

I am using Veeam Monitor

I get warnings for 3 Esxi hosts.

1. Host 1 & 2
Warning : Host Hardware Status
Host hardware status for Memory subsystem has changed to Warning level

Memory Usage : 148725
Capacity : 262109

( I can see the warning in Vcenter Server)

2.Host 3
Warning : Host Swap memory
"Memory Swap Used" (70 MB) is above a defined threshold (64 MB)

Memory Usage : 102045
Capacity : 262109
( There is no warning for this in vcenter server)
risingflight
Influencer
 
Posts: 23
Liked: never
Joined: Sun Nov 29, 2015 6:04 pm
Full Name: RisingFlight

Re: Host Swap memory

Veeam Logoby Shestakov » Wed Dec 23, 2015 11:17 am

Hi,
All alarms have description and resolution fields describing possible reasons and actions to resolve. Please take a look.

1. It was triggered by some of the host`s hardware sensors. Please check the "Hardware" tab for the alarmed host for more info. You can also review the related VMware KB.
It`s not related to memory though.

2. In second case applications inside some VMs demand much more memory than the VMs have, so overhead is swapped to the disk. Probably those VMs are undersized. I would consider running Undersized VMs report to check it and get statistics-based suggestion of how much memory to add.

Thanks!
Shestakov
Veeam Software
 
Posts: 4861
Liked: 396 times
Joined: Wed May 21, 2014 11:03 am
Location: Saint Petersburg
Full Name: Nikita Shestakov

Re: Host Swap memory

Veeam Logoby Vitaliy S. » Wed Dec 23, 2015 11:22 am

risingflight wrote:( There is no warning for this in vcenter server)

Just wanted to add that Veeam ONE has a lot more predefined alarms compared to the vCenter Server list.
Vitaliy S.
Veeam Software
 
Posts: 19564
Liked: 1104 times
Joined: Mon Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov

Re: Host Swap memory

Veeam Logoby readie » Tue Jan 26, 2016 4:55 pm

risingflight wrote:2.Host 3
Warning : Host Swap memory
"Memory Swap Used" (70 MB) is above a defined threshold (64 MB)

Memory Usage : 102045
Capacity : 262109
( There is no warning for this in vcenter server)

I too am troubled by 'Host Swap Memory' although I have tweaked ALL my VMs so that they now have no alarms at all (only just now, so perhaps it might take some time to filter up to the host).
Interested in your thresholds, as mine are set (by default, I haven't altered them) to 64KB warning and 128KB error.
My values do seem very low (I'm not too worried about KB of memory here or there, in hosts which have 96GB RAM) so should I just tweak thresholds up a bit?
My actual values (reported by Veeam One) are 106MB, 16MB and 8MB (all well over my threshold of 128KB).
Bob Eadie
Computer Manager at Bedford School, UK (since 1999).
Veeam user since 2009.
readie
Expert
 
Posts: 156
Liked: 30 times
Joined: Sun Dec 05, 2010 9:29 am
Full Name: Bob Eadie

Re: Host Swap memory

Veeam Logoby Shestakov » Tue Jan 26, 2016 11:09 pm

Hello Bob,
Generally 106MB od memory swapped is too much. I would check the memory usage on those hosts with the triggered alarms. Also try to run Undersized VMs report.
You may have some vms being undresized. Thanks!
Shestakov
Veeam Software
 
Posts: 4861
Liked: 396 times
Joined: Wed May 21, 2014 11:03 am
Location: Saint Petersburg
Full Name: Nikita Shestakov

Re: Host Swap memory

Veeam Logoby readie » Thu Jan 28, 2016 9:39 am

Run 'Undersized VMs' report and nothing. Our infrastructure is generally over-provisioned, with RAM and CPU to spare.
An interesting 'feature' of the VEEAM ONE thresholds, is that for each VM the 'VM memory swap usage' thresholds are 64MB and 128MB (warning and error) but for the HOST 'Host swap memory' they are 64KB and 128KB - that's KB not MB.
I would have expected the HOST thresholds to be a few times the VM thresholds, as (if I understand it correctly) the Host Swap Memory is the sum of the VM swap memories???
Was this just a slip by someone setting up thresholds, or have I misunderstood?
I am really having difficulty 'getting rid of' these alarms (even given that our system is overprovisioned - but of course I might have something else setup incorrectly).
Bob Eadie
Computer Manager at Bedford School, UK (since 1999).
Veeam user since 2009.
readie
Expert
 
Posts: 156
Liked: 30 times
Joined: Sun Dec 05, 2010 9:29 am
Full Name: Bob Eadie

Re: Host Swap memory

Veeam Logoby Shestakov » Thu Jan 28, 2016 12:17 pm

Let`s go a bit deeper here.
If you have memory swap alarm triggered, some VMs had a real lack of memory.
Here is how memory reclamation works:
Image

Besides the limitation on the reclaimed memory size, both page sharing and ballooning take time to reclaim memory. The pagesharing speed depends on the page scan rate and the sharing opportunity. Ballooning speed relies on the guest operating system’s response time for memory allocation. In contrast, hypervisor swapping is a guaranteed technique to reclaim a specific amount of memory within a specific amount of time. However, hypervisor swapping may severely penalize guest performance. This occurs when the hypervisor has no knowledge about which guest physical pages should be swapped out, and the swapping may cause unintended interactions with the native memory management policies in the guest operating system.

Here is a great paper about Memory Resource Management in VMware to read more.

Have you also observed "High balloon memory utilization" and/or other memory-related alarms?

readie wrote:( There is no warning for this in vcenter server)

Check if there were any pikes of this metric in vcenter when the Veeam ONE alarm is triggered.

readie wrote:An interesting 'feature' of the VEEAM ONE thresholds, is that for each VM the 'VM memory swap usage' thresholds are 64MB and 128MB (warning and error) but for the HOST 'Host swap memory' they are 64KB and 128KB - that's KB not MB.
I would have expected the HOST thresholds to be a few times the VM thresholds, as (if I understand it correctly) the Host Swap Memory is the sum of the VM swap memories???
Was this just a slip by someone setting up thresholds, or have I misunderstood?

Looks like a bug. I`ve checked my lab, I have both measured in MB. What product version are you at?

Thanks!
Shestakov
Veeam Software
 
Posts: 4861
Liked: 396 times
Joined: Wed May 21, 2014 11:03 am
Location: Saint Petersburg
Full Name: Nikita Shestakov

Re: Host Swap memory

Veeam Logoby readie » Fri Jan 29, 2016 9:37 am

Thanks for detailed reply. Our system NEVER seems to do ballooning at all - I have always assumed because we are over-provisioned with more RAM in hosts than VMs require.
Will read the article, thanks.
I'm on verion 9 of Veeam ONE, but have upgraded and upgraded since about V7, or 6, or 5 . . . so I wonder whether new UPGRADES do not reset these thresholds. I am tempted to change mine to MB . . . but was waiting for your indication that is what you have.
Bob Eadie
Computer Manager at Bedford School, UK (since 1999).
Veeam user since 2009.
readie
Expert
 
Posts: 156
Liked: 30 times
Joined: Sun Dec 05, 2010 9:29 am
Full Name: Bob Eadie

Re: Host Swap memory

Veeam Logoby Shestakov » Fri Jan 29, 2016 10:42 am

How often did the alarm trigger? If you don`t have a lack of memory and ballooning is not used at all, there is no reason for hypervisor to swap the memory. So I wonder if you can see memory swap in vSphere client performance chart.

As for thresholds, you can definitely change them.
Upgrades themselves should not cause random threshold changes. Have you upgraded directly from v7 or from v8?
Shestakov
Veeam Software
 
Posts: 4861
Liked: 396 times
Joined: Wed May 21, 2014 11:03 am
Location: Saint Petersburg
Full Name: Nikita Shestakov

Re: Host Swap memory

Veeam Logoby readie » Fri Jan 29, 2016 2:25 pm

The HOST alarms are permanent (but not now, I have changed KB to MB). Very occasionally a VM shows an alarm for 'VM swap memory usage' when it goes above 64M/128M . . . but a few have perhaps 8 or 10 MB of swap if I look in VM resource usage, even just after a reboot.
Should VM ballooning trigger before using swap? Since I am over provisioned, is there any way I can check whether ballooning is doing what it should do?
Upgrading - I am just guessing (since I've never changed them) that the thresholds were KB several versions ago, and upgrading (we generally go for the latest V, so have gone from 5 to 6, 6 to 7, 7 to 8 etc) has never reset them to MB??
You may gather than I am only just starting to use these alarms (generally have seen so many that I couldn't face trying to find what they all were and eliminate them). BUT I now think I'm on top of the last one, and any new alarms WILL indicate something is up.
Bob Eadie
Computer Manager at Bedford School, UK (since 1999).
Veeam user since 2009.
readie
Expert
 
Posts: 156
Liked: 30 times
Joined: Sun Dec 05, 2010 9:29 am
Full Name: Bob Eadie

Re: Host Swap memory

Veeam Logoby Shestakov » Sat Jan 30, 2016 9:12 pm

readie wrote:Should VM ballooning trigger before using swap? Since I am over provisioned, is there any way I can check whether ballooning is doing what it should do?

In general yes, ballooning reclamation technique starts working before the memory swap, which is actually the last resort.
Ballooning may be turned off/on manually.

If it is turned on, the way to check if it actually works is to run a performance(memory) burner in the lab and inspect ballooning performance/alarms.
Shestakov
Veeam Software
 
Posts: 4861
Liked: 396 times
Joined: Wed May 21, 2014 11:03 am
Location: Saint Petersburg
Full Name: Nikita Shestakov


Return to Veeam ONE



Who is online

Users browsing this forum: No registered users and 6 guests