Veeam VMware Collector: Health Service Handle Count thresho

[BenCc]

Hello,

I'm getting the alerts:

Veeam VMware Collector: Health Service Handle Count threshold
Veeam VMware Collector: Health Service (Monitoring Host) Private Bytes threshold

I googled and found:
http://www.veeam.com/kb1783

But there is no "Handles" in the Task Manager on Windows Server 2012 R2.

Am I supposed to install a second Veeam Collector to solve this problem, or what am I supposed to do?

Thanks a lot.

[sergey.g]

Hi,

You can add handles column into the task manager by right-clicking on the headers bar and selecting "Select columns"


By default threshold for these monitors is 10000 handles, and 2GB private bytes.

For handles values above 10000 is not something unexpected, Microsoft recommends to increase threshold for handles to 30K for SCOM agents with management packs which produce big number of monitored objects.

More than 2GB of private bytes is also a high, but not unexpectedly high value. When there are a lot of monitored objects this threshold could be easily breached. Check if this Collector system is alowed to consume more memory and increase private bytes threshold to 3GB. By the way, the health service or monitoring host processes should peak at these values, but not always stay at that level.

Check if you have any other errors or alerts for this server. Check Operations Manager event log on the Collector Server, if everything is working smoothly, you can just increase the threshold if you are OK with such memory usage on this Collector Server. If you notice any other errors about failed workflows or dropped data - it's an indication that SCOM agent reached it's performance capacity and cannot scale anymore. In this case solution would be to add another collector into your monitoring infrastructure, this should significantly off-load your current Collector system.

Let me know if you have any other questions.
Thanks.

[BenCc]

Hello,

Thanks for the answer. I have override the value of the Health Service Handle Count to 30000.
Our SCOM server has 124 GB of RAM, so 2 GB really isn't alot. I have changed the threshhold to 10 GB.

Am I supposed to do anything else, or should the problem be resolved now?

EDIT: so after changing the Health Service Handle Count to 30000, I got an alert:
Health Service Handle Count metric is currently 57289 which exceeds the threshold.
are 57000 handles normal? We have around 200 VMs in our vSphere Environment.

[sergey.g]

Hi,

So, since it's a Management Server - Health Service will not be restarted when crossing default Microsoft thresholds (30K and 1,5 Gb), however in your particular case I would perform a Health Service restart, we need to check if this 57K handles is something that has been accumulating for some time or there is an issue with some monitored objects. Could you restart the System Center Management service or Microsoft Monitoring Agent (depending on your System Center version and deployment)? Let us know if handles are climbing quickly to 50K again.

Also it would be interesting to get a sense of how busy your System Center health service is, could you check number of workflows in the "Veeam for VMware > Veeam Collectors > Performance Views > Workflow Count" performance view?

Handles could climb to these values, but should not stay there and should not climb to 50K quickly.

Thanks.

[BenCc]

I restarted the Management server.

The Health service is now sitting at around 3k - 4k handles in the Task Manager after the restart. I will check if it will climb again.

The workflow count was sitting at between 15k - 20k for the past few days before the restart.

[BenCc]

Hello, it's me again.

The Monitoring host will now climb over 30k handles, even after restarting the server after like 2 hours it's already over 30k.

I saw in the Task Manager thereare multiple Monitoring Hosts. Interestingly the one which has over 30k handles is the one running the notifcation service account. Should I just Override the value?

[sergey.g]

Hi,

Thank you for the feedback. Is it possible to check if handles grow even further? After 30k, or you have a Health Service restart in place? If it's a management server, then health service should not restart automatically.

From our collaborative work with Microsoft during the development process we had a sense that 30k handles is something to take a closer look, but if growth stopped at certain point after 30k it's OK and expected.

The thing is that this is happening internally in the .NET code, there are garbage cleaning processes which should clean up and free handles, but to understand the procedure even Microsoft needs to perform a lot of investigations, take several memory dumps to understand wh handles are not beeing released or why they are not being released fast enough to prevent constant handles growth.

It would be interesting to know number of workflows on this server. You can check them in SCOM in the "Operations Manager > Management Group Details > Performance Data" for a management server and in the "Operations Manager > Agent Details > Agent Performance" for SCOM agent.

I would appreciate the above additional information to better understand the nature of handles growth on this Veeam Collector.

Thanks.

[BenCc]

Hello sergey

Yes this is the Management server.

The handles seem to be growing the whole time, currently at 94k:



Under "Operations Manager > Management Group Details > Performance Data" I can't find any Performance Counter which says the number of worklows?

[sergey.g]

Hi,

I would recommend to move this management server out of the notifications resource pool. Our Management Pack can cause high handle count for MonitoringHost.exe, but usualy it's not a 100k number. I'm afraid you have some kind of combination of management pack and notification workflows which prevent garbage collector from releasing handles in a timely manner. Do you use this serv_scom_notify RunAs account only for notifications or also in some other RunAs profiles? Maybe as a default action account?

There is a special tool from microsoft's sysinternals package - handle.exe (https://technet.microsoft.com/en-us/sys ... s/bb896655)

Could you run it with the following paramters and provide us with the output (you can send the file via Diricet Message, our open a support case with our tech support team - they'll help you to send us necessary information)

Code: Select all

.\handle.exe -s

And then also this:

Code: Select all

.\handle.exe -p monitoringhost -u -a

Workflows counter should be there, it's in the core System Center management packs

[BenCc]

Hello,

The handles have raised to 275k over the night.

The account serv_scom_notify is only assigned to the notification Profile, to no other. The Default action account is an other user.

I have sent you the Output of the handle.exe in a private message.

I found the workflow Counter. Workflows are constantly between 16k - 17k.