Using a Management Server outside of resource pools

[Jarv]

Hi,
As previous posts on here have shown I have successfully updated my MP from 57 to 6.0 and everything is working "OK". I say OK because I am actually having performance problems. This isn’t down to Veeam (6.0 seems much more efficient than 5.7 thanks!) but more because as well as monitoring Veeam on my 2 current MP's I also have a number of agentless servers, a gateway to another company and also some CISCO UCS monitoring.
Watching the video from Marnix Wolf I noticed that as well as monitoring Veeam from Management Servers (which I am doing) he is also recommending these MS should be taken out of the standard resource pools. I am OK with doing this - creating a new "Veeam" resource pool for a new management server and using Powershell when needed to take the new MS out of the standard pools.
Firstly I perhaps don’t understand SCOM enough to know what taking the MS out of the notifications pool does for example? Also even though I currently have TWO management servers doing this role could I get away with one MS outside of the normal resource pools and just as a failover keep the collector server on my other 2 but only to be used in case I need to failover? I guess I am asking do I need TWO new MS in the new resource pool or get away with 1.
thanks

[sergey.g]

Hi Steve,

I would say that this is a subject for a test, but let us know parameters of your virtual infrastructure (number of hosts, clusters, vcenters, datastores, VMs) and we can do a rough estimation of whether dedicated SCOM Management server can support such a load. If you are taking a management server out of default resource pool - it means that it's not participating is some management tasks, such as network or linux monitoring for example. Notifications pool is responsible for sending notifications via different notification channels. Agentless monitoring is assigned to a certain servers when you configure it, so if you want to remove this task from a certain server, just make sure it's not specified as a proxy agent for the agentless-managed computer.

With respect to your question about failover. We always recommend to have a stand-by collector server for failover, but I'm afraid if dedicated management server is working with almost critical load, then in case of failover, the stand-by server which is also doing additional stuff won't be able to handle all of the additional load. So, I would recommend to use two regular management servers as a stand-by collectors while a dedicated management server is performing all of the Veeam work when it's healthy.

I hope this answers your question. Let me know if you have any other questions.

Thanks

[Jarv]

Sergey,
We have 7 vCenters, 10 clusters, 29 hosts and 980 VM’s listed in the console at the moment.
My two current management servers (“X” and “Y”) are pretty meaty. They are physical boxes with 32GB of RAM each and a couple of NIC.
However this is complicated by –
Both are collector servers for Veeam
Around 20 agentless (Windows 2000 servers) split over the two MS (X and Y)
Monitoring of 3 CISCO UCS implementations (on X and Y)
A gateway connection (for 20 servers) from another domain off server X
With all of this as a whole I am finding the health of my SCOM servers normally grey state or red.
However I tried moving the entire collecting load to Y and then put all the other parts (UCS etc) into Maintenance Mode as a test. With this Y does seem to be able to cope on its own. Hence why I was thinking of moving the Veeam stuff off this anyway and create a new virtual MS (“Z”) outside of the pools so Veeam is never affected (I dare say I would also be better creating a new MS for the gateway also but that’s another story)
Thanks!

[Jarv]

Just to put a little more meat on the bones from the last post.
Since moving all my veeam collection to one server (and putting everything else into MM) it does seem to be "coping" in the respect the actual management servers and have bothed stayed in a happy green state..
That said I am recieving 4 or 5 "Veeam Collector: Health Service Handle Count threshold - Health Service Handle Count has exceeded the threshold" critical alerts each day on that server.
One thing I should add from the above stats is that two MS obviously also monitor 1200 windows servers split over them (which obviously includes the 980 VM's)

From the above I do feel if I built a further VM Management Server that does not have the stress of any of the above and only veeam collection it would work a lot better

[sergey.g]

Hello Steve,

With respect to Handle Count monitor - I would recommend to check this performance counter "Handle count" of the health service on the affected server. The default threshold is 10000, Microsoft set this as a threshold and they beleive that it's a critical amount of handles. But I beleive that if you see 11000 handles in your environment and everything seems to work properly, you are absolutely free to override the threshold, so that the monitor isn't firing an alert very often.

A dedicated Veeam MS would definitely make you infrastructure more healthy, just because 3 SCOM servers is better than 2. But I think virtual SCOM server should have at least same number of CPUs as physical MS, enough memory and fast storage, so that it can handle all veaam collection tasks same way as physical server .

Also keep in mind that in case of failure - all your servers will not be able to handle the load as it happening right now when everything is enabled. I would recommend to do the thing you are already considering - to setup another SCOM server which can take gateway tasks. In this case you can make all 3 non-Veeam servers a stand-by Veeam collectors and in this case even in case of dedicated Veeam MS failure, these 3 servers should be able to handle the load.

If you are not interested right now in fault-tolerance of your monitoring infrastructure, then I beleive that a dedicated Veeam MS should do its job.

Thank you very much for detailed information about your infrastructure, we really appreciate a real-life example of much improved performance of our collectors.

Thanks.

[Jarv]

Sergey thanks again for that excellent information.
I have successfully installed a collector to a third management server in my TEST environment and then taken this server out of the relevant resource pools before going live.
However I then realised I had put the collector on there but not the extensions which I presume is just as important for putting the workload on the new server??
A question on this though. I removed the extensions from one MS and just installed again on the new one. What I had done though of course was lose all my connections to the ESX boxes etc.. Is there any way of exporting that config first or is it a case of having to do it again manually? Also can I point my redundant collectors (as things stand) to the new extensions server or do I have to reinstall to fill in the server name again? I couldn’t find anything obvious in the registry to just repoint.
Maybe all of this is possible with the PowerShell extensions?? (Just as a side note a Whitebooard Friday on what you can do with VEShell would be great!)
Just another small point and this is on my two test environments and live. In the SCOM console my Veeam Collector versions have never changed from 1421 to 6.0.0.1423 after applying the patch to them. In the extensions UI however they correctly show as 1423. Interestingly if I look in the registry they also seem to indicate they are 1421.

[sergey.g]

Hi,

With respect to extensions service - you do not need to transfer it, it can be installed anywhere in the network, it just need an access to vSphere and to all collectors. If you already removed the old one and installed a new one, you can re-assign collectors by changing LicenseServer parameter in the "HKLM\Software\Veeam\Veeam Virtualization Extensions for VMware" registry key, just change it to the new extensions service host and restart the collector.

You cannot transfer credentials from one extensions server to another, but you can transfer configuration(you'll have to manually re-configure passwords for connections) although this option is not supported and we are not recommending it, use on your own risk(I would recommend to re-create connections manually): you just need to copy vemState.db.xml file in folder "C:\ProgramData\Veeam\Veeam Virtualization Extensions for VMware" from one server to another and restart the extensions service.

Good idea about some webinar/video on things you can do with VEShell, although writing PowerShell cmdlets on the whiteboard could be quite challenging

With respect to collector version - it looks like a bug, most likely our patch setup application is not updating registry entries, we'll take a look, thank you very much for reporting the issue.

Thanks.

[Jarv]

I just thought I would add another important aspect of my move of collector service to a server outside of the "normal" resource pools. This isnt a Veeam issue/problem at all and you could have the same problem if you created a new resource pool for network monitoring for example. Chances are you will encounter it though -

I created a new resource pool "All Veeam collector service Management Servers"
I added my new MS and then removed it from the other resource pools

The management server then went into a grey state. This down to some run as credentials not transferring across to the new MS/resource pool

In the console I got one alert - "System Center Health Credentials Not Set". This is helpful but it dosent tell you the actual account only the baffling SSID. To resolve this I ran the following SCOM powershell -

Get-ScomRunAsAccount | Sort Name | % {$string = $null;$_.SecureStorageId | % {
$string = $string + "{0:X2}" -f $_}
$_.Name;" $string"
}

In my case this pointed to the “Global Service Monitor Run As Account Configuration”

I checked this account under Administration and it was only being distributed to the AMSRP. Therefore I added the new Veeam resource pool. Still the MS was in grey.
For some reason no alert was created but if you check the event log for the MS you have added for event 1108 I found some more accounts that needed sorting out. In the event log it actually states the account name but it dosent actually match word for word the SCOM console version so again use the powershell to match the SSID. In my case I also had to modify the following -
Data Warehouse Action Account
Data Warehouse Report Deployment Account

After that MS went into green state, I installed the Veeam collector and bingo!
Sorry long post but hopefully help someone.

[Alec King]

Steve,

That's great information, thanks very much! We are planning to leverage Management Server Resource Pools in the Veeam MP in an upcoming version - so that info will be useful for us.

Thanks again and please keep posting your comments and questions in the MP forum!