vCenter Memory usage

[logixcom]

I have opened a support case as well, but wanted to ask the community.

We upgraded to vCenter 7.0u3 a while back and since then we have been observing what appears to be a slow memory leak causing a heap overflow in memory for vsphere-ui service. It usually takes about a week and a half and the UI becomes unresponsive. We have to reboot the vCenter appliance to get it back to good. Users see a spinning circle when trying to log in, those logged in see very slow performance. vCenter is set to "Medium" for size which is more than enough for our environment (12 hosts, 230 VMs). We have just added an additional 2GB of memory to the appliance (total 26GB) and also doubled the heap size for the vsphere-ui service to try and help, results are still unknown; as we just did it. What lead us to this was an alarm in vCenter "Health changed to yellow" and also in the core appliance screen showing heap memory 90% on the services status screen. We used an hprof file analyzer to see that com.veeam.backupservice.VeeamBackupServiceImpl was the number one usage of memory, 276MB.

Has anyone seen this before?

[Mildur]

Hello logixcom

Please don't forget to provide the case number when you post about a technical issue here in the forum. I could find your case number myself. The case was opened yesterday evening. I strongly suggest to keep working with our support team until we have a conclusion.

Did you also opened a case with VMware to check the appliance from their side?

Best,
Fabian

[d.artzen]

Hi logixcom,

while we did not have the problems with the Vcenter UI becoming unresponsive, I have seen the system warning for vsphere-ui in our systems as well, including the information about >90% heap memory usage of that service. We did not research directly into the reason for this, so I can't say if it's also com.veeam.backupservice.VeeamBackupServiceImpl in our case. We solved this by increasing the memory for the vsphere-ui service to double according to this VMware KB article: https://kb.vmware.com/s/article/2150757. This has worked for us so far, the warning did not appear again. There is one downside however: The change is not persistent,after a reboot of the Vcenter Appliance the value is reset and you have to do the steps in the KB again, otherwise the warning reappears after some time.

Best regards
Daniel

[logixcom]

Case #06289889

[Mildur]

Thank you. Let's see what we can find out from the logs.

Best,
Fabian

[logixcom]

Been a long journey, but so far the fix was to reboot all Veeam Servers. We don't reboot them often for obvious reasons (other than to patch Windows once a quarter). VMware said it was a Veeam issue and Veeam said they don't look at VMWare logs, so neither was much assistance. But so far, stopping all jobs first (seemed to be the difference from past reboots) and then rebooting all Veeam servers seems to have helped the issue and the session counts and memory usage from Veeam has dropped to the point of no more outages for 2 weeks now. Previously vCenter would "lock" up once a week almost like clock work. We did also add 2gb more memory to vCenter and tweaked the heap as mentioned in the KB above, but it did not help first go around, perhaps because the change did not stick after a reboot of vCenter appliance. Now we have changed to just restarting the VMWare UI service when needed rather than a full reboot of vCenter appliance.