Bug 1281855

Description of problem:
Connected CFME to xlarge vmware provider and the MiqEmsRefreshCoreWorker exits due to exceeding its default memory threshold of 400MiB RSS.  A 5.4.3.1 appliance does not exhibit this same behavior when connected to the same sized provider.  A 5.4.3.1 appliance appears to use ~20MiB less memory with the MiqEmsRefreshCoreWorker rather than the 5.5.0.9/10 appliances.  Additionally the constant recycling of of this worker consumes an excessive amount of CPU.

xlarge vmware provider is:
10,000 VMs, 5,000 online VMs
200 hosts
221 data stores
20 clusters
200 resource pools

Version-Release number of selected component (if applicable):
5.5.0.9
5.5.0.10

How reproducible:
Inconsistent between runs.  

I have witnessed three different behaviors:
1. MiqEmsRefreshCoreWorker exceeds its memory threshold during refresh and consistently there afterwards resulting in wasted cpu resources.
2. MiqEmsRefreshCoreWorker exceeds its memory threshold during a refresh, thus exiting consistently, but once refresh is complete the last started worker stabilizes at a hair below the memory threshold ~398MiB RSS.
3. MiqEmsRefreshCoreWorker does not exceed its memory threshold even during a refresh operation.

Scenario 1 is the worst case, witnessed on 5.5.0.10.  Scenario 3 is the ideal situation and is what occurs on 5.4.3.1, and has been witnessed on 5.5.0.10 once.


Steps to Reproduce:
1.
2.
3.

Actual results:
With large scale vmware providers MiqEmsRefreshCoreWorker will sometimes exceed its 400MiB RSS Memory threshold causing a "worker exit storm" consuming a tremendous amount of CPU from an appliance.  There is no notification to the end user or administrator that this is occurring, without viewing the logs or seeing saw tooth memory usage or high cpu usage of the appliance itself.

Expected results:
MiqEmsRefreshCoreWorker not to strain the appliance's resources.  Solutions to this issue are:
1. Release note or instructions to raise MiqEmsRefreshCoreWorker default memory threshold when connecting to xlarge vmware sized providers
2. Reduce memory foot print of MiqEmsRefreshCoreWorker.
3. Raise default Memory threshold of MiqEmsRefreshCoreWorker in the configuration.  (There are other workers who exceed their memory defaults when connected to this sized environment so this is the least desirable solution IMO.)

Additional info:
In the instance where a MiqEmsRefreshCoreWorker was consistently exiting and restarting even after an initial refresh, I had tuned the worker memory threshold to 500MiB.  The worker ended up stabilizing at 416MiB RSS Memory.  This is odd as in other Memory Baseline tests I have seen the worker stabilize at ~395-398MiB RSS with it only breaching during the initial refresh operation.  The memory utilization appears to be inconsistent and right at the boarder of it's memory threshold with this sized environment.