Bug 1296638

The setting of this functionality broke by:

commit 49dc2581fed7acfc50c5a7d4c984289de0031906
Author: Keenan Brock <kbrock>
Date:   Sat Nov 21 14:33:53 2015 -0500

    path_to_my_worker_settings

I'm still tracking down the reading of that variable and if it is/has been respected

Alex,

Are you still seeing behavior like this?
We updated the code that reads/defaults these parameters.
We also rewrote the configuration system.

(In reply to Keenan Brock from comment #3)
> Alex,
> 
> Are you still seeing behavior like this?
> We updated the code that reads/defaults these parameters.
> We also rewrote the configuration system.

Hi Keenan,

I reviewed 5.6.0.5 (beta2.4) and still see the memory threshold in the UI as 200MiB for C&U collectors but I have witnessed the RSS memory usage greater than 340MiB during tests with a large vmware provider.  Additionally I do not see a :memory_threshold: configured option under ems_metrics_collector_worker - defaults so I assume it is defaulting to queue_worker_base which is 400MiB.

Perhaps the patches haven't made there way into 5.6.0.5 yet?

This is a configuration issue in the core.

Also suggesting moving this to 5.7

Dan, I noticed this Bz while looking at another issue and thought it was in the queue area... Can you have someone look at this?

I tried tracking this one down but couldn't understand the code in app/views/ops/_settings_workers_tab.html.haml and
app/controllers/ops_controller/settings/common.rb

I believe the problem is that the UI code is walking the hashes for the existing settings and new settings and assuming a specific structure.

    :queue_worker_base:
      :defaults:
        :cpu_usage_threshold: 100.percent
        :dequeue_method: :drb
        :memory_threshold: 500.megabytes
        :poll_method: :normal
        :queue_timeout: 10.minutes
      :ems_metrics_collector_worker:
        :defaults:
          :count: 2
          :nice_delta: 3
          :poll_method: :escalate

I believe it's trying to look at 
[:queue_worker_base][:ems_metrics_collector_worker:][:defaults][:memory_threshold], failing to find it and defaulting back to 200.megabytes.

I don't understand where 200 is coming from though since the fallback seems to be 400 megabytes if it's not found (in common.rb:1024):

      qwb[:ems_metrics_collector_worker] ||= {}
      qwb[:ems_metrics_collector_worker][:defaults] ||= {}
      w = qwb[:ems_metrics_collector_worker][:defaults]
      raw = @edit[:current].get_raw_worker_setting(:MiqEmsMetricsCollectorWorker)
      w[:count] = raw[:defaults][:count] || 2
      w[:memory_threshold] = rails_method_to_human_size(raw[:defaults][:memory_threshold] || 400.megabytes)
      @sb[:ems_metrics_collector_threshold] = []

Correction:

Dan, I noticed this Bz while looking at another issue and thought it was in the WRONG queue/assignment... Can you have someone look at this?

https://github.com/ManageIQ/manageiq/pull/12999

Note, this 400 MB value reported in this BZ was subsequently modified to 500 in https://bugzilla.redhat.com/show_bug.cgi?id=1391687 via
https://github.com/ManageIQ/manageiq/pull/12484

Sorry, changing description to reflect that change.

Note, that ems_refresh_core_worker just like Metrics Collector Workers and many other workers are inheriting the 500 MB memory_threshold from queue_worker_base and would probably exhibit similar problems as reported in this BZ.

New commit detected on ManageIQ/manageiq/master:
https://github.com/ManageIQ/manageiq/commit/1f2687fc89328eccb37724dbeddf6311e9dffbac

commit 1f2687fc89328eccb37724dbeddf6311e9dffbac
Author:     Harpreet Kataria <hkataria>
AuthorDate: Mon Dec 5 17:02:16 2016 -0500
Commit:     Harpreet Kataria <hkataria>
CommitDate: Mon Dec 5 17:02:16 2016 -0500

    Added a missing default memeory threshold setting
    
    Added a missing default memeory threshold setting for C & U Data Collectors that was causing drop down to the firt item in list as selected value by default.
    
    https://bugzilla.redhat.com/show_bug.cgi?id=1296638

 config/settings.yml | 1 +
 1 file changed, 1 insertion(+)

while connected to a 1000 vm RHVM environment, this seems to have been fixed in 580x. 

reference: https://gist.github.com/arcolife/648c83a7f53ee6a706dd8fda278080e1

[----] W, [2017-04-11T07:50:57.579104 #40011:1045140]  WARN -- : MIQ(MiqServer#validate_worker) Worker [ManageIQ::Providers::Redhat::InfraManager::MetricsCollectorWorker] with ID: [42], PID: [21634], GUID: [ef1e1b5a-1eac-11e7-8366-001a4a22391a] process memory usage [420320000] exceeded limit [419430400], requesting worker to exit


checks out against the UI params:

      :ems_metrics_collector_worker:
        :defaults:
          :count: 2
          :memory_threshold: 400.megabytes
          :nice_delta: 3
          :poll_method: :escalate