Description of problem: Customer is running the Registered Hosts report in Satellite 6.8 and using a search term : installed_package_name = splunkforwarder. Customer has about 1200 hosts registered to satellite. The report errors out and seems to cause a OOM error killing off a sidekiq process. Version-Release number of selected component (if applicable): How reproducible: Every time certain search terms are used. Not all search terms error out. Jun 24 11:57:26 rhnsat-03 kernel: Out of memory: Kill process 1323 (sidekiq) score 454 or sacrifice child Jun 24 11:57:26 rhnsat-03 kernel: Killed process 1323 (sidekiq), UID 994, total-vm:18395528kB, anon-rss:17486112kB, file-rss:0kB, shmem-rss:0kB Jun 24 11:57:28 rhnsat-03 systemd: dynflow-sidekiq: main process exited, code=killed, status=9/KILL Jun 24 11:57:28 rhnsat-03 systemd: Unit dynflow-sidekiq entered failed state. Jun 24 11:57:28 rhnsat-03 systemd: dynflow-sidekiq failed. Jun 24 11:57:29 rhnsat-03 systemd: dynflow-sidekiq holdoff time over, scheduling restart. Jun 24 11:57:29 rhnsat-03 systemd: Stopped Foreman jobs daemon - worker on sidekiq. Jun 24 11:57:29 rhnsat-03 systemd: Starting Foreman jobs daemon - worker on sidekiq... Jun 24 11:57:32 rhnsat-03 dynflow-sidekiq@worker: 2021-06-24T15:57:32.486Z 43034 TID-b8smm INFO: GitLab reliable fetch activated! Jun 24 11:57:32 rhnsat-03 dynflow-sidekiq@worker: 2021-06-24T15:57:32.486Z 43034 TID-mnx0u INFO: Booting Sidekiq 5.2.7 with redis options {:id=>"Sidekiq-server-PID-43034", :url=>"redis://localhost:6379/0"} Jun 24 11:57:42 rhnsat-03 dynflow-sidekiq@worker: /usr/share/foreman/lib/foreman.rb:8: warning: already initialized constant Foreman::UUID_REGEXP Jun 24 11:57:42 rhnsat-03 dynflow-sidekiq@worker: /usr/share/foreman/lib/foreman.rb:8: warning: previous definition of UUID_REGEXP was here Jun 24 11:57:55 rhnsat-03 dynflow-sidekiq@worker: /opt/theforeman/tfm/root/usr/share/gems/gems/foreman_theme_satellite-7.0.1.5/app/models/concerns/distributor_version.rb:5: warning: already initialized constant Katello::Glue::Provider::DISTRIBUTOR_VERSION Jun 24 11:57:55 rhnsat-03 dynflow-sidekiq@worker: /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.1.27/app/models/katello/glue/provider.rb:3: warning: previous definition of DISTRIBUTOR_VERSION was here Jun 24 11:58:01 rhnsat-03 dynflow-sidekiq@worker: /opt/theforeman/tfm/root/usr/share/gems/gems/foreman_azure_rm-2.1.3/app/models/foreman_azure_rm/azure_rm_compute.rb:13: warning: circular argument reference - sdk Jun 24 11:58:17 rhnsat-03 systemd: Started Foreman jobs daemon - worker on sidekiq. Task export data: INput: --- job_class: TemplateRenderJob queue: default job_data: job_class: TemplateRenderJob job_id: beebaa52-1352-4772-be55-80e2ef0f7abb provider_job_id: beebaa52-1352-4772-be55-80e2ef0f7abb queue_name: default priority: arguments: - template_id: 108-Host - Registered Content Hosts gzip: false send_mail: false mail_to: casl.edu format: html input_values: '5': value: installed_package_name = splunkforwarder _aj_hash_with_indifferent_access: true _aj_hash_with_indifferent_access: true _aj_hash_with_indifferent_access: true - user_id: 35 _aj_symbol_keys: - user_id executions: 0 exception_executions: {} locale: en timezone: America/New_York enqueued_at: '2021-06-24T15:56:42Z' current_request_id: 84477708-32cb-4383-96c5-c1459b574e60 current_timezone: America/New_York current_user_id: 35 current_organization_id: 5 current_location_id: 2 output: Error: StandardError Could not transition step from pending to running, step already in running. --- - "/opt/theforeman/tfm/root/usr/share/gems/gems/algebrick-0.7.3/lib/algebrick/matchers/abstract.rb:74:in `block in assigns'" - "/opt/theforeman/tfm/root/usr/share/gems/gems/algebrick-0.7.3/lib/algebrick/matchers/abstract.rb:73:in `tap'" - "/opt/theforeman/tfm/root/usr/share/gems/gems/algebrick-0.7.3/lib/algebrick/matchers/abstract.rb:73:in `assigns'" - "/opt/theforeman/tfm/root/usr/share/gems/gems/algebrick-0.7.3/lib/algebrick/matching.rb:56:in `match_value'" - "/opt/theforeman/tfm/root/usr/share/gems/gems/algebrick-0.7.3/lib/algebrick/matching.rb:36:in `block in match?'" - "/opt/theforeman/tfm/root/usr/share/gems/gems/algebrick-0.7.3/lib/algebrick/matching.rb:35:in `each'" - "/opt/theforeman/tfm/root/usr/share/gems/gems/algebrick-0.7.3/lib/algebrick/matching.rb:35:in `match?'" - "/opt/theforeman/tfm/root/usr/share/gems/gems/algebrick-0.7.3/lib/algebrick/matching.rb:23:in `match'" - "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/execution_plan/steps/error.rb:13:in `new'" - "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action.rb:501:in `set_error'" - "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action.rb:559:in `execute_run'" - "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action.rb:285:in `execute'" - "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:18:in `block (2 levels) in execute'" - "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/execution_plan/steps/abstract.rb:167:in `with_meta_calculation'" - "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:17:in `block in execute'" - "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:32:in `open_action'" - "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:16:in `execute'" - "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/director.rb:68:in `execute'" - "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/executors/sidekiq/worker_jobs.rb:11:in `block (2 levels) in perform'" - "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/executors.rb:18:in `run_user_code'" - "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/executors/sidekiq/worker_jobs.rb:9:in `block in perform'" - "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/executors/sidekiq/worker_jobs.rb:25:in `with_telemetry'" - "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/executors/sidekiq/worker_jobs.rb:8:in `perform'" - "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/executors/sidekiq/serialization.rb:27:in `perform'" - "/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:192:in `execute_job'" - "/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:165:in `block (2 levels) in process'" - "/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/middleware/chain.rb:128:in `block in invoke'" - "/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/middleware/chain.rb:133:in `invoke'" - "/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:164:in `block in process'" - "/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:137:in `block (6 levels) in dispatch'" - "/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/job_retry.rb:109:in `local'" - "/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:136:in `block (5 levels) in dispatch'" - "/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq.rb:37:in `block in '" - "/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:132:in `block (4 levels) in dispatch'" - "/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:250:in `stats'" - "/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:127:in `block (3 levels) in dispatch'" - "/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/job_logger.rb:8:in `call'" - "/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:126:in `block (2 levels) in dispatch'" - "/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/job_retry.rb:74:in `global'" - "/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:125:in `block in dispatch'" - "/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/logging.rb:48:in `with_context'" - "/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/logging.rb:42:in `with_job_hash_context'" - "/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:124:in `dispatch'" - "/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:163:in `process'" - "/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:83:in `process_one'" - "/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:71:in `run'" - "/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/util.rb:16:in `watchdog'" - "/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/util.rb:25:in `block in safe_thread'" - "/opt/theforeman/tfm/root/usr/share/gems/gems/logging-2.3.0/lib/logging/diagnostic_context.rb:474:in `block in create_with_logging_context'" Actual results: report fails to complete, OOM error Expected results: Report generates as expected
Just updating as the Satellite version is 6.9.1
Is the issue related to searching by "installed_package_name" only? This search term may need to get optimized. Or are there other search terms that cause problems?
I can't answer regarding other search terms, as this was my specific need. The search itself is something that should not be so resource intensive as to cause OOM. I understand the "runaway query" scenario, but this was very simple. RH support provided this SQL which has no noticible performance impact: ... select h.name,kip.name,kip.nvra from hosts as h, katello_host_installed_packages as khip, katello_installed_packages as kip where kip.id=khip.installed_package_id and khip.host_id=h.id and kip.name like '$1' ...
Upon review of our valid but aging backlog the Satellite Team has concluded that this Bugzilla does not meet the criteria for a resolution in the near term, and are planning to close in a month. This message may be a repeat of a previous update and the bug is again being considered to be closed. If you have any concerns about this, please contact your Red Hat Account team. Thank you.
This issue persists on 6.13.2 There does not seem to be much interest in the satellite reporting features. I will mention as I have in other BZs, that features such as this, if they are not to be usable in any practical sense, should be removed from future releases.