Red Hat Satellite engineering is moving the tracking of its product development work on Satellite to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "Satellite project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs will be migrated starting at the end of May. If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "Satellite project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/SAT-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
Bug 1976213 - Search filter on report causes task to crash, and sometimes OOM error
Summary: Search filter on report causes task to crash, and sometimes OOM error
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Satellite
Classification: Red Hat
Component: Reporting
Version: 6.9.0
Hardware: Unspecified
OS: Unspecified
medium
medium
Target Milestone: 6.15.0
Assignee: Partha Aji
QA Contact: Satellite QE Team
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2021-06-25 14:25 UTC by Gary Scarborough
Modified: 2024-10-01 18:47 UTC (History)
9 users (show)

Fixed In Version: rubygem-katello-4.11.0.3-1
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2024-04-23 17:10:46 UTC
Target Upstream Version:
Embargoed:

Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Foreman Issue Tracker 35974 0 Normal New Host search by installed_package_name invokes OOM killer 2023-09-14 18:40:48 UTC
Red Hat Issue Tracker SAT-15167 0 None None None 2024-04-11 17:16:56 UTC
Red Hat Issue Tracker SAT-20104 0 None None None 2023-09-14 18:41:51 UTC
Red Hat Product Errata RHSA-2024:2010 0 None None None 2024-04-23 17:10:48 UTC

Description Gary Scarborough 2021-06-25 14:25:13 UTC 
Description of problem:

Customer is running the Registered Hosts report in Satellite 6.8 and using a search term :  installed_package_name = splunkforwarder.  Customer has about 1200 hosts registered to satellite.  The report errors out and seems to cause a OOM error killing off a sidekiq process.

Version-Release number of selected component (if applicable):


How reproducible:

Every time certain search terms are used.  Not all search terms error out.



Jun 24 11:57:26 rhnsat-03 kernel: Out of memory: Kill process 1323 (sidekiq) score 454 or sacrifice child
Jun 24 11:57:26 rhnsat-03 kernel: Killed process 1323 (sidekiq), UID 994, total-vm:18395528kB, anon-rss:17486112kB, file-rss:0kB, shmem-rss:0kB
Jun 24 11:57:28 rhnsat-03 systemd: dynflow-sidekiq: main process exited, code=killed, status=9/KILL
Jun 24 11:57:28 rhnsat-03 systemd: Unit dynflow-sidekiq entered failed state.
Jun 24 11:57:28 rhnsat-03 systemd: dynflow-sidekiq failed.
Jun 24 11:57:29 rhnsat-03 systemd: dynflow-sidekiq holdoff time over, scheduling restart.
Jun 24 11:57:29 rhnsat-03 systemd: Stopped Foreman jobs daemon - worker on sidekiq.
Jun 24 11:57:29 rhnsat-03 systemd: Starting Foreman jobs daemon - worker on sidekiq...
Jun 24 11:57:32 rhnsat-03 dynflow-sidekiq@worker: 2021-06-24T15:57:32.486Z 43034 TID-b8smm INFO: GitLab reliable fetch activated!
Jun 24 11:57:32 rhnsat-03 dynflow-sidekiq@worker: 2021-06-24T15:57:32.486Z 43034 TID-mnx0u INFO: Booting Sidekiq 5.2.7 with redis options {:id=>"Sidekiq-server-PID-43034", :url=>"redis://localhost:6379/0"}
Jun 24 11:57:42 rhnsat-03 dynflow-sidekiq@worker: /usr/share/foreman/lib/foreman.rb:8: warning: already initialized constant Foreman::UUID_REGEXP
Jun 24 11:57:42 rhnsat-03 dynflow-sidekiq@worker: /usr/share/foreman/lib/foreman.rb:8: warning: previous definition of UUID_REGEXP was here
Jun 24 11:57:55 rhnsat-03 dynflow-sidekiq@worker: /opt/theforeman/tfm/root/usr/share/gems/gems/foreman_theme_satellite-7.0.1.5/app/models/concerns/distributor_version.rb:5: warning: already initialized constant Katello::Glue::Provider::DISTRIBUTOR_VERSION
Jun 24 11:57:55 rhnsat-03 dynflow-sidekiq@worker: /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.1.27/app/models/katello/glue/provider.rb:3: warning: previous definition of DISTRIBUTOR_VERSION was here
Jun 24 11:58:01 rhnsat-03 dynflow-sidekiq@worker: /opt/theforeman/tfm/root/usr/share/gems/gems/foreman_azure_rm-2.1.3/app/models/foreman_azure_rm/azure_rm_compute.rb:13: warning: circular argument reference - sdk
Jun 24 11:58:17 rhnsat-03 systemd: Started Foreman jobs daemon - worker on sidekiq.


Task export data:

INput:

---
job_class: TemplateRenderJob
queue: default
job_data:
  job_class: TemplateRenderJob
  job_id: beebaa52-1352-4772-be55-80e2ef0f7abb
  provider_job_id: beebaa52-1352-4772-be55-80e2ef0f7abb
  queue_name: default
  priority: 
  arguments:
  - template_id: 108-Host - Registered Content Hosts
    gzip: false
    send_mail: false
    mail_to: casl.edu
    format: html
    input_values:
      '5':
        value: installed_package_name = splunkforwarder
        _aj_hash_with_indifferent_access: true
      _aj_hash_with_indifferent_access: true
    _aj_hash_with_indifferent_access: true
  - user_id: 35
    _aj_symbol_keys:
    - user_id
  executions: 0
  exception_executions: {}
  locale: en
  timezone: America/New_York
  enqueued_at: '2021-06-24T15:56:42Z'
current_request_id: 84477708-32cb-4383-96c5-c1459b574e60
current_timezone: America/New_York
current_user_id: 35
current_organization_id: 5
current_location_id: 2



output:



 Error:

StandardError

Could not transition step from pending to running, step already in running.

---
- "/opt/theforeman/tfm/root/usr/share/gems/gems/algebrick-0.7.3/lib/algebrick/matchers/abstract.rb:74:in
  `block in assigns'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/algebrick-0.7.3/lib/algebrick/matchers/abstract.rb:73:in
  `tap'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/algebrick-0.7.3/lib/algebrick/matchers/abstract.rb:73:in
  `assigns'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/algebrick-0.7.3/lib/algebrick/matching.rb:56:in
  `match_value'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/algebrick-0.7.3/lib/algebrick/matching.rb:36:in
  `block in match?'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/algebrick-0.7.3/lib/algebrick/matching.rb:35:in
  `each'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/algebrick-0.7.3/lib/algebrick/matching.rb:35:in
  `match?'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/algebrick-0.7.3/lib/algebrick/matching.rb:23:in
  `match'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/execution_plan/steps/error.rb:13:in
  `new'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action.rb:501:in
  `set_error'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action.rb:559:in
  `execute_run'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action.rb:285:in
  `execute'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:18:in
  `block (2 levels) in execute'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/execution_plan/steps/abstract.rb:167:in
  `with_meta_calculation'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:17:in
  `block in execute'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:32:in
  `open_action'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:16:in
  `execute'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/director.rb:68:in
  `execute'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/executors/sidekiq/worker_jobs.rb:11:in
  `block (2 levels) in perform'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/executors.rb:18:in
  `run_user_code'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/executors/sidekiq/worker_jobs.rb:9:in
  `block in perform'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/executors/sidekiq/worker_jobs.rb:25:in
  `with_telemetry'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/executors/sidekiq/worker_jobs.rb:8:in
  `perform'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/executors/sidekiq/serialization.rb:27:in
  `perform'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:192:in
  `execute_job'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:165:in
  `block (2 levels) in process'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/middleware/chain.rb:128:in
  `block in invoke'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/middleware/chain.rb:133:in
  `invoke'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:164:in
  `block in process'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:137:in
  `block (6 levels) in dispatch'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/job_retry.rb:109:in
  `local'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:136:in
  `block (5 levels) in dispatch'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq.rb:37:in
  `block in '"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:132:in
  `block (4 levels) in dispatch'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:250:in
  `stats'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:127:in
  `block (3 levels) in dispatch'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/job_logger.rb:8:in
  `call'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:126:in
  `block (2 levels) in dispatch'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/job_retry.rb:74:in
  `global'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:125:in
  `block in dispatch'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/logging.rb:48:in
  `with_context'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/logging.rb:42:in
  `with_job_hash_context'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:124:in
  `dispatch'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:163:in
  `process'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:83:in
  `process_one'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:71:in
  `run'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/util.rb:16:in
  `watchdog'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/util.rb:25:in
  `block in safe_thread'"
- "/opt/theforeman/tfm/root/usr/share/gems/gems/logging-2.3.0/lib/logging/diagnostic_context.rb:474:in
  `block in create_with_logging_context'"









Actual results:

report fails to complete, OOM error


Expected results:

Report generates as expected
Comment 4 Gary Scarborough 2021-06-25 14:36:04 UTC 
Just updating as the Satellite version is 6.9.1
Comment 5 Marek Hulan 2021-09-09 15:14:46 UTC 
Is the issue related to searching by "installed_package_name" only? This search term may need to get optimized. Or are there other search terms that cause problems?
Comment 6 Charles Slivkoff 2021-09-09 15:38:58 UTC 
I can't answer regarding other search terms, as this was my specific need.

The search itself is something that should not be so resource intensive as to cause OOM.  I understand the "runaway query" scenario, but this was very simple.

RH support provided this SQL which has no noticible performance impact:

...

select h.name,kip.name,kip.nvra
                from hosts as h,
                katello_host_installed_packages as khip,
                katello_installed_packages as kip
                where kip.id=khip.installed_package_id
                and khip.host_id=h.id
                and kip.name like '$1'

...
Comment 11 Brad Buckingham 2023-07-21 21:06:39 UTC 
Upon review of our valid but aging backlog the Satellite Team has concluded that this Bugzilla does not meet the criteria for a resolution in the near term, and are planning to close in a month. This message may be a repeat of a previous update and the bug is again being considered to be closed. If you have any concerns about this, please contact your Red Hat Account team.  Thank you.
Comment 12 Charles Slivkoff 2023-07-26 19:48:13 UTC 
This issue persists on 6.13.2

There does not seem to be much interest in the satellite reporting features.

I will mention as I have in other BZs, that features such as this, if they are not to be usable in any practical sense, should be removed from future releases.
Comment 13 Brad Buckingham 2023-09-01 20:55:28 UTC 
Charles,

I appreciate your feedback and interest in Satellite's reporting.  I am going to update the bugzilla so that it doesn't go through auto-closure so that additional investigation can be performed.  There was a private comment where an engineer was looking to confirm if this persisted on 6.13 and I appreciate your feedback that it does.

To your point regarding 'interest in the satellite reporting features', the Reporting feature has had over 150 bugzillas resolved since it was introduced, including over 50 enhancement requests.  While it is unfortunate that this issue does exist, the data does indicate that there is interest and activity in this area.

Thank you for your understanding.
Comment 16 Bryan Kearney 2024-01-11 04:01:22 UTC 
Upstream bug assigned to paji
Comment 17 Bryan Kearney 2024-01-11 04:01:25 UTC 
Upstream bug assigned to paji
Comment 18 Bryan Kearney 2024-01-18 16:01:22 UTC 
Moving this bug to POST for triage into Satellite since the upstream issue https://projects.theforeman.org/issues/35974 has been resolved.
Comment 23 errata-xmlrpc 2024-04-23 17:10:46 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Important: Satellite 6.15.0 release), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2024:2010
Note You need to log in before you can comment on or make changes to this bug.