Bug 1976213
| Summary: | Search filter on report causes task to crash, and sometimes OOM error | ||
|---|---|---|---|
| Product: | Red Hat Satellite | Reporter: | Gary Scarborough <gscarbor> |
| Component: | Reporting | Assignee: | Partha Aji <paji> |
| Status: | CLOSED ERRATA | QA Contact: | Satellite QE Team <sat-qe-bz-list> |
| Severity: | medium | Docs Contact: | |
| Priority: | medium | ||
| Version: | 6.9.0 | CC: | casl, ikaur, jlenz, lvrtelov, mhulan, paji, pmendezh, rlavi, zhunting |
| Target Milestone: | 6.15.0 | Keywords: | Performance, Triaged |
| Target Release: | Unused | ||
| Hardware: | Unspecified | ||
| OS: | Unspecified | ||
| Whiteboard: | |||
| Fixed In Version: | rubygem-katello-4.11.0.3-1 | Doc Type: | If docs needed, set a value |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2024-04-23 17:10:46 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
Just updating as the Satellite version is 6.9.1 Is the issue related to searching by "installed_package_name" only? This search term may need to get optimized. Or are there other search terms that cause problems? I can't answer regarding other search terms, as this was my specific need.
The search itself is something that should not be so resource intensive as to cause OOM. I understand the "runaway query" scenario, but this was very simple.
RH support provided this SQL which has no noticible performance impact:
...
select h.name,kip.name,kip.nvra
from hosts as h,
katello_host_installed_packages as khip,
katello_installed_packages as kip
where kip.id=khip.installed_package_id
and khip.host_id=h.id
and kip.name like '$1'
...
Upon review of our valid but aging backlog the Satellite Team has concluded that this Bugzilla does not meet the criteria for a resolution in the near term, and are planning to close in a month. This message may be a repeat of a previous update and the bug is again being considered to be closed. If you have any concerns about this, please contact your Red Hat Account team. Thank you. This issue persists on 6.13.2 There does not seem to be much interest in the satellite reporting features. I will mention as I have in other BZs, that features such as this, if they are not to be usable in any practical sense, should be removed from future releases. Charles, I appreciate your feedback and interest in Satellite's reporting. I am going to update the bugzilla so that it doesn't go through auto-closure so that additional investigation can be performed. There was a private comment where an engineer was looking to confirm if this persisted on 6.13 and I appreciate your feedback that it does. To your point regarding 'interest in the satellite reporting features', the Reporting feature has had over 150 bugzillas resolved since it was introduced, including over 50 enhancement requests. While it is unfortunate that this issue does exist, the data does indicate that there is interest and activity in this area. Thank you for your understanding. Upstream bug assigned to paji Upstream bug assigned to paji Moving this bug to POST for triage into Satellite since the upstream issue https://projects.theforeman.org/issues/35974 has been resolved. Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Important: Satellite 6.15.0 release), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2024:2010 |
Description of problem: Customer is running the Registered Hosts report in Satellite 6.8 and using a search term : installed_package_name = splunkforwarder. Customer has about 1200 hosts registered to satellite. The report errors out and seems to cause a OOM error killing off a sidekiq process. Version-Release number of selected component (if applicable): How reproducible: Every time certain search terms are used. Not all search terms error out. Jun 24 11:57:26 rhnsat-03 kernel: Out of memory: Kill process 1323 (sidekiq) score 454 or sacrifice child Jun 24 11:57:26 rhnsat-03 kernel: Killed process 1323 (sidekiq), UID 994, total-vm:18395528kB, anon-rss:17486112kB, file-rss:0kB, shmem-rss:0kB Jun 24 11:57:28 rhnsat-03 systemd: dynflow-sidekiq: main process exited, code=killed, status=9/KILL Jun 24 11:57:28 rhnsat-03 systemd: Unit dynflow-sidekiq entered failed state. Jun 24 11:57:28 rhnsat-03 systemd: dynflow-sidekiq failed. Jun 24 11:57:29 rhnsat-03 systemd: dynflow-sidekiq holdoff time over, scheduling restart. Jun 24 11:57:29 rhnsat-03 systemd: Stopped Foreman jobs daemon - worker on sidekiq. Jun 24 11:57:29 rhnsat-03 systemd: Starting Foreman jobs daemon - worker on sidekiq... Jun 24 11:57:32 rhnsat-03 dynflow-sidekiq@worker: 2021-06-24T15:57:32.486Z 43034 TID-b8smm INFO: GitLab reliable fetch activated! Jun 24 11:57:32 rhnsat-03 dynflow-sidekiq@worker: 2021-06-24T15:57:32.486Z 43034 TID-mnx0u INFO: Booting Sidekiq 5.2.7 with redis options {:id=>"Sidekiq-server-PID-43034", :url=>"redis://localhost:6379/0"} Jun 24 11:57:42 rhnsat-03 dynflow-sidekiq@worker: /usr/share/foreman/lib/foreman.rb:8: warning: already initialized constant Foreman::UUID_REGEXP Jun 24 11:57:42 rhnsat-03 dynflow-sidekiq@worker: /usr/share/foreman/lib/foreman.rb:8: warning: previous definition of UUID_REGEXP was here Jun 24 11:57:55 rhnsat-03 dynflow-sidekiq@worker: /opt/theforeman/tfm/root/usr/share/gems/gems/foreman_theme_satellite-7.0.1.5/app/models/concerns/distributor_version.rb:5: warning: already initialized constant Katello::Glue::Provider::DISTRIBUTOR_VERSION Jun 24 11:57:55 rhnsat-03 dynflow-sidekiq@worker: /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.1.27/app/models/katello/glue/provider.rb:3: warning: previous definition of DISTRIBUTOR_VERSION was here Jun 24 11:58:01 rhnsat-03 dynflow-sidekiq@worker: /opt/theforeman/tfm/root/usr/share/gems/gems/foreman_azure_rm-2.1.3/app/models/foreman_azure_rm/azure_rm_compute.rb:13: warning: circular argument reference - sdk Jun 24 11:58:17 rhnsat-03 systemd: Started Foreman jobs daemon - worker on sidekiq. Task export data: INput: --- job_class: TemplateRenderJob queue: default job_data: job_class: TemplateRenderJob job_id: beebaa52-1352-4772-be55-80e2ef0f7abb provider_job_id: beebaa52-1352-4772-be55-80e2ef0f7abb queue_name: default priority: arguments: - template_id: 108-Host - Registered Content Hosts gzip: false send_mail: false mail_to: casl.edu format: html input_values: '5': value: installed_package_name = splunkforwarder _aj_hash_with_indifferent_access: true _aj_hash_with_indifferent_access: true _aj_hash_with_indifferent_access: true - user_id: 35 _aj_symbol_keys: - user_id executions: 0 exception_executions: {} locale: en timezone: America/New_York enqueued_at: '2021-06-24T15:56:42Z' current_request_id: 84477708-32cb-4383-96c5-c1459b574e60 current_timezone: America/New_York current_user_id: 35 current_organization_id: 5 current_location_id: 2 output: Error: StandardError Could not transition step from pending to running, step already in running. --- - "/opt/theforeman/tfm/root/usr/share/gems/gems/algebrick-0.7.3/lib/algebrick/matchers/abstract.rb:74:in `block in assigns'" - "/opt/theforeman/tfm/root/usr/share/gems/gems/algebrick-0.7.3/lib/algebrick/matchers/abstract.rb:73:in `tap'" - "/opt/theforeman/tfm/root/usr/share/gems/gems/algebrick-0.7.3/lib/algebrick/matchers/abstract.rb:73:in `assigns'" - "/opt/theforeman/tfm/root/usr/share/gems/gems/algebrick-0.7.3/lib/algebrick/matching.rb:56:in `match_value'" - "/opt/theforeman/tfm/root/usr/share/gems/gems/algebrick-0.7.3/lib/algebrick/matching.rb:36:in `block in match?'" - "/opt/theforeman/tfm/root/usr/share/gems/gems/algebrick-0.7.3/lib/algebrick/matching.rb:35:in `each'" - "/opt/theforeman/tfm/root/usr/share/gems/gems/algebrick-0.7.3/lib/algebrick/matching.rb:35:in `match?'" - "/opt/theforeman/tfm/root/usr/share/gems/gems/algebrick-0.7.3/lib/algebrick/matching.rb:23:in `match'" - "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/execution_plan/steps/error.rb:13:in `new'" - "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action.rb:501:in `set_error'" - "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action.rb:559:in `execute_run'" - "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/action.rb:285:in `execute'" - "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:18:in `block (2 levels) in execute'" - "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/execution_plan/steps/abstract.rb:167:in `with_meta_calculation'" - "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:17:in `block in execute'" - "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:32:in `open_action'" - "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:16:in `execute'" - "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/director.rb:68:in `execute'" - "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/executors/sidekiq/worker_jobs.rb:11:in `block (2 levels) in perform'" - "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/executors.rb:18:in `run_user_code'" - "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/executors/sidekiq/worker_jobs.rb:9:in `block in perform'" - "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/executors/sidekiq/worker_jobs.rb:25:in `with_telemetry'" - "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/executors/sidekiq/worker_jobs.rb:8:in `perform'" - "/opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.7/lib/dynflow/executors/sidekiq/serialization.rb:27:in `perform'" - "/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:192:in `execute_job'" - "/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:165:in `block (2 levels) in process'" - "/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/middleware/chain.rb:128:in `block in invoke'" - "/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/middleware/chain.rb:133:in `invoke'" - "/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:164:in `block in process'" - "/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:137:in `block (6 levels) in dispatch'" - "/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/job_retry.rb:109:in `local'" - "/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:136:in `block (5 levels) in dispatch'" - "/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq.rb:37:in `block in '" - "/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:132:in `block (4 levels) in dispatch'" - "/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:250:in `stats'" - "/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:127:in `block (3 levels) in dispatch'" - "/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/job_logger.rb:8:in `call'" - "/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:126:in `block (2 levels) in dispatch'" - "/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/job_retry.rb:74:in `global'" - "/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:125:in `block in dispatch'" - "/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/logging.rb:48:in `with_context'" - "/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/logging.rb:42:in `with_job_hash_context'" - "/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:124:in `dispatch'" - "/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:163:in `process'" - "/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:83:in `process_one'" - "/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:71:in `run'" - "/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/util.rb:16:in `watchdog'" - "/opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/util.rb:25:in `block in safe_thread'" - "/opt/theforeman/tfm/root/usr/share/gems/gems/logging-2.3.0/lib/logging/diagnostic_context.rb:474:in `block in create_with_logging_context'" Actual results: report fails to complete, OOM error Expected results: Report generates as expected