Bug 2063717
| Summary: | puma sometimes throws Errno::ENOENT: No such file or directory @ rb_sysopen - /usr/share/foreman/tmp/cache/0E3/781/.permissions_check.220838320.2072127.637105 | ||
|---|---|---|---|
| Product: | Red Hat Satellite | Reporter: | Jan Jansky <jjansky> |
| Component: | Infrastructure | Assignee: | Eric Helms <ehelms> |
| Status: | CLOSED ERRATA | QA Contact: | Radek Mynar <rmynar> |
| Severity: | medium | Docs Contact: | |
| Priority: | unspecified | ||
| Version: | 6.9.6 | CC: | ahumbe, apatel, arunas.z.vaznevicius, bbuckingham, bkearney, ehelms, fperalta, hyu, jhutar, jpasqual, jpathan, jsenkyri, lstejska, mhulan, pmendezh, rlavi, rmynar, satellite6-bugs, saydas |
| Target Milestone: | 6.15.0 | Keywords: | Performance, Triaged |
| Target Release: | Unused | ||
| Hardware: | Unspecified | ||
| OS: | Unspecified | ||
| Whiteboard: | |||
| Fixed In Version: | Doc Type: | If docs needed, set a value | |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2024-04-23 17:11:03 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
See if this KCS helps https://access.redhat.com/solutions/3782821 Yes, I can try to a look it This issue can happen if we use the same non-admin user to make API calls concurrently and continuously for more than 2 minutes because the cache life is 2 minutes. The race condition occurs when 1 or more API request trying to read the deleted cache and raise error.
How to reproduce the issue:
1. Open a terminal (Either in satellite or in other host) and then make multiple concurrent requests to query the smart proxies.
~~~
irb
require 'rest_client'
250.times {10.times { Thread.new { begin; RestClient::Resource.new("https://satellite.example.com/api/smart_proxies", user: "username", password: "pass", timeout: 3600, open_timeout: 3600, verify_ssl: OpenSSL::SSL::VERIFY_NONE).get; rescue StandardError => e; p e.message; end }}; sleep 2}
~~~
2. In Satellite run, tail the production.log.
~~~
tail -f /var/log/foreman/production.log | grep -i "No such file"
~~~
3. After some time you should start seeing the errors like below:
~~~
2023-04-26T18:02:31 [I|app|42c1284e] Backtrace for 'Action failed' error (Errno::ENOENT): No such file or directory @ rb_sysopen - /usr/share/foreman/tmp/cache/10D/6F1/.permissions_check.234040.2314370.945374
2023-04-26T18:02:31 [I|app|10a508c4] Backtrace for 'Action failed' error (Errno::ENOENT): No such file or directory @ rb_sysopen - /usr/share/foreman/tmp/cache/10D/6F1/.permissions_check.234060.2314364.423521
2023-04-26T18:02:31 [I|app|a51a7938] Backtrace for 'Action failed' error (Errno::ENOENT): No such file or directory @ rb_sysopen - /usr/share/foreman/tmp/cache/10D/6F1/.permissions_check.233940.2314370.586758
~~~
Additional info:
To make the error easier to reproduce, you can reduce the "delay" to something like "10.seconds".
/usr/share/foreman/app/models/user.rb
~~~
def taxonomy_and_child_ids(taxonomies)
delay = Rails.env.test? ? 0 : 2.minutes <======================
Rails.cache.fetch("user/#{id}/taxonomy_and_child_ids/#{taxonomies}", expires_in: delay) do
top_level = send(taxonomies) + taxonomies.to_s.classify.constantize.unscoped.select { |tax| tax.ignore?('user') }
top_level.each_with_object([]) do |taxonomy, ids|
ids.concat taxonomy.subtree_ids
end.uniq
end
end
~~~
*** Bug 2122461 has been marked as a duplicate of this bug. *** We believe that at a certain scale, the file cache cannot handle the traffic. To alleviate this, we need to introduce a Redis based caching option for users to enable. Redis is already a part of the deployment architecture used by both Dynflow and Pulp for content caching. Bulk setting Target Milestone = 6.15.0 where sat-6.15.0+ is set. Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Important: Satellite 6.15.0 release), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2024:2010 |
Description of problem: During normal operations puma sometimes throw Errno::ENOENT: No such file or directory @ rb_sysopen - /usr/share/foreman/tmp/cache/0E3/781/.permissions_check.220838320.2072127.637105 ... 2022-03-03T06:05:09 [I|app|5d13507c] Completed 500 Internal Server Error in 1129ms (Views: 0.2ms | ActiveRecord: 38.9ms | Allocations: 87301) Version-Release number of selected component (if applicable): satellite-6.9.6.1-1.el7sat.noarch How reproducible: Only on customer side Actual results: 2022-03-03T06:05:08 [I|app|5d13507c] Started GET "/api/v2/hosts/548/errata?page=1&per_page=1" for 10.196.65.30 at 2022-03-03 06:05:08 +0000 2022-03-03T06:05:08 [I|app|5d13507c] Processing by Katello::Api::V2::HostErrataController#index as HTML 2022-03-03T06:05:08 [I|app|5d13507c] Parameters: {"page"=>"1", "per_page"=>"1", "api_version"=>"v2", "host_id"=>"548"} 2022-03-03T06:05:09 [W|app|5d13507c] Scoped order is ignored, it's forced to be batch order. 2022-03-03T06:05:09 [I|app|5d13507c] Authorized user orchit_prod(orchit_prod) 2022-03-03T06:05:09 [E|app|5d13507c] Errno::ENOENT: No such file or directory @ rb_sysopen - /usr/share/foreman/tmp/cache/0E3/781/.permissions_check.220838320.2072127.637105 5d13507c | /opt/rh/rh-ruby25/root/usr/share/ruby/fileutils.rb:1057:in `initialize' 5d13507c | /opt/rh/rh-ruby25/root/usr/share/ruby/fileutils.rb:1057:in `open' 5d13507c | /opt/rh/rh-ruby25/root/usr/share/ruby/fileutils.rb:1057:in `rescue in block in touch' 5d13507c | /opt/rh/rh-ruby25/root/usr/share/ruby/fileutils.rb:1053:in `block in touch' 5d13507c | /opt/rh/rh-ruby25/root/usr/share/ruby/fileutils.rb:1051:in `each' 5d13507c | /opt/rh/rh-ruby25/root/usr/share/ruby/fileutils.rb:1051:in `touch' 5d13507c | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.4/lib/active_support/core_ext/file/atomic.rb:65:in `probe_stat_in' 5d13507c | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.4/lib/active_support/core_ext/file/atomic.rb:35:in `block in atomic_write' 5d13507c | /opt/rh/rh-ruby25/root/usr/share/ruby/tempfile.rb:295:in `open' 5d13507c | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.4/lib/active_support/core_ext/file/atomic.rb:24:in `atomic_write' 5d13507c | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.4/lib/active_support/cache/file_store.rb:87:in `write_entry' 5d13507c | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.4/lib/active_support/cache/strategy/local_cache.rb:150:in `write_entry' 5d13507c | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.4/lib/active_support/cache.rb:465:in `block in write' 5d13507c | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.4/lib/active_support/cache.rb:686:in `block in instrument' 5d13507c | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.4/lib/active_support/notifications.rb:182:in `instrument' 5d13507c | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.4/lib/active_support/cache.rb:686:in `instrument' 5d13507c | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.4/lib/active_support/cache.rb:463:in `write' 5d13507c | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.4/lib/active_support/cache.rb:720:in `save_block_result_to_cache' 5d13507c | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.4/lib/active_support/cache.rb:332:in `fetch' 5d13507c | /usr/share/foreman/app/models/user.rb:522:in `taxonomy_and_child_ids' 5d13507c | /usr/share/foreman/app/models/user.rb:536:in `location_and_child_ids' 5d13507c | /usr/share/foreman/app/models/taxonomies/location.rb:22:in `block in <class:Location>' 5d13507c | /opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.4/lib/active_record/relation.rb:412:in `instance_exec' 5d13507c | /opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.4/lib/active_record/relation.rb:412:in `block in _exec_scope' 5d13507c | /opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.4/lib/active_record/relation.rb:784:in `_scoping' 5d13507c | /opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.4/lib/active_record/relation.rb:412:in `_exec_scope' 5d13507c | /opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.4/lib/active_record/scoping/named.rb:192:in `block in scope' 5d13507c | /usr/share/foreman/app/models/user.rb:497:in `my_locations' 5d13507c | /usr/share/foreman/app/controllers/concerns/application_shared.rb:28:in `block in set_taxonomy' 5d13507c | /usr/share/foreman/app/controllers/concerns/application_shared.rb:27:in `each' 5d13507c | /usr/share/foreman/app/controllers/concerns/application_shared.rb:27:in `set_taxonomy' 5d13507c | /usr/share/foreman/app/controllers/concerns/application_shared.rb:18:in `set_current_user' 5d13507c | /usr/share/foreman/app/controllers/concerns/foreman/controller/authentication.rb:17:in `authenticate' 5d13507c | /usr/share/foreman/app/controllers/api/base_controller.rb:192:in `authorize' 5d13507c | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.4/lib/active_support/callbacks.rb:428:in `block in make_lambda' 5d13507c | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.4/lib/active_support/callbacks.rb:200:in `block (2 levels) in halting' 5d13507c | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.4/lib/abstract_controller/callbacks.rb:34:in `block (2 levels) in <module:Callbacks>' 5d13507c | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.4/lib/active_support/callbacks.rb:201:in `block in halting' 5d13507c | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.4/lib/active_support/callbacks.rb:513:in `block in invoke_before' 5d13507c | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.4/lib/active_support/callbacks.rb:513:in `each' 5d13507c | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.4/lib/active_support/callbacks.rb:513:in `invoke_before' 5d13507c | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.4/lib/active_support/callbacks.rb:110:in `block in run_callbacks' 5d13507c | /usr/share/foreman/app/models/concerns/foreman/thread_session.rb:32:in `clear_thread' 5d13507c | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.4/lib/active_support/callbacks.rb:121:in `block in run_callbacks' 5d13507c | /usr/share/foreman/app/controllers/concerns/foreman/controller/topbar_sweeper.rb:12:in `set_topbar_sweeper_controller' 5d13507c | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.4/lib/active_support/callbacks.rb:121:in `block in run_callbacks' 5d13507c | /opt/theforeman/tfm/root/usr/share/gems/gems/audited-4.9.0/lib/audited/sweeper.rb:14:in `around' 5d13507c | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.4/lib/active_support/callbacks.rb:121:in `block in run_callbacks' 5d13507c | /opt/theforeman/tfm/root/usr/share/gems/gems/audited-4.9.0/lib/audited/sweeper.rb:14:in `around' 5d13507c | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.4/lib/active_support/callbacks.rb:121:in `block in run_callbacks' 5d13507c | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.4/lib/active_support/callbacks.rb:139:in `run_callbacks' 5d13507c | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.4/lib/abstract_controller/callbacks.rb:41:in `process_action' 5d13507c | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.4/lib/action_controller/metal/rescue.rb:22:in `process_action' 5d13507c | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.4/lib/action_controller/metal/instrumentation.rb:33:in `block in process_action' 5d13507c | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.4/lib/active_support/notifications.rb:180:in `block in instrument' 5d13507c | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.4/lib/active_support/notifications/instrumenter.rb:24:in `instrument' 5d13507c | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.4/lib/active_support/notifications.rb:180:in `instrument' 5d13507c | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.4/lib/action_controller/metal/instrumentation.rb:32:in `process_action' 5d13507c | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.4/lib/action_controller/metal/params_wrapper.rb:245:in `process_action' 5d13507c | /opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.4/lib/active_record/railties/controller_runtime.rb:27:in `process_action' 5d13507c | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.1.40/app/controllers/katello/concerns/api/api_controller.rb:40:in `process_action' 5d13507c | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.4/lib/abstract_controller/base.rb:136:in `process' 5d13507c | /opt/theforeman/tfm/root/usr/share/gems/gems/actionview-6.0.3.4/lib/action_view/rendering.rb:39:in `process' 5d13507c | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.4/lib/action_controller/metal.rb:190:in `dispatch' 5d13507c | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.4/lib/action_controller/metal.rb:254:in `dispatch' 5d13507c | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.4/lib/action_dispatch/routing/route_set.rb:50:in `dispatch' 5d13507c | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.4/lib/action_dispatch/routing/route_set.rb:33:in `serve' 5d13507c | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.4/lib/action_dispatch/routing/mapper.rb:18:in `block in <class:Constraints>' 5d13507c | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.4/lib/action_dispatch/routing/mapper.rb:48:in `serve' 5d13507c | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.4/lib/action_dispatch/journey/router.rb:49:in `block in serve' 5d13507c | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.4/lib/action_dispatch/journey/router.rb:32:in `each' 5d13507c | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.4/lib/action_dispatch/journey/router.rb:32:in `serve' 5d13507c | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.4/lib/action_dispatch/routing/route_set.rb:834:in `call' 5d13507c | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.1.40/lib/katello/middleware/event_daemon.rb:10:in `call' 5d13507c | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.4/lib/action_dispatch/middleware/static.rb:126:in `call' 5d13507c | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.4/lib/action_dispatch/middleware/static.rb:126:in `call' 5d13507c | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.4/lib/action_dispatch/middleware/static.rb:126:in `call' 5d13507c | /opt/theforeman/tfm/root/usr/share/gems/gems/apipie-dsl-2.3.0/lib/apipie_dsl/static_dispatcher.rb:67:in `call' 5d13507c | /opt/theforeman/tfm/root/usr/share/gems/gems/apipie-rails-0.5.17/lib/apipie/static_dispatcher.rb:66:in `call' 5d13507c | /opt/theforeman/tfm/root/usr/share/gems/gems/apipie-rails-0.5.17/lib/apipie/extractor/recorder.rb:137:in `call' 5d13507c | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.4/lib/action_dispatch/middleware/static.rb:126:in `call' 5d13507c | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.4/lib/action_dispatch/middleware/static.rb:126:in `call' 5d13507c | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.4/lib/action_dispatch/middleware/static.rb:126:in `call' 5d13507c | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.4/lib/action_dispatch/middleware/static.rb:126:in `call' 5d13507c | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.4/lib/action_dispatch/middleware/static.rb:126:in `call' 5d13507c | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.4/lib/action_dispatch/middleware/static.rb:126:in `call' 5d13507c | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.4/lib/action_dispatch/middleware/static.rb:126:in `call' 5d13507c | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.4/lib/action_dispatch/middleware/static.rb:126:in `call' 5d13507c | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.4/lib/action_dispatch/middleware/static.rb:126:in `call' 5d13507c | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.4/lib/action_dispatch/middleware/static.rb:126:in `call' 5d13507c | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.4/lib/action_dispatch/middleware/static.rb:126:in `call' 5d13507c | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.4/lib/action_dispatch/middleware/static.rb:126:in `call' 5d13507c | /usr/share/foreman/lib/foreman/middleware/telemetry.rb:10:in `call' 5d13507c | /opt/theforeman/tfm/root/usr/share/gems/gems/apipie-rails-0.5.17/lib/apipie/middleware/checksum_in_headers.rb:27:in `call' 5d13507c | /usr/share/foreman/lib/foreman/middleware/catch_json_parse_errors.rb:9:in `call' 5d13507c | /opt/theforeman/tfm/root/usr/share/gems/gems/rack-2.2.3/lib/rack/tempfile_reaper.rb:15:in `call' 5d13507c | /opt/theforeman/tfm/root/usr/share/gems/gems/rack-2.2.3/lib/rack/etag.rb:27:in `call' 5d13507c | /opt/theforeman/tfm/root/usr/share/gems/gems/rack-2.2.3/lib/rack/conditional_get.rb:27:in `call' 5d13507c | /opt/theforeman/tfm/root/usr/share/gems/gems/rack-2.2.3/lib/rack/head.rb:12:in `call' 5d13507c | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.4/lib/action_dispatch/http/content_security_policy.rb:18:in `call' 5d13507c | /usr/share/foreman/lib/foreman/middleware/logging_context_session.rb:22:in `call' 5d13507c | /opt/theforeman/tfm/root/usr/share/gems/gems/rack-2.2.3/lib/rack/session/abstract/id.rb:266:in `context' 5d13507c | /opt/theforeman/tfm/root/usr/share/gems/gems/rack-2.2.3/lib/rack/session/abstract/id.rb:260:in `call' 5d13507c | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.4/lib/action_dispatch/middleware/cookies.rb:648:in `call' 5d13507c | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.4/lib/action_dispatch/middleware/callbacks.rb:27:in `block in call' 5d13507c | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.4/lib/active_support/callbacks.rb:101:in `run_callbacks' 5d13507c | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.4/lib/action_dispatch/middleware/callbacks.rb:26:in `call' 5d13507c | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.4/lib/action_dispatch/middleware/actionable_exceptions.rb:18:in `call' 5d13507c | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.4/lib/action_dispatch/middleware/debug_exceptions.rb:32:in `call' 5d13507c | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.4/lib/action_dispatch/middleware/show_exceptions.rb:33:in `call' 5d13507c | /opt/theforeman/tfm/root/usr/share/gems/gems/railties-6.0.3.4/lib/rails/rack/logger.rb:37:in `call_app' 5d13507c | /opt/theforeman/tfm/root/usr/share/gems/gems/railties-6.0.3.4/lib/rails/rack/logger.rb:28:in `call' 5d13507c | /opt/theforeman/tfm/root/usr/share/gems/gems/sprockets-rails-3.2.1/lib/sprockets/rails/quiet_assets.rb:13:in `call' 5d13507c | /usr/share/foreman/lib/foreman/middleware/logging_context_request.rb:11:in `call' 5d13507c | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.4/lib/action_dispatch/middleware/remote_ip.rb:81:in `call' 5d13507c | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.4/lib/action_dispatch/middleware/request_id.rb:27:in `call' 5d13507c | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.1.40/lib/katello/prevent_json_parsing.rb:12:in `call' 5d13507c | /opt/theforeman/tfm/root/usr/share/gems/gems/rack-2.2.3/lib/rack/method_override.rb:24:in `call' 5d13507c | /opt/theforeman/tfm/root/usr/share/gems/gems/rack-2.2.3/lib/rack/runtime.rb:22:in `call' 5d13507c | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.4/lib/active_support/cache/strategy/local_cache_middleware.rb:29:in `call' 5d13507c | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.4/lib/action_dispatch/middleware/executor.rb:14:in `call' 5d13507c | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.4/lib/action_dispatch/middleware/static.rb:126:in `call' 5d13507c | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.4/lib/action_dispatch/middleware/static.rb:126:in `call' 5d13507c | /opt/theforeman/tfm/root/usr/share/gems/gems/rack-2.2.3/lib/rack/sendfile.rb:110:in `call' 5d13507c | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.4/lib/action_dispatch/middleware/host_authorization.rb:76:in `call' 5d13507c | /opt/theforeman/tfm/root/usr/share/gems/gems/secure_headers-6.3.0/lib/secure_headers/middleware.rb:11:in `call' 5d13507c | /opt/theforeman/tfm/root/usr/share/gems/gems/railties-6.0.3.4/lib/rails/engine.rb:527:in `call' 5d13507c | /opt/theforeman/tfm/root/usr/share/gems/gems/railties-6.0.3.4/lib/rails/railtie.rb:190:in `public_send' 5d13507c | /opt/theforeman/tfm/root/usr/share/gems/gems/railties-6.0.3.4/lib/rails/railtie.rb:190:in `method_missing' 5d13507c | /opt/theforeman/tfm/root/usr/share/gems/gems/rack-2.2.3/lib/rack/urlmap.rb:74:in `block in call' 5d13507c | /opt/theforeman/tfm/root/usr/share/gems/gems/rack-2.2.3/lib/rack/urlmap.rb:58:in `each' 5d13507c | /opt/theforeman/tfm/root/usr/share/gems/gems/rack-2.2.3/lib/rack/urlmap.rb:58:in `call' 5d13507c | /opt/theforeman/tfm/root/usr/share/gems/gems/puma-4.3.6/lib/puma/configuration.rb:228:in `call' 5d13507c | /opt/theforeman/tfm/root/usr/share/gems/gems/puma-4.3.6/lib/puma/server.rb:713:in `handle_request' 5d13507c | /opt/theforeman/tfm/root/usr/share/gems/gems/puma-4.3.6/lib/puma/server.rb:472:in `process_client' 5d13507c | /opt/theforeman/tfm/root/usr/share/gems/gems/puma-4.3.6/lib/puma/server.rb:328:in `block in run' 5d13507c | /opt/theforeman/tfm/root/usr/share/gems/gems/puma-4.3.6/lib/puma/thread_pool.rb:134:in `block in spawn_thread' 5d13507c | /opt/theforeman/tfm/root/usr/share/gems/gems/logging-2.3.0/lib/logging/diagnostic_context.rb:474:in `block in create_with_logging_context' 2022-03-03T06:05:09 [I|app|5d13507c] Completed 500 Internal Server Error in 1129ms (Views: 0.2ms | ActiveRecord: 38.9ms | Allocations: 87301) Expected results: Completed 200 OK Additional info: