Note: This bug is displayed in read-only format because
the product is no longer active in Red Hat Bugzilla.
Red Hat Satellite engineering is moving the tracking of its product development work on Satellite to Red Hat Jira (issues.redhat.com). If you're a Red Hat customer, please continue to file support cases via the Red Hat customer portal. If you're not, please head to the "Satellite project" in Red Hat Jira and file new tickets here. Individual Bugzilla bugs will be migrated starting at the end of May. If you cannot log in to RH Jira, please consult article #7032570. That failing, please send an e-mail to the RH Jira admins at rh-issues@redhat.com to troubleshoot your issue as a user management inquiry. The email creates a ServiceNow ticket with Red Hat. Individual Bugzilla bugs that are migrated will be moved to status "CLOSED", resolution "MIGRATED", and set with "MigratedToJIRA" in "Keywords". The link to the successor Jira issue will be found under "Links", have a little "two-footprint" icon next to it, and direct you to the "Satellite project" in Red Hat Jira (issue links are of type "https://issues.redhat.com/browse/SAT-XXXX", where "X" is a digit). This same link will be available in a blue banner at the top of the page informing you that that bug has been migrated.
puma sometimes throws Errno::ENOENT: No such file or directory @ rb_sysopen - /usr/share/foreman/tmp/cache/0E3/781/.permissions_check.220838320.2072127.637105
This issue can happen if we use the same non-admin user to make API calls concurrently and continuously for more than 2 minutes because the cache life is 2 minutes. The race condition occurs when 1 or more API request trying to read the deleted cache and raise error.
How to reproduce the issue:
1. Open a terminal (Either in satellite or in other host) and then make multiple concurrent requests to query the smart proxies.
~~~
irb
require 'rest_client'
250.times {10.times { Thread.new { begin; RestClient::Resource.new("https://satellite.example.com/api/smart_proxies", user: "username", password: "pass", timeout: 3600, open_timeout: 3600, verify_ssl: OpenSSL::SSL::VERIFY_NONE).get; rescue StandardError => e; p e.message; end }}; sleep 2}
~~~
2. In Satellite run, tail the production.log.
~~~
tail -f /var/log/foreman/production.log | grep -i "No such file"
~~~
3. After some time you should start seeing the errors like below:
~~~
2023-04-26T18:02:31 [I|app|42c1284e] Backtrace for 'Action failed' error (Errno::ENOENT): No such file or directory @ rb_sysopen - /usr/share/foreman/tmp/cache/10D/6F1/.permissions_check.234040.2314370.945374
2023-04-26T18:02:31 [I|app|10a508c4] Backtrace for 'Action failed' error (Errno::ENOENT): No such file or directory @ rb_sysopen - /usr/share/foreman/tmp/cache/10D/6F1/.permissions_check.234060.2314364.423521
2023-04-26T18:02:31 [I|app|a51a7938] Backtrace for 'Action failed' error (Errno::ENOENT): No such file or directory @ rb_sysopen - /usr/share/foreman/tmp/cache/10D/6F1/.permissions_check.233940.2314370.586758
~~~
Additional info:
To make the error easier to reproduce, you can reduce the "delay" to something like "10.seconds".
/usr/share/foreman/app/models/user.rb
~~~
def taxonomy_and_child_ids(taxonomies)
delay = Rails.env.test? ? 0 : 2.minutes <======================
Rails.cache.fetch("user/#{id}/taxonomy_and_child_ids/#{taxonomies}", expires_in: delay) do
top_level = send(taxonomies) + taxonomies.to_s.classify.constantize.unscoped.select { |tax| tax.ignore?('user') }
top_level.each_with_object([]) do |taxonomy, ids|
ids.concat taxonomy.subtree_ids
end.uniq
end
end
~~~
We believe that at a certain scale, the file cache cannot handle the traffic. To alleviate this, we need to introduce a Redis based caching option for users to enable. Redis is already a part of the deployment architecture used by both Dynflow and Pulp for content caching.
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.
For information on the advisory (Important: Satellite 6.15.0 release), and where to find the updated
files, follow the link below.
If the solution does not work for you, open a new bug report.
https://access.redhat.com/errata/RHSA-2024:2010
Description of problem: During normal operations puma sometimes throw Errno::ENOENT: No such file or directory @ rb_sysopen - /usr/share/foreman/tmp/cache/0E3/781/.permissions_check.220838320.2072127.637105 ... 2022-03-03T06:05:09 [I|app|5d13507c] Completed 500 Internal Server Error in 1129ms (Views: 0.2ms | ActiveRecord: 38.9ms | Allocations: 87301) Version-Release number of selected component (if applicable): satellite-6.9.6.1-1.el7sat.noarch How reproducible: Only on customer side Actual results: 2022-03-03T06:05:08 [I|app|5d13507c] Started GET "/api/v2/hosts/548/errata?page=1&per_page=1" for 10.196.65.30 at 2022-03-03 06:05:08 +0000 2022-03-03T06:05:08 [I|app|5d13507c] Processing by Katello::Api::V2::HostErrataController#index as HTML 2022-03-03T06:05:08 [I|app|5d13507c] Parameters: {"page"=>"1", "per_page"=>"1", "api_version"=>"v2", "host_id"=>"548"} 2022-03-03T06:05:09 [W|app|5d13507c] Scoped order is ignored, it's forced to be batch order. 2022-03-03T06:05:09 [I|app|5d13507c] Authorized user orchit_prod(orchit_prod) 2022-03-03T06:05:09 [E|app|5d13507c] Errno::ENOENT: No such file or directory @ rb_sysopen - /usr/share/foreman/tmp/cache/0E3/781/.permissions_check.220838320.2072127.637105 5d13507c | /opt/rh/rh-ruby25/root/usr/share/ruby/fileutils.rb:1057:in `initialize' 5d13507c | /opt/rh/rh-ruby25/root/usr/share/ruby/fileutils.rb:1057:in `open' 5d13507c | /opt/rh/rh-ruby25/root/usr/share/ruby/fileutils.rb:1057:in `rescue in block in touch' 5d13507c | /opt/rh/rh-ruby25/root/usr/share/ruby/fileutils.rb:1053:in `block in touch' 5d13507c | /opt/rh/rh-ruby25/root/usr/share/ruby/fileutils.rb:1051:in `each' 5d13507c | /opt/rh/rh-ruby25/root/usr/share/ruby/fileutils.rb:1051:in `touch' 5d13507c | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.4/lib/active_support/core_ext/file/atomic.rb:65:in `probe_stat_in' 5d13507c | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.4/lib/active_support/core_ext/file/atomic.rb:35:in `block in atomic_write' 5d13507c | /opt/rh/rh-ruby25/root/usr/share/ruby/tempfile.rb:295:in `open' 5d13507c | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.4/lib/active_support/core_ext/file/atomic.rb:24:in `atomic_write' 5d13507c | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.4/lib/active_support/cache/file_store.rb:87:in `write_entry' 5d13507c | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.4/lib/active_support/cache/strategy/local_cache.rb:150:in `write_entry' 5d13507c | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.4/lib/active_support/cache.rb:465:in `block in write' 5d13507c | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.4/lib/active_support/cache.rb:686:in `block in instrument' 5d13507c | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.4/lib/active_support/notifications.rb:182:in `instrument' 5d13507c | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.4/lib/active_support/cache.rb:686:in `instrument' 5d13507c | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.4/lib/active_support/cache.rb:463:in `write' 5d13507c | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.4/lib/active_support/cache.rb:720:in `save_block_result_to_cache' 5d13507c | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.4/lib/active_support/cache.rb:332:in `fetch' 5d13507c | /usr/share/foreman/app/models/user.rb:522:in `taxonomy_and_child_ids' 5d13507c | /usr/share/foreman/app/models/user.rb:536:in `location_and_child_ids' 5d13507c | /usr/share/foreman/app/models/taxonomies/location.rb:22:in `block in <class:Location>' 5d13507c | /opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.4/lib/active_record/relation.rb:412:in `instance_exec' 5d13507c | /opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.4/lib/active_record/relation.rb:412:in `block in _exec_scope' 5d13507c | /opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.4/lib/active_record/relation.rb:784:in `_scoping' 5d13507c | /opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.4/lib/active_record/relation.rb:412:in `_exec_scope' 5d13507c | /opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.4/lib/active_record/scoping/named.rb:192:in `block in scope' 5d13507c | /usr/share/foreman/app/models/user.rb:497:in `my_locations' 5d13507c | /usr/share/foreman/app/controllers/concerns/application_shared.rb:28:in `block in set_taxonomy' 5d13507c | /usr/share/foreman/app/controllers/concerns/application_shared.rb:27:in `each' 5d13507c | /usr/share/foreman/app/controllers/concerns/application_shared.rb:27:in `set_taxonomy' 5d13507c | /usr/share/foreman/app/controllers/concerns/application_shared.rb:18:in `set_current_user' 5d13507c | /usr/share/foreman/app/controllers/concerns/foreman/controller/authentication.rb:17:in `authenticate' 5d13507c | /usr/share/foreman/app/controllers/api/base_controller.rb:192:in `authorize' 5d13507c | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.4/lib/active_support/callbacks.rb:428:in `block in make_lambda' 5d13507c | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.4/lib/active_support/callbacks.rb:200:in `block (2 levels) in halting' 5d13507c | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.4/lib/abstract_controller/callbacks.rb:34:in `block (2 levels) in <module:Callbacks>' 5d13507c | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.4/lib/active_support/callbacks.rb:201:in `block in halting' 5d13507c | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.4/lib/active_support/callbacks.rb:513:in `block in invoke_before' 5d13507c | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.4/lib/active_support/callbacks.rb:513:in `each' 5d13507c | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.4/lib/active_support/callbacks.rb:513:in `invoke_before' 5d13507c | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.4/lib/active_support/callbacks.rb:110:in `block in run_callbacks' 5d13507c | /usr/share/foreman/app/models/concerns/foreman/thread_session.rb:32:in `clear_thread' 5d13507c | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.4/lib/active_support/callbacks.rb:121:in `block in run_callbacks' 5d13507c | /usr/share/foreman/app/controllers/concerns/foreman/controller/topbar_sweeper.rb:12:in `set_topbar_sweeper_controller' 5d13507c | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.4/lib/active_support/callbacks.rb:121:in `block in run_callbacks' 5d13507c | /opt/theforeman/tfm/root/usr/share/gems/gems/audited-4.9.0/lib/audited/sweeper.rb:14:in `around' 5d13507c | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.4/lib/active_support/callbacks.rb:121:in `block in run_callbacks' 5d13507c | /opt/theforeman/tfm/root/usr/share/gems/gems/audited-4.9.0/lib/audited/sweeper.rb:14:in `around' 5d13507c | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.4/lib/active_support/callbacks.rb:121:in `block in run_callbacks' 5d13507c | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.4/lib/active_support/callbacks.rb:139:in `run_callbacks' 5d13507c | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.4/lib/abstract_controller/callbacks.rb:41:in `process_action' 5d13507c | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.4/lib/action_controller/metal/rescue.rb:22:in `process_action' 5d13507c | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.4/lib/action_controller/metal/instrumentation.rb:33:in `block in process_action' 5d13507c | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.4/lib/active_support/notifications.rb:180:in `block in instrument' 5d13507c | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.4/lib/active_support/notifications/instrumenter.rb:24:in `instrument' 5d13507c | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.4/lib/active_support/notifications.rb:180:in `instrument' 5d13507c | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.4/lib/action_controller/metal/instrumentation.rb:32:in `process_action' 5d13507c | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.4/lib/action_controller/metal/params_wrapper.rb:245:in `process_action' 5d13507c | /opt/theforeman/tfm/root/usr/share/gems/gems/activerecord-6.0.3.4/lib/active_record/railties/controller_runtime.rb:27:in `process_action' 5d13507c | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.1.40/app/controllers/katello/concerns/api/api_controller.rb:40:in `process_action' 5d13507c | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.4/lib/abstract_controller/base.rb:136:in `process' 5d13507c | /opt/theforeman/tfm/root/usr/share/gems/gems/actionview-6.0.3.4/lib/action_view/rendering.rb:39:in `process' 5d13507c | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.4/lib/action_controller/metal.rb:190:in `dispatch' 5d13507c | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.4/lib/action_controller/metal.rb:254:in `dispatch' 5d13507c | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.4/lib/action_dispatch/routing/route_set.rb:50:in `dispatch' 5d13507c | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.4/lib/action_dispatch/routing/route_set.rb:33:in `serve' 5d13507c | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.4/lib/action_dispatch/routing/mapper.rb:18:in `block in <class:Constraints>' 5d13507c | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.4/lib/action_dispatch/routing/mapper.rb:48:in `serve' 5d13507c | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.4/lib/action_dispatch/journey/router.rb:49:in `block in serve' 5d13507c | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.4/lib/action_dispatch/journey/router.rb:32:in `each' 5d13507c | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.4/lib/action_dispatch/journey/router.rb:32:in `serve' 5d13507c | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.4/lib/action_dispatch/routing/route_set.rb:834:in `call' 5d13507c | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.1.40/lib/katello/middleware/event_daemon.rb:10:in `call' 5d13507c | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.4/lib/action_dispatch/middleware/static.rb:126:in `call' 5d13507c | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.4/lib/action_dispatch/middleware/static.rb:126:in `call' 5d13507c | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.4/lib/action_dispatch/middleware/static.rb:126:in `call' 5d13507c | /opt/theforeman/tfm/root/usr/share/gems/gems/apipie-dsl-2.3.0/lib/apipie_dsl/static_dispatcher.rb:67:in `call' 5d13507c | /opt/theforeman/tfm/root/usr/share/gems/gems/apipie-rails-0.5.17/lib/apipie/static_dispatcher.rb:66:in `call' 5d13507c | /opt/theforeman/tfm/root/usr/share/gems/gems/apipie-rails-0.5.17/lib/apipie/extractor/recorder.rb:137:in `call' 5d13507c | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.4/lib/action_dispatch/middleware/static.rb:126:in `call' 5d13507c | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.4/lib/action_dispatch/middleware/static.rb:126:in `call' 5d13507c | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.4/lib/action_dispatch/middleware/static.rb:126:in `call' 5d13507c | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.4/lib/action_dispatch/middleware/static.rb:126:in `call' 5d13507c | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.4/lib/action_dispatch/middleware/static.rb:126:in `call' 5d13507c | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.4/lib/action_dispatch/middleware/static.rb:126:in `call' 5d13507c | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.4/lib/action_dispatch/middleware/static.rb:126:in `call' 5d13507c | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.4/lib/action_dispatch/middleware/static.rb:126:in `call' 5d13507c | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.4/lib/action_dispatch/middleware/static.rb:126:in `call' 5d13507c | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.4/lib/action_dispatch/middleware/static.rb:126:in `call' 5d13507c | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.4/lib/action_dispatch/middleware/static.rb:126:in `call' 5d13507c | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.4/lib/action_dispatch/middleware/static.rb:126:in `call' 5d13507c | /usr/share/foreman/lib/foreman/middleware/telemetry.rb:10:in `call' 5d13507c | /opt/theforeman/tfm/root/usr/share/gems/gems/apipie-rails-0.5.17/lib/apipie/middleware/checksum_in_headers.rb:27:in `call' 5d13507c | /usr/share/foreman/lib/foreman/middleware/catch_json_parse_errors.rb:9:in `call' 5d13507c | /opt/theforeman/tfm/root/usr/share/gems/gems/rack-2.2.3/lib/rack/tempfile_reaper.rb:15:in `call' 5d13507c | /opt/theforeman/tfm/root/usr/share/gems/gems/rack-2.2.3/lib/rack/etag.rb:27:in `call' 5d13507c | /opt/theforeman/tfm/root/usr/share/gems/gems/rack-2.2.3/lib/rack/conditional_get.rb:27:in `call' 5d13507c | /opt/theforeman/tfm/root/usr/share/gems/gems/rack-2.2.3/lib/rack/head.rb:12:in `call' 5d13507c | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.4/lib/action_dispatch/http/content_security_policy.rb:18:in `call' 5d13507c | /usr/share/foreman/lib/foreman/middleware/logging_context_session.rb:22:in `call' 5d13507c | /opt/theforeman/tfm/root/usr/share/gems/gems/rack-2.2.3/lib/rack/session/abstract/id.rb:266:in `context' 5d13507c | /opt/theforeman/tfm/root/usr/share/gems/gems/rack-2.2.3/lib/rack/session/abstract/id.rb:260:in `call' 5d13507c | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.4/lib/action_dispatch/middleware/cookies.rb:648:in `call' 5d13507c | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.4/lib/action_dispatch/middleware/callbacks.rb:27:in `block in call' 5d13507c | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.4/lib/active_support/callbacks.rb:101:in `run_callbacks' 5d13507c | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.4/lib/action_dispatch/middleware/callbacks.rb:26:in `call' 5d13507c | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.4/lib/action_dispatch/middleware/actionable_exceptions.rb:18:in `call' 5d13507c | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.4/lib/action_dispatch/middleware/debug_exceptions.rb:32:in `call' 5d13507c | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.4/lib/action_dispatch/middleware/show_exceptions.rb:33:in `call' 5d13507c | /opt/theforeman/tfm/root/usr/share/gems/gems/railties-6.0.3.4/lib/rails/rack/logger.rb:37:in `call_app' 5d13507c | /opt/theforeman/tfm/root/usr/share/gems/gems/railties-6.0.3.4/lib/rails/rack/logger.rb:28:in `call' 5d13507c | /opt/theforeman/tfm/root/usr/share/gems/gems/sprockets-rails-3.2.1/lib/sprockets/rails/quiet_assets.rb:13:in `call' 5d13507c | /usr/share/foreman/lib/foreman/middleware/logging_context_request.rb:11:in `call' 5d13507c | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.4/lib/action_dispatch/middleware/remote_ip.rb:81:in `call' 5d13507c | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.4/lib/action_dispatch/middleware/request_id.rb:27:in `call' 5d13507c | /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.18.1.40/lib/katello/prevent_json_parsing.rb:12:in `call' 5d13507c | /opt/theforeman/tfm/root/usr/share/gems/gems/rack-2.2.3/lib/rack/method_override.rb:24:in `call' 5d13507c | /opt/theforeman/tfm/root/usr/share/gems/gems/rack-2.2.3/lib/rack/runtime.rb:22:in `call' 5d13507c | /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.4/lib/active_support/cache/strategy/local_cache_middleware.rb:29:in `call' 5d13507c | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.4/lib/action_dispatch/middleware/executor.rb:14:in `call' 5d13507c | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.4/lib/action_dispatch/middleware/static.rb:126:in `call' 5d13507c | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.4/lib/action_dispatch/middleware/static.rb:126:in `call' 5d13507c | /opt/theforeman/tfm/root/usr/share/gems/gems/rack-2.2.3/lib/rack/sendfile.rb:110:in `call' 5d13507c | /opt/theforeman/tfm/root/usr/share/gems/gems/actionpack-6.0.3.4/lib/action_dispatch/middleware/host_authorization.rb:76:in `call' 5d13507c | /opt/theforeman/tfm/root/usr/share/gems/gems/secure_headers-6.3.0/lib/secure_headers/middleware.rb:11:in `call' 5d13507c | /opt/theforeman/tfm/root/usr/share/gems/gems/railties-6.0.3.4/lib/rails/engine.rb:527:in `call' 5d13507c | /opt/theforeman/tfm/root/usr/share/gems/gems/railties-6.0.3.4/lib/rails/railtie.rb:190:in `public_send' 5d13507c | /opt/theforeman/tfm/root/usr/share/gems/gems/railties-6.0.3.4/lib/rails/railtie.rb:190:in `method_missing' 5d13507c | /opt/theforeman/tfm/root/usr/share/gems/gems/rack-2.2.3/lib/rack/urlmap.rb:74:in `block in call' 5d13507c | /opt/theforeman/tfm/root/usr/share/gems/gems/rack-2.2.3/lib/rack/urlmap.rb:58:in `each' 5d13507c | /opt/theforeman/tfm/root/usr/share/gems/gems/rack-2.2.3/lib/rack/urlmap.rb:58:in `call' 5d13507c | /opt/theforeman/tfm/root/usr/share/gems/gems/puma-4.3.6/lib/puma/configuration.rb:228:in `call' 5d13507c | /opt/theforeman/tfm/root/usr/share/gems/gems/puma-4.3.6/lib/puma/server.rb:713:in `handle_request' 5d13507c | /opt/theforeman/tfm/root/usr/share/gems/gems/puma-4.3.6/lib/puma/server.rb:472:in `process_client' 5d13507c | /opt/theforeman/tfm/root/usr/share/gems/gems/puma-4.3.6/lib/puma/server.rb:328:in `block in run' 5d13507c | /opt/theforeman/tfm/root/usr/share/gems/gems/puma-4.3.6/lib/puma/thread_pool.rb:134:in `block in spawn_thread' 5d13507c | /opt/theforeman/tfm/root/usr/share/gems/gems/logging-2.3.0/lib/logging/diagnostic_context.rb:474:in `block in create_with_logging_context' 2022-03-03T06:05:09 [I|app|5d13507c] Completed 500 Internal Server Error in 1129ms (Views: 0.2ms | ActiveRecord: 38.9ms | Allocations: 87301) Expected results: Completed 200 OK Additional info: