Bug 1549088
| Summary: | Various Action:: Tasks types stop with warning: "no such file or directory" for file in foreman cache hierarchy | ||
|---|---|---|---|
| Product: | Red Hat Satellite | Reporter: | Ben <ben.argyle> |
| Component: | Hosts | Assignee: | Tomer Brisker <tbrisker> |
| Status: | CLOSED ERRATA | QA Contact: | Jan Hutař <jhutar> |
| Severity: | medium | Docs Contact: | |
| Priority: | medium | ||
| Version: | 6.2.14 | CC: | akapse, apatel, arahaman, arusso, bkearney, chsimon, dgross, egolov, fadamo, hyu, inecas, jalviso, kgaikwad, kuemmer, mmccune, smajumda, sthirugn, vijsingh, vwariyal |
| Target Milestone: | 6.5.0 | Keywords: | PrioBumpField, Triaged |
| Target Release: | Unused | ||
| Hardware: | Unspecified | ||
| OS: | Unspecified | ||
| Whiteboard: | |||
| Fixed In Version: | foreman-1.20.1.25-1 | Doc Type: | If docs needed, set a value |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2019-05-14 12:37:00 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
|
Description
Ben
2018-02-26 11:28:51 UTC
I have a similar problem importing the manifest. Actions::Pulp::Repository::RefreshDistributor No such file or directory @ rb_sysopen - /usr/share/foreman/tmp/cache/462/C60/.permissions_check.84342060.4987.10157 No such file or directory @ rb_sysopen - /usr/share/foreman/tmp/cache/462/C60/.permissions_check.51684160.4987.279382 /opt/rh/rh-ruby24/root/usr/share/ruby/fileutils.rb:1055:in `initialize' /opt/rh/rh-ruby24/root/usr/share/ruby/fileutils.rb:1055:in `open' /opt/rh/rh-ruby24/root/usr/share/ruby/fileutils.rb:1055:in `rescue in block in touch' /opt/rh/rh-ruby24/root/usr/share/ruby/fileutils.rb:1051:in `block in touch' /opt/rh/rh-ruby24/root/usr/share/ruby/fileutils.rb:1049:in `each' /opt/rh/rh-ruby24/root/usr/share/ruby/fileutils.rb:1049:in `touch' /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/activesupport-5.1.4/lib/active_support/core_ext/file/atomic.rb:63:in `probe_stat_in' /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/activesupport-5.1.4/lib/active_support/core_ext/file/atomic.rb:33:in `block in atomic_write' /opt/rh/rh-ruby24/root/usr/share/ruby/tempfile.rb:295:in `open' /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/activesupport-5.1.4/lib/active_support/core_ext/file/atomic.rb:22:in `atomic_write' /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/activesupport-5.1.4/lib/active_support/cache/file_store.rb:83:in `write_entry' /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/activesupport-5.1.4/lib/active_support/cache/strategy/local_cache.rb:119:in `write_entry' /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/activesupport-5.1.4/lib/active_support/cache.rb:400:in `block in write' /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/activesupport-5.1.4/lib/active_support/cache.rb:555:in `block in instrument' /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/activesupport-5.1.4/lib/active_support/notifications.rb:168:in `instrument' /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/activesupport-5.1.4/lib/active_support/cache.rb:555:in `instrument' /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/activesupport-5.1.4/lib/active_support/cache.rb:398:in `write' /usr/share/foreman/app/models/setting.rb:97:in `[]' /usr/share/foreman/lib/foreman/http_proxy.rb:4:in `http_proxy' /usr/share/foreman/lib/foreman/http_proxy.rb:12:in `proxy_http_request?' /usr/share/foreman/lib/foreman/http_proxy/net_http_extension.rb:7:in `request' /opt/theforeman/tfm/root/usr/share/gems/gems/rest-client-2.0.1/lib/restclient/request.rb:446:in `net_http_do_request' /opt/theforeman/tfm/root/usr/share/gems/gems/rest-client-2.0.1/lib/restclient/request.rb:721:in `block in transmit' /opt/rh/rh-ruby24/root/usr/share/ruby/net/http.rb:877:in `start' /opt/theforeman/tfm/root/usr/share/gems/gems/rest-client-2.0.1/lib/restclient/request.rb:715:in `transmit' /opt/theforeman/tfm/root/usr/share/gems/gems/rest-client-2.0.1/lib/restclient/request.rb:145:in `execute' /opt/theforeman/tfm/root/usr/share/gems/gems/rest-client-2.0.1/lib/restclient/request.rb:52:in `execute' /opt/theforeman/tfm/root/usr/share/gems/gems/rest-client-2.0.1/lib/restclient/resource.rb:76:in `put' /opt/theforeman/tfm/root/usr/share/gems/gems/runcible-2.6.0/lib/runcible/base.rb:80:in `get_response' /opt/theforeman/tfm/root/usr/share/gems/gems/runcible-2.6.0/lib/runcible/base.rb:68:in `call' /opt/theforeman/tfm/root/usr/share/gems/gems/runcible-2.6.0/lib/runcible/resources/repository.rb:178:in `update_distributor' /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.6.0/app/lib/actions/pulp/repository/refresh_distributor.rb:14:in `run' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.37/lib/dynflow/action.rb:531:in `block (3 levels) in execute_run' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.37/lib/dynflow/middleware/stack.rb:26:in `pass' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.37/lib/dynflow/middleware.rb:18:in `pass' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.37/lib/dynflow/middleware.rb:31:in `run' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.37/lib/dynflow/middleware/stack.rb:22:in `call' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.37/lib/dynflow/middleware/stack.rb:26:in `pass' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.37/lib/dynflow/middleware.rb:18:in `pass' /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.6.0/app/lib/actions/middleware/remote_action.rb:16:in `block in run' /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.6.0/app/lib/actions/middleware/remote_action.rb:40:in `block in as_remote_user' /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.6.0/app/models/katello/concerns/user_extensions.rb:21:in `cp_config' /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.6.0/app/lib/actions/middleware/remote_action.rb:27:in `as_cp_user' /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.6.0/app/lib/actions/middleware/remote_action.rb:39:in `as_remote_user' /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.6.0/app/lib/actions/middleware/remote_action.rb:16:in `run' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.37/lib/dynflow/middleware/stack.rb:22:in `call' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.37/lib/dynflow/middleware/stack.rb:26:in `pass' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.37/lib/dynflow/middleware.rb:18:in `pass' /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-0.11.2/app/lib/actions/middleware/rails_executor_wrap.rb:14:in `block in run' /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/activesupport-5.1.4/lib/active_support/execution_wrapper.rb:85:in `wrap' /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-0.11.2/app/lib/actions/middleware/rails_executor_wrap.rb:13:in `run' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.37/lib/dynflow/middleware/stack.rb:22:in `call' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.37/lib/dynflow/middleware/stack.rb:26:in `pass' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.37/lib/dynflow/middleware.rb:18:in `pass' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.37/lib/dynflow/action/progress.rb:30:in `with_progress_calculation' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.37/lib/dynflow/action/progress.rb:16:in `run' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.37/lib/dynflow/middleware/stack.rb:22:in `call' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.37/lib/dynflow/middleware/stack.rb:26:in `pass' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.37/lib/dynflow/middleware.rb:18:in `pass' /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.6.0/app/lib/actions/middleware/keep_locale.rb:11:in `block in run' /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.6.0/app/lib/actions/middleware/keep_locale.rb:22:in `with_locale' /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.6.0/app/lib/actions/middleware/keep_locale.rb:11:in `run' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.37/lib/dynflow/middleware/stack.rb:22:in `call' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.37/lib/dynflow/middleware/stack.rb:26:in `pass' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.37/lib/dynflow/middleware.rb:18:in `pass' /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-0.11.2/app/lib/actions/middleware/keep_current_user.rb:28:in `block in run' /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-0.11.2/app/lib/actions/middleware/keep_current_user.rb:51:in `restore_curent_user' /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-0.11.2/app/lib/actions/middleware/keep_current_user.rb:28:in `run' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.37/lib/dynflow/middleware/stack.rb:22:in `call' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.37/lib/dynflow/middleware/stack.rb:26:in `pass' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.37/lib/dynflow/middleware.rb:18:in `pass' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.37/lib/dynflow/middleware.rb:31:in `run' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.37/lib/dynflow/middleware/stack.rb:22:in `call' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.37/lib/dynflow/middleware/world.rb:30:in `execute' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.37/lib/dynflow/action.rb:530:in `block (2 levels) in execute_run' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.37/lib/dynflow/action.rb:529:in `catch' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.37/lib/dynflow/action.rb:529:in `block in execute_run' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.37/lib/dynflow/action.rb:444:in `block in with_error_handling' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.37/lib/dynflow/action.rb:444:in `catch' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.37/lib/dynflow/action.rb:444:in `with_error_handling' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.37/lib/dynflow/action.rb:524:in `execute_run' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.37/lib/dynflow/action.rb:277:in `execute' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.37/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:9:in `block (2 levels) in execute' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.37/lib/dynflow/execution_plan/steps/abstract.rb:155:in `with_meta_calculation' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.37/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:8:in `block in execute' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.37/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:22:in `open_action' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.37/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:7:in `execute' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.37/lib/dynflow/director.rb:42:in `execute' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.37/lib/dynflow/executors/parallel/worker.rb:12:in `block in on_message' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.37/lib/dynflow/executors.rb:11:in `run_user_code' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.37/lib/dynflow/executors/parallel/worker.rb:11:in `on_message' /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.4/lib/concurrent/actor/context.rb:46:in `on_envelope' /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.4/lib/concurrent/actor/behaviour/executes_context.rb:7:in `on_envelope' /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.4/lib/concurrent/actor/behaviour/abstract.rb:25:in `pass' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-0.8.37/lib/dynflow/actor.rb:26:in `on_envelope' /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.4/lib/concurrent/actor/behaviour/abstract.rb:25:in `pass' /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.4/lib/concurrent/actor/behaviour/awaits.rb:15:in `on_envelope' /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.4/lib/concurrent/actor/behaviour/abstract.rb:25:in `pass' /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.4/lib/concurrent/actor/behaviour/sets_results.rb:14:in `on_envelope' /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.4/lib/concurrent/actor/behaviour/abstract.rb:25:in `pass' /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.4/lib/concurrent/actor/behaviour/buffer.rb:38:in `process_envelope' /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.4/lib/concurrent/actor/behaviour/buffer.rb:31:in `process_envelopes?' /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.4/lib/concurrent/actor/behaviour/buffer.rb:20:in `on_envelope' /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.4/lib/concurrent/actor/behaviour/abstract.rb:25:in `pass' /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.4/lib/concurrent/actor/behaviour/termination.rb:55:in `on_envelope' /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.4/lib/concurrent/actor/behaviour/abstract.rb:25:in `pass' /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.4/lib/concurrent/actor/behaviour/removes_child.rb:10:in `on_envelope' /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.4/lib/concurrent/actor/behaviour/abstract.rb:25:in `pass' /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.4/lib/concurrent/actor/behaviour/sets_results.rb:14:in `on_envelope' /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.4/lib/concurrent/actor/core.rb:161:in `process_envelope' /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.4/lib/concurrent/actor/core.rb:95:in `block in on_envelope' /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.4/lib/concurrent/actor/core.rb:118:in `block (2 levels) in schedule_execution' /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/concurrent-ruby-1.0.5/lib/concurrent/synchronization/mri_lockable_object.rb:38:in `block in synchronize' /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/concurrent-ruby-1.0.5/lib/concurrent/synchronization/mri_lockable_object.rb:38:in `synchronize' /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/concurrent-ruby-1.0.5/lib/concurrent/synchronization/mri_lockable_object.rb:38:in `synchronize' /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.4/lib/concurrent/actor/core.rb:115:in `block in schedule_execution' /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/concurrent-ruby-1.0.5/lib/concurrent/executor/serialized_execution.rb:18:in `call' /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/concurrent-ruby-1.0.5/lib/concurrent/executor/serialized_execution.rb:96:in `work' /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/concurrent-ruby-1.0.5/lib/concurrent/executor/serialized_execution.rb:77:in `block in call_job' /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/concurrent-ruby-1.0.5/lib/concurrent/executor/ruby_thread_pool_executor.rb:348:in `run_task' /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/concurrent-ruby-1.0.5/lib/concurrent/executor/ruby_thread_pool_executor.rb:337:in `block (3 levels) in create_worker' /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/concurrent-ruby-1.0.5/lib/concurrent/executor/ruby_thread_pool_executor.rb:320:in `loop' /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/concurrent-ruby-1.0.5/lib/concurrent/executor/ruby_thread_pool_executor.rb:320:in `block (2 levels) in create_worker' /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/concurrent-ruby-1.0.5/lib/concurrent/executor/ruby_thread_pool_executor.rb:319:in `catch' /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/concurrent-ruby-1.0.5/lib/concurrent/executor/ruby_thread_pool_executor.rb:319:in `block in create_worker' /opt/theforeman/tfm/root/usr/share/gems/gems/logging-2.2.2/lib/logging/diagnostic_context.rb:474:in `block in create_with_logging_context' I see this issue multiple times a day on Satellite 6.4.0, most commonly with host updates and repo syncs. Here's the latest example from this morning for an 'Actions::Pulp::Repository::Sync' task: Exception: Errno::ENOENT: No such file or directory @ rb_sysopen - /usr/share/foreman/tmp/cache/462/C60/.permissions_check.89693460.16159.160059 Backtrace: /opt/rh/rh-ruby24/root/usr/share/ruby/fileutils.rb:1055:in `initialize' /opt/rh/rh-ruby24/root/usr/share/ruby/fileutils.rb:1055:in `open' /opt/rh/rh-ruby24/root/usr/share/ruby/fileutils.rb:1055:in `rescue in block in touch' /opt/rh/rh-ruby24/root/usr/share/ruby/fileutils.rb:1051:in `block in touch' /opt/rh/rh-ruby24/root/usr/share/ruby/fileutils.rb:1049:in `each' /opt/rh/rh-ruby24/root/usr/share/ruby/fileutils.rb:1049:in `touch' /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/activesupport-5.1.6/lib/active_support/core_ext/file/atomic.rb:63:in `probe_stat_in' /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/activesupport-5.1.6/lib/active_support/core_ext/file/atomic.rb:33:in `block in atomic_write' /opt/rh/rh-ruby24/root/usr/share/ruby/tempfile.rb:295:in `open' /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/activesupport-5.1.6/lib/active_support/core_ext/file/atomic.rb:22:in `atomic_write' /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/activesupport-5.1.6/lib/active_support/cache/file_store.rb:83:in `write_entry' /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/activesupport-5.1.6/lib/active_support/cache/strategy/local_cache.rb:119:in `write_entry' /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/activesupport-5.1.6/lib/active_support/cache.rb:400:in `block in write' /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/activesupport-5.1.6/lib/active_support/cache.rb:555:in `block in instrument' /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/activesupport-5.1.6/lib/active_support/notifications.rb:168:in `instrument' /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/activesupport-5.1.6/lib/active_support/cache.rb:555:in `instrument' /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/activesupport-5.1.6/lib/active_support/cache.rb:398:in `write' /usr/share/foreman/app/models/setting.rb:99:in `[]' /usr/share/foreman/lib/foreman/http_proxy.rb:4:in `http_proxy' /usr/share/foreman/lib/foreman/http_proxy.rb:13:in `proxy_http_request?' /usr/share/foreman/lib/foreman/http_proxy/rest_client_extension.rb:5:in `proxy_uri' /opt/theforeman/tfm/root/usr/share/gems/gems/rest-client-2.0.1/lib/restclient/request.rb:424:in `net_http_object' /opt/theforeman/tfm/root/usr/share/gems/gems/rest-client-2.0.1/lib/restclient/request.rb:651:in `transmit' /opt/theforeman/tfm/root/usr/share/gems/gems/rest-client-2.0.1/lib/restclient/request.rb:145:in `execute' /opt/theforeman/tfm/root/usr/share/gems/gems/rest-client-2.0.1/lib/restclient/request.rb:52:in `execute' /opt/theforeman/tfm/root/usr/share/gems/gems/rest-client-2.0.1/lib/restclient/resource.rb:51:in `get' /opt/theforeman/tfm/root/usr/share/gems/gems/runcible-2.8.1/lib/runcible/base.rb:92:in `get_response' /opt/theforeman/tfm/root/usr/share/gems/gems/runcible-2.8.1/lib/runcible/base.rb:72:in `call' /opt/theforeman/tfm/root/usr/share/gems/gems/runcible-2.8.1/lib/runcible/resources/task.rb:18:in `poll' /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.7.0.41/app/lib/actions/pulp/repository/sync.rb:17:in `invoke_external_task' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.0.5.1/lib/dynflow/action/polling.rb:83:in `initiate_external_action' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.0.5.1/lib/dynflow/action/polling.rb:18:in `run' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.0.5.1/lib/dynflow/action/cancellable.rb:13:in `run' /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.7.0.41/app/lib/actions/pulp/abstract_async_task.rb:45:in `run' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.0.5.1/lib/dynflow/action.rb:538:in `block (3 levels) in execute_run' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.0.5.1/lib/dynflow/middleware/stack.rb:26:in `pass' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.0.5.1/lib/dynflow/middleware.rb:18:in `pass' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.0.5.1/lib/dynflow/middleware.rb:31:in `run' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.0.5.1/lib/dynflow/middleware/stack.rb:22:in `call' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.0.5.1/lib/dynflow/middleware/stack.rb:26:in `pass' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.0.5.1/lib/dynflow/middleware.rb:18:in `pass' /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.7.0.41/app/lib/actions/middleware/remote_action.rb:16:in `block in run' /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.7.0.41/app/lib/actions/middleware/remote_action.rb:40:in `block in as_remote_user' /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.7.0.41/app/models/katello/concerns/user_extensions.rb:21:in `cp_config' /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.7.0.41/app/lib/actions/middleware/remote_action.rb:27:in `as_cp_user' /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.7.0.41/app/lib/actions/middleware/remote_action.rb:39:in `as_remote_user' /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.7.0.41/app/lib/actions/middleware/remote_action.rb:16:in `run' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.0.5.1/lib/dynflow/middleware/stack.rb:22:in `call' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.0.5.1/lib/dynflow/middleware/stack.rb:26:in `pass' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.0.5.1/lib/dynflow/middleware.rb:18:in `pass' /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-0.13.4.2/app/lib/actions/middleware/rails_executor_wrap.rb:14:in `block in run' /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/activesupport-5.1.6/lib/active_support/execution_wrapper.rb:85:in `wrap' /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-0.13.4.2/app/lib/actions/middleware/rails_executor_wrap.rb:13:in `run' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.0.5.1/lib/dynflow/middleware/stack.rb:22:in `call' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.0.5.1/lib/dynflow/middleware/stack.rb:26:in `pass' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.0.5.1/lib/dynflow/middleware.rb:18:in `pass' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.0.5.1/lib/dynflow/action/progress.rb:30:in `with_progress_calculation' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.0.5.1/lib/dynflow/action/progress.rb:16:in `run' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.0.5.1/lib/dynflow/middleware/stack.rb:22:in `call' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.0.5.1/lib/dynflow/middleware/stack.rb:26:in `pass' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.0.5.1/lib/dynflow/middleware.rb:18:in `pass' /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.7.0.41/app/lib/actions/middleware/keep_locale.rb:11:in `block in run' /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.7.0.41/app/lib/actions/middleware/keep_locale.rb:22:in `with_locale' /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.7.0.41/app/lib/actions/middleware/keep_locale.rb:11:in `run' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.0.5.1/lib/dynflow/middleware/stack.rb:22:in `call' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.0.5.1/lib/dynflow/middleware/stack.rb:26:in `pass' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.0.5.1/lib/dynflow/middleware.rb:18:in `pass' /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-0.13.4.2/app/lib/actions/middleware/keep_current_user.rb:15:in `block in run' /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-0.13.4.2/app/lib/actions/middleware/keep_current_user.rb:43:in `restore_curent_user' /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-0.13.4.2/app/lib/actions/middleware/keep_current_user.rb:15:in `run' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.0.5.1/lib/dynflow/middleware/stack.rb:22:in `call' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.0.5.1/lib/dynflow/middleware/stack.rb:26:in `pass' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.0.5.1/lib/dynflow/middleware.rb:18:in `pass' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.0.5.1/lib/dynflow/middleware.rb:31:in `run' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.0.5.1/lib/dynflow/middleware/stack.rb:22:in `call' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.0.5.1/lib/dynflow/middleware/world.rb:30:in `execute' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.0.5.1/lib/dynflow/action.rb:537:in `block (2 levels) in execute_run' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.0.5.1/lib/dynflow/action.rb:536:in `catch' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.0.5.1/lib/dynflow/action.rb:536:in `block in execute_run' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.0.5.1/lib/dynflow/action.rb:451:in `block in with_error_handling' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.0.5.1/lib/dynflow/action.rb:451:in `catch' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.0.5.1/lib/dynflow/action.rb:451:in `with_error_handling' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.0.5.1/lib/dynflow/action.rb:531:in `execute_run' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.0.5.1/lib/dynflow/action.rb:278:in `execute' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.0.5.1/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:17:in `block (2 levels) in execute' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.0.5.1/lib/dynflow/execution_plan/steps/abstract.rb:162:in `with_meta_calculation' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.0.5.1/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:16:in `block in execute' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.0.5.1/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:30:in `open_action' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.0.5.1/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:15:in `execute' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.0.5.1/lib/dynflow/director.rb:43:in `execute' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.0.5.1/lib/dynflow/executors/parallel/worker.rb:12:in `block in on_message' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.0.5.1/lib/dynflow/executors.rb:12:in `run_user_code' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.0.5.1/lib/dynflow/executors/parallel/worker.rb:11:in `on_message' /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.4/lib/concurrent/actor/context.rb:46:in `on_envelope' /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.4/lib/concurrent/actor/behaviour/executes_context.rb:7:in `on_envelope' /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.4/lib/concurrent/actor/behaviour/abstract.rb:25:in `pass' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.0.5.1/lib/dynflow/actor.rb:26:in `on_envelope' /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.4/lib/concurrent/actor/behaviour/abstract.rb:25:in `pass' /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.4/lib/concurrent/actor/behaviour/awaits.rb:15:in `on_envelope' /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.4/lib/concurrent/actor/behaviour/abstract.rb:25:in `pass' /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.4/lib/concurrent/actor/behaviour/sets_results.rb:14:in `on_envelope' /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.4/lib/concurrent/actor/behaviour/abstract.rb:25:in `pass' /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.4/lib/concurrent/actor/behaviour/buffer.rb:38:in `process_envelope' /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.4/lib/concurrent/actor/behaviour/buffer.rb:31:in `process_envelopes?' /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.4/lib/concurrent/actor/behaviour/buffer.rb:20:in `on_envelope' /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.4/lib/concurrent/actor/behaviour/abstract.rb:25:in `pass' /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.4/lib/concurrent/actor/behaviour/termination.rb:55:in `on_envelope' /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.4/lib/concurrent/actor/behaviour/abstract.rb:25:in `pass' /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.4/lib/concurrent/actor/behaviour/removes_child.rb:10:in `on_envelope' /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.4/lib/concurrent/actor/behaviour/abstract.rb:25:in `pass' /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.4/lib/concurrent/actor/behaviour/sets_results.rb:14:in `on_envelope' /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.4/lib/concurrent/actor/core.rb:161:in `process_envelope' /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.4/lib/concurrent/actor/core.rb:95:in `block in on_envelope' /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.4/lib/concurrent/actor/core.rb:118:in `block (2 levels) in schedule_execution' /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/concurrent-ruby-1.0.5/lib/concurrent/synchronization/mri_lockable_object.rb:38:in `block in synchronize' /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/concurrent-ruby-1.0.5/lib/concurrent/synchronization/mri_lockable_object.rb:38:in `synchronize' /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/concurrent-ruby-1.0.5/lib/concurrent/synchronization/mri_lockable_object.rb:38:in `synchronize' /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.4/lib/concurrent/actor/core.rb:115:in `block in schedule_execution' /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/concurrent-ruby-1.0.5/lib/concurrent/executor/serialized_execution.rb:18:in `call' /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/concurrent-ruby-1.0.5/lib/concurrent/executor/serialized_execution.rb:96:in `work' /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/concurrent-ruby-1.0.5/lib/concurrent/executor/serialized_execution.rb:77:in `block in call_job' /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/concurrent-ruby-1.0.5/lib/concurrent/executor/ruby_thread_pool_executor.rb:348:in `run_task' /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/concurrent-ruby-1.0.5/lib/concurrent/executor/ruby_thread_pool_executor.rb:337:in `block (3 levels) in create_worker' /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/concurrent-ruby-1.0.5/lib/concurrent/executor/ruby_thread_pool_executor.rb:320:in `loop' /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/concurrent-ruby-1.0.5/lib/concurrent/executor/ruby_thread_pool_executor.rb:320:in `block (2 levels) in create_worker' /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/concurrent-ruby-1.0.5/lib/concurrent/executor/ruby_thread_pool_executor.rb:319:in `catch' /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/concurrent-ruby-1.0.5/lib/concurrent/executor/ruby_thread_pool_executor.rb:319:in `block in create_worker' /opt/theforeman/tfm/root/usr/share/gems/gems/logging-2.2.2/lib/logging/diagnostic_context.rb:474:in `block in create_with_logging_context' Seems like specific issue in Rails framework, other references https://github.com/rails/rails/issues/26396, https://github.com/bfosberry/GameSocial/issues/273, perhaps some race condition. I wasn't able to reproduce the issue so far. Reliable reproduction steps would help getting to the root cause of it. Here is a reproducer that showed the issue for me:
```
cat <<EOF | foreman-rake console | tee ~/repro/pulp-$(date -Isec).log
conf.return_format = ""
log = Logger.new(STDOUT)
pulp_id = Katello::Repository.first.pulp_id
Katello.pulp_server.extensions.repository
threads = 4.times.map do |i|
Thread.new do
log.info "Start thread no. #{i+1}"
iteration = 0
cycle = 0
while true
begin
cycle += 1
if cycle == 1000
iteration += 1
cycle = 0
log.info "=== iteration #{iteration}k"
end
Katello.pulp_server.extensions.repository.distributions(pulp_id)
rescue Exception => e
log.info "Error at iteration #{iteration * 1000 + cycle}"
log.info "#{e.class} #{e.message}"
log.info e.backtrace.join("\n")
end
end
end
end
threads.each(&:join)
EOF
```
in separate console
```
i=0; while echo "" | foreman-rake console; do echo ===== $i; i=$((i+1)); sleep 1; done
```
Any foreman-rake command would probably do the trick. I've first noticed it when
all of my observations of the issue were happening at HH:00 or HH:30, which pointed
out to some rake tasks we run in cron.
Further investigation showed, that the issue was using combination of cache.read/write,
where we were supposed to use `fetch` instead.
In upstream, this has been already changed https://github.com/theforeman/foreman/pull/6257/files#diff-8c3f2a547e7f7d66e1756971a1ef20c2L95
and after making the changes locally, I was not able to reproduce the issue anymore.
Connecting redmine issue https://projects.theforeman.org/issues/25493 from this bug Moving this bug to POST for triage into Satellite 6 since the upstream issue https://projects.theforeman.org/issues/25493 has been resolved. Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2019:1222 |