Description of problem: Remote execution task sometimes fails with the following error: Action: Actions::RemoteExecution::RunHostJob Input: {"host"=>{"id"=>184, "name"=>"docker1container14.example.com"}, "job_category"=>"Commands", "description"=>"Run date", "delegated_action_id"=>2, "locale"=>"en", "current_user_id"=>4} Output: {} Exception: ActiveRecord::SubclassNotFound: Invalid single-table inheritance type: Katello::ErrataStatus is not a subclass of HostStatus::ExecutionStatus Backtrace: /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/activerecord-5.1.6/lib/active_record/inheritance.rb:203:in `find_sti_class' /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/activerecord-5.1.6/lib/active_record/inheritance.rb:177:in `discriminate_class_for_record' /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/activerecord-5.1.6/lib/active_record/persistence.rb:67:in `instantiate' /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/activerecord-5.1.6/lib/active_record/querying.rb:50:in `block (2 levels) in find_by_sql' /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/activerecord-5.1.6/lib/active_record/result.rb:55:in `block in each' /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/activerecord-5.1.6/lib/active_record/result.rb:55:in `each' /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/activerecord-5.1.6/lib/active_record/result.rb:55:in `each' /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/activerecord-5.1.6/lib/active_record/querying.rb:50:in `map' /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/activerecord-5.1.6/lib/active_record/querying.rb:50:in `block in find_by_sql' /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/activesupport-5.1.6/lib/active_support/notifications/instrumenter.rb:21:in `instrument' /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/activerecord-5.1.6/lib/active_record/querying.rb:49:in `find_by_sql' /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/activerecord-5.1.6/lib/active_record/relation.rb:678:in `exec_queries' /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/activerecord-5.1.6/lib/active_record/association_relation.rb:32:in `exec_queries' /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/activerecord-5.1.6/lib/active_record/relation.rb:546:in `load' /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/activerecord-5.1.6/lib/active_record/relation.rb:255:in `records' /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/activerecord-5.1.6/lib/active_record/relation/finder_methods.rb:508:in `find_take' /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/activerecord-5.1.6/lib/active_record/relation/finder_methods.rb:100:in `take' /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/activerecord-5.1.6/lib/active_record/associations/singular_association.rb:39:in `find_target' /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/activerecord-5.1.6/lib/active_record/associations/association.rb:157:in `load_target' /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/activerecord-5.1.6/lib/active_record/associations/association.rb:53:in `reload' /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/activerecord-5.1.6/lib/active_record/associations/singular_association.rb:7:in `reader' /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/activerecord-5.1.6/lib/active_record/associations/builder/association.rb:111:in `execution_status_object' /opt/theforeman/tfm/root/usr/share/gems/gems/foreman_remote_execution-1.5.3/app/lib/actions/remote_execution/run_host_job.rb:123:in `update_host_status' /opt/theforeman/tfm/root/usr/share/gems/gems/foreman_remote_execution-1.5.3/app/lib/actions/remote_execution/run_host_job.rb:57:in `finalize' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.0.5/lib/dynflow/action.rb:560:in `block (2 levels) in execute_finalize' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.0.5/lib/dynflow/middleware/stack.rb:26:in `pass' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.0.5/lib/dynflow/middleware.rb:18:in `pass' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.0.5/lib/dynflow/middleware.rb:39:in `finalize' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.0.5/lib/dynflow/middleware/stack.rb:22:in `call' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.0.5/lib/dynflow/middleware/stack.rb:26:in `pass' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.0.5/lib/dynflow/middleware.rb:18:in `pass' /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-0.13.2/app/lib/actions/middleware/rails_executor_wrap.rb:20:in `block in finalize' /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/activesupport-5.1.6/lib/active_support/execution_wrapper.rb:85:in `wrap' /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-0.13.2/app/lib/actions/middleware/rails_executor_wrap.rb:19:in `finalize' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.0.5/lib/dynflow/middleware/stack.rb:22:in `call' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.0.5/lib/dynflow/middleware/stack.rb:26:in `pass' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.0.5/lib/dynflow/middleware.rb:18:in `pass' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.0.5/lib/dynflow/action/progress.rb:30:in `with_progress_calculation' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.0.5/lib/dynflow/action/progress.rb:22:in `finalize' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.0.5/lib/dynflow/middleware/stack.rb:22:in `call' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.0.5/lib/dynflow/middleware/stack.rb:26:in `pass' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.0.5/lib/dynflow/middleware.rb:18:in `pass' /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.7.0.rc1/app/lib/actions/middleware/keep_locale.rb:15:in `block in finalize' /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.7.0.rc1/app/lib/actions/middleware/keep_locale.rb:22:in `with_locale' /opt/theforeman/tfm/root/usr/share/gems/gems/katello-3.7.0.rc1/app/lib/actions/middleware/keep_locale.rb:15:in `finalize' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.0.5/lib/dynflow/middleware/stack.rb:22:in `call' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.0.5/lib/dynflow/middleware/stack.rb:26:in `pass' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.0.5/lib/dynflow/middleware.rb:18:in `pass' /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-0.13.2/app/lib/actions/middleware/keep_current_user.rb:19:in `block in finalize' /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-0.13.2/app/lib/actions/middleware/keep_current_user.rb:43:in `restore_curent_user' /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-0.13.2/app/lib/actions/middleware/keep_current_user.rb:19:in `finalize' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.0.5/lib/dynflow/middleware/stack.rb:22:in `call' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.0.5/lib/dynflow/middleware/world.rb:30:in `execute' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.0.5/lib/dynflow/action.rb:559:in `block in execute_finalize' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.0.5/lib/dynflow/action.rb:451:in `block in with_error_handling' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.0.5/lib/dynflow/action.rb:451:in `catch' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.0.5/lib/dynflow/action.rb:451:in `with_error_handling' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.0.5/lib/dynflow/action.rb:558:in `execute_finalize' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.0.5/lib/dynflow/action.rb:278:in `execute' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.0.5/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:17:in `block (2 levels) in execute' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.0.5/lib/dynflow/execution_plan/steps/abstract.rb:162:in `with_meta_calculation' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.0.5/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:16:in `block in execute' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.0.5/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:30:in `open_action' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.0.5/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:15:in `execute' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.0.5/lib/dynflow/director/sequential_manager.rb:68:in `run_step' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.0.5/lib/dynflow/director/sequential_manager.rb:53:in `dispatch' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.0.5/lib/dynflow/director/sequential_manager.rb:60:in `block in run_in_sequence' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.0.5/lib/dynflow/director/sequential_manager.rb:60:in `each' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.0.5/lib/dynflow/director/sequential_manager.rb:60:in `all?' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.0.5/lib/dynflow/director/sequential_manager.rb:60:in `run_in_sequence' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.0.5/lib/dynflow/director/sequential_manager.rb:49:in `dispatch' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.0.5/lib/dynflow/director/sequential_manager.rb:27:in `block in finalize' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.0.5/lib/dynflow/middleware/stack.rb:26:in `pass' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.0.5/lib/dynflow/middleware.rb:18:in `pass' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.0.5/lib/dynflow/middleware.rb:47:in `finalize_phase' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.0.5/lib/dynflow/middleware/stack.rb:22:in `call' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.0.5/lib/dynflow/middleware/stack.rb:26:in `pass' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.0.5/lib/dynflow/middleware.rb:18:in `pass' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.0.5/lib/dynflow/middleware.rb:47:in `finalize_phase' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.0.5/lib/dynflow/middleware/stack.rb:22:in `call' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.0.5/lib/dynflow/middleware/stack.rb:26:in `pass' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.0.5/lib/dynflow/middleware.rb:18:in `pass' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.0.5/lib/dynflow/middleware.rb:47:in `finalize_phase' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.0.5/lib/dynflow/middleware/stack.rb:22:in `call' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.0.5/lib/dynflow/middleware/stack.rb:26:in `pass' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.0.5/lib/dynflow/middleware.rb:18:in `pass' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.0.5/lib/dynflow/middleware.rb:47:in `finalize_phase' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.0.5/lib/dynflow/middleware/stack.rb:22:in `call' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.0.5/lib/dynflow/middleware/stack.rb:26:in `pass' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.0.5/lib/dynflow/middleware.rb:18:in `pass' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.0.5/lib/dynflow/middleware.rb:47:in `finalize_phase' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.0.5/lib/dynflow/middleware/stack.rb:22:in `call' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.0.5/lib/dynflow/middleware/world.rb:30:in `execute' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.0.5/lib/dynflow/director/sequential_manager.rb:26:in `finalize' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.0.5/lib/dynflow/director.rb:67:in `execute' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.0.5/lib/dynflow/executors/parallel/worker.rb:12:in `block in on_message' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.0.5/lib/dynflow/executors.rb:12:in `run_user_code' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.0.5/lib/dynflow/executors/parallel/worker.rb:11:in `on_message' /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.4/lib/concurrent/actor/context.rb:46:in `on_envelope' /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.4/lib/concurrent/actor/behaviour/executes_context.rb:7:in `on_envelope' /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.4/lib/concurrent/actor/behaviour/abstract.rb:25:in `pass' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.0.5/lib/dynflow/actor.rb:26:in `on_envelope' /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.4/lib/concurrent/actor/behaviour/abstract.rb:25:in `pass' /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.4/lib/concurrent/actor/behaviour/awaits.rb:15:in `on_envelope' /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.4/lib/concurrent/actor/behaviour/abstract.rb:25:in `pass' /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.4/lib/concurrent/actor/behaviour/sets_results.rb:14:in `on_envelope' /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.4/lib/concurrent/actor/behaviour/abstract.rb:25:in `pass' /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.4/lib/concurrent/actor/behaviour/buffer.rb:38:in `process_envelope' /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.4/lib/concurrent/actor/behaviour/buffer.rb:31:in `process_envelopes?' /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.4/lib/concurrent/actor/behaviour/buffer.rb:20:in `on_envelope' /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.4/lib/concurrent/actor/behaviour/abstract.rb:25:in `pass' /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.4/lib/concurrent/actor/behaviour/termination.rb:55:in `on_envelope' /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.4/lib/concurrent/actor/behaviour/abstract.rb:25:in `pass' /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.4/lib/concurrent/actor/behaviour/removes_child.rb:10:in `on_envelope' /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.4/lib/concurrent/actor/behaviour/abstract.rb:25:in `pass' /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.4/lib/concurrent/actor/behaviour/sets_results.rb:14:in `on_envelope' /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.4/lib/concurrent/actor/core.rb:161:in `process_envelope' /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.4/lib/concurrent/actor/core.rb:95:in `block in on_envelope' /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.4/lib/concurrent/actor/core.rb:118:in `block (2 levels) in schedule_execution' /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/concurrent-ruby-1.0.5/lib/concurrent/synchronization/mri_lockable_object.rb:38:in `block in synchronize' /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/concurrent-ruby-1.0.5/lib/concurrent/synchronization/mri_lockable_object.rb:38:in `synchronize' /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/concurrent-ruby-1.0.5/lib/concurrent/synchronization/mri_lockable_object.rb:38:in `synchronize' /opt/theforeman/tfm/root/usr/share/gems/gems/concurrent-ruby-edge-0.2.4/lib/concurrent/actor/core.rb:115:in `block in schedule_execution' /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/concurrent-ruby-1.0.5/lib/concurrent/executor/serialized_execution.rb:18:in `call' /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/concurrent-ruby-1.0.5/lib/concurrent/executor/serialized_execution.rb:96:in `work' /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/concurrent-ruby-1.0.5/lib/concurrent/executor/serialized_execution.rb:77:in `block in call_job' /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/concurrent-ruby-1.0.5/lib/concurrent/executor/ruby_thread_pool_executor.rb:348:in `run_task' /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/concurrent-ruby-1.0.5/lib/concurrent/executor/ruby_thread_pool_executor.rb:337:in `block (3 levels) in create_worker' /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/concurrent-ruby-1.0.5/lib/concurrent/executor/ruby_thread_pool_executor.rb:320:in `loop' /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/concurrent-ruby-1.0.5/lib/concurrent/executor/ruby_thread_pool_executor.rb:320:in `block (2 levels) in create_worker' /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/concurrent-ruby-1.0.5/lib/concurrent/executor/ruby_thread_pool_executor.rb:319:in `catch' /opt/theforeman/tfm-ror51/root/usr/share/gems/gems/concurrent-ruby-1.0.5/lib/concurrent/executor/ruby_thread_pool_executor.rb:319:in `block in create_worker' /opt/theforeman/tfm/root/usr/share/gems/gems/logging-2.2.2/lib/logging/diagnostic_context.rb:474:in `block in create_with_logging_context' Version-Release number of selected component (if applicable): Satellite 6.4 Snap 9 How reproducible: Occasionally Steps to Reproduce: Run lot of parallel ReX jobs. My scenario was running a simple "date" command on 200 docker containers via ssh. Actual results: An "ActiveRecord::SubclassNotFound: Invalid single-table inheritance type: Katello::ErrataStatus is not a subclass of HostStatus::ExecutionStatus" error, the task is marked as failed. Expected results: No error, task is marked as successful.
*** This bug has been marked as a duplicate of bug 1594826 ***