Hide Forgot
Description of problem: When running a remote job, if the host has not been setup with the appropriate SSH key (Satellite's /var/lib/foreman-proxy/ssh/id_rsa_foreman_proxy.pub) in its /root/.ssh/authorized_keys file, the job will fail. It will fail with a Ruby traceback that provides no details as to the actual problem, making it much more difficult for users to know what the problem is and and correcting it. Steps to Reproduce: 1. Run a job against a host that has not been setup with foreman-proxy's SSH key. Actual results: Job will fail with the following traceback in the task's Errors tab. Nowhere is SSH or keys mentioned. StandardError: Job execution failed Backtrace: /opt/theforeman/tfm/root/usr/share/gems/gems/algebrick-0.7.3/lib/algebrick/matchers/abstract.rb:74:in `block in assigns' /opt/theforeman/tfm/root/usr/share/gems/gems/algebrick-0.7.3/lib/algebrick/matchers/abstract.rb:73:in `tap' /opt/theforeman/tfm/root/usr/share/gems/gems/algebrick-0.7.3/lib/algebrick/matchers/abstract.rb:73:in `assigns' /opt/theforeman/tfm/root/usr/share/gems/gems/algebrick-0.7.3/lib/algebrick/matching.rb:56:in `match_value' /opt/theforeman/tfm/root/usr/share/gems/gems/algebrick-0.7.3/lib/algebrick/matching.rb:36:in `block in match?' /opt/theforeman/tfm/root/usr/share/gems/gems/algebrick-0.7.3/lib/algebrick/matching.rb:35:in `each' /opt/theforeman/tfm/root/usr/share/gems/gems/algebrick-0.7.3/lib/algebrick/matching.rb:35:in `match?' /opt/theforeman/tfm/root/usr/share/gems/gems/algebrick-0.7.3/lib/algebrick/matching.rb:23:in `match' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.9/lib/dynflow/execution_plan/steps/error.rb:13:in `new' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.9/lib/dynflow/action.rb:502:in `set_error' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.9/lib/dynflow/action.rb:465:in `error!' /opt/theforeman/tfm/root/usr/share/gems/gems/foreman_remote_execution-4.5.6/app/lib/actions/remote_execution/run_host_job.rb:98:in `check_exit_status' /opt/theforeman/tfm/root/usr/share/gems/gems/foreman_remote_execution-4.5.6/app/lib/actions/remote_execution/run_host_job.rb:69:in `finalize' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.9/lib/dynflow/action.rb:594:in `block (2 levels) in execute_finalize' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.9/lib/dynflow/middleware/stack.rb:27:in `pass' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.9/lib/dynflow/middleware.rb:19:in `pass' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.9/lib/dynflow/middleware.rb:40:in `finalize' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.9/lib/dynflow/middleware/stack.rb:23:in `call' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.9/lib/dynflow/middleware/stack.rb:27:in `pass' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.9/lib/dynflow/middleware.rb:19:in `pass' /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-4.1.5/app/lib/actions/middleware/rails_executor_wrap.rb:20:in `block in finalize' /opt/theforeman/tfm/root/usr/share/gems/gems/activesupport-6.0.3.7/lib/active_support/execution_wrapper.rb:88:in `wrap' /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-4.1.5/app/lib/actions/middleware/rails_executor_wrap.rb:19:in `finalize' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.9/lib/dynflow/middleware/stack.rb:23:in `call' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.9/lib/dynflow/middleware/stack.rb:27:in `pass' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.9/lib/dynflow/middleware.rb:19:in `pass' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.9/lib/dynflow/action/progress.rb:31:in `with_progress_calculation' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.9/lib/dynflow/action/progress.rb:23:in `finalize' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.9/lib/dynflow/middleware/stack.rb:23:in `call' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.9/lib/dynflow/middleware/stack.rb:27:in `pass' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.9/lib/dynflow/middleware.rb:19:in `pass' /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-4.1.5/app/lib/actions/middleware/load_setting_values.rb:25:in `finalize' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.9/lib/dynflow/middleware/stack.rb:23:in `call' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.9/lib/dynflow/middleware/stack.rb:27:in `pass' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.9/lib/dynflow/middleware.rb:19:in `pass' /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-4.1.5/app/lib/actions/middleware/keep_current_request_id.rb:19:in `block in finalize' /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-4.1.5/app/lib/actions/middleware/keep_current_request_id.rb:52:in `restore_current_request_id' /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-4.1.5/app/lib/actions/middleware/keep_current_request_id.rb:19:in `finalize' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.9/lib/dynflow/middleware/stack.rb:23:in `call' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.9/lib/dynflow/middleware/stack.rb:27:in `pass' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.9/lib/dynflow/middleware.rb:19:in `pass' /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-4.1.5/app/lib/actions/middleware/keep_current_timezone.rb:19:in `block in finalize' /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-4.1.5/app/lib/actions/middleware/keep_current_timezone.rb:44:in `restore_curent_timezone' /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-4.1.5/app/lib/actions/middleware/keep_current_timezone.rb:19:in `finalize' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.9/lib/dynflow/middleware/stack.rb:23:in `call' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.9/lib/dynflow/middleware/stack.rb:27:in `pass' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.9/lib/dynflow/middleware.rb:19:in `pass' /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-4.1.5/app/lib/actions/middleware/keep_current_user.rb:25:in `block in finalize' /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-4.1.5/app/lib/actions/middleware/keep_current_user.rb:54:in `restore_curent_user' /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-4.1.5/app/lib/actions/middleware/keep_current_user.rb:25:in `finalize' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.9/lib/dynflow/middleware/stack.rb:23:in `call' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.9/lib/dynflow/middleware/stack.rb:27:in `pass' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.9/lib/dynflow/middleware.rb:19:in `pass' /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-4.1.5/app/lib/actions/middleware/keep_current_taxonomies.rb:19:in `block in finalize' /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-4.1.5/app/lib/actions/middleware/keep_current_taxonomies.rb:45:in `restore_current_taxonomies' /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-4.1.5/app/lib/actions/middleware/keep_current_taxonomies.rb:19:in `finalize' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.9/lib/dynflow/middleware/stack.rb:23:in `call' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.9/lib/dynflow/middleware/world.rb:31:in `execute' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.9/lib/dynflow/action.rb:593:in `block in execute_finalize' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.9/lib/dynflow/action.rb:473:in `block in with_error_handling' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.9/lib/dynflow/action.rb:473:in `catch' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.9/lib/dynflow/action.rb:473:in `with_error_handling' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.9/lib/dynflow/action.rb:592:in `execute_finalize' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.9/lib/dynflow/action.rb:286:in `execute' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.9/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:18:in `block (2 levels) in execute' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.9/lib/dynflow/execution_plan/steps/abstract.rb:167:in `with_meta_calculation' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.9/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:17:in `block in execute' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.9/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:32:in `open_action' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.9/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:16:in `execute' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.9/lib/dynflow/director/sequential_manager.rb:78:in `run_step' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.9/lib/dynflow/director/sequential_manager.rb:63:in `dispatch' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.9/lib/dynflow/director/sequential_manager.rb:70:in `block in run_in_sequence' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.9/lib/dynflow/director/sequential_manager.rb:70:in `all?' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.9/lib/dynflow/director/sequential_manager.rb:70:in `run_in_sequence' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.9/lib/dynflow/director/sequential_manager.rb:59:in `dispatch' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.9/lib/dynflow/director/sequential_manager.rb:28:in `block in finalize' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.9/lib/dynflow/middleware/stack.rb:27:in `pass' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.9/lib/dynflow/middleware.rb:19:in `pass' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.9/lib/dynflow/middleware.rb:48:in `finalize_phase' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.9/lib/dynflow/middleware/stack.rb:23:in `call' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.9/lib/dynflow/middleware/stack.rb:27:in `pass' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.9/lib/dynflow/middleware.rb:19:in `pass' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.9/lib/dynflow/middleware.rb:48:in `finalize_phase' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.9/lib/dynflow/middleware/stack.rb:23:in `call' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.9/lib/dynflow/middleware/stack.rb:27:in `pass' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.9/lib/dynflow/middleware.rb:19:in `pass' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.9/lib/dynflow/middleware.rb:48:in `finalize_phase' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.9/lib/dynflow/middleware/stack.rb:23:in `call' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.9/lib/dynflow/middleware/stack.rb:27:in `pass' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.9/lib/dynflow/middleware.rb:19:in `pass' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.9/lib/dynflow/middleware.rb:48:in `finalize_phase' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.9/lib/dynflow/middleware/stack.rb:23:in `call' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.9/lib/dynflow/middleware/stack.rb:27:in `pass' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.9/lib/dynflow/middleware.rb:19:in `pass' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.9/lib/dynflow/middleware.rb:48:in `finalize_phase' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.9/lib/dynflow/middleware/stack.rb:23:in `call' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.9/lib/dynflow/middleware/stack.rb:27:in `pass' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.9/lib/dynflow/middleware.rb:19:in `pass' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.9/lib/dynflow/middleware.rb:48:in `finalize_phase' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.9/lib/dynflow/middleware/stack.rb:23:in `call' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.9/lib/dynflow/middleware/stack.rb:27:in `pass' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.9/lib/dynflow/middleware.rb:19:in `pass' /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-4.1.5/app/lib/actions/middleware/keep_current_user.rb:29:in `block in finalize_phase' /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-4.1.5/app/lib/actions/middleware/keep_current_user.rb:54:in `restore_curent_user' /opt/theforeman/tfm/root/usr/share/gems/gems/foreman-tasks-4.1.5/app/lib/actions/middleware/keep_current_user.rb:29:in `finalize_phase' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.9/lib/dynflow/middleware/stack.rb:23:in `call' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.9/lib/dynflow/middleware/stack.rb:27:in `pass' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.9/lib/dynflow/middleware.rb:19:in `pass' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.9/lib/dynflow/middleware.rb:48:in `finalize_phase' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.9/lib/dynflow/middleware/stack.rb:23:in `call' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.9/lib/dynflow/middleware/world.rb:31:in `execute' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.9/lib/dynflow/director/sequential_manager.rb:27:in `finalize' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.9/lib/dynflow/director.rb:123:in `execute' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.9/lib/dynflow/executors/sidekiq/worker_jobs.rb:11:in `block (2 levels) in perform' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.9/lib/dynflow/executors.rb:18:in `run_user_code' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.9/lib/dynflow/executors/sidekiq/worker_jobs.rb:9:in `block in perform' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.9/lib/dynflow/executors/sidekiq/worker_jobs.rb:25:in `with_telemetry' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.9/lib/dynflow/executors/sidekiq/worker_jobs.rb:8:in `perform' /opt/theforeman/tfm/root/usr/share/gems/gems/dynflow-1.4.9/lib/dynflow/executors/sidekiq/serialization.rb:27:in `perform' /opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:192:in `execute_job' /opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:165:in `block (2 levels) in process' /opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/middleware/chain.rb:128:in `block in invoke' /opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/middleware/chain.rb:133:in `invoke' /opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:164:in `block in process' /opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:137:in `block (6 levels) in dispatch' /opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/job_retry.rb:109:in `local' /opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:136:in `block (5 levels) in dispatch' /opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq.rb:37:in `block in <module:Sidekiq>' /opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:132:in `block (4 levels) in dispatch' /opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:250:in `stats' /opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:127:in `block (3 levels) in dispatch' /opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/job_logger.rb:8:in `call' /opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:126:in `block (2 levels) in dispatch' /opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/job_retry.rb:74:in `global' /opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:125:in `block in dispatch' /opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/logging.rb:48:in `with_context' /opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/logging.rb:42:in `with_job_hash_context' /opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:124:in `dispatch' /opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:163:in `process' /opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:83:in `process_one' /opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:71:in `run' /opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/util.rb:16:in `watchdog' /opt/theforeman/tfm/root/usr/share/gems/gems/sidekiq-5.2.7/lib/sidekiq/util.rb:25:in `block in safe_thread' /opt/theforeman/tfm/root/usr/share/gems/gems/logging-2.3.0/lib/logging/diagnostic_context.rb:474:in `block in create_with_logging_context' Expected results: A more meaningful error is given.
This is being worked on. On a side note, as far as remote execution goes, tasks are not meant to be inspected directly by users and may contain raw data just as you're describing. The best way would be to look at the hosts output (job invocation details page > click hostname in the table below).
(In reply to Adam Ruzicka from comment #1) === > This is being worked on. === Great to know. === > The best way would be to look at the hosts output (job invocation details page > click hostname in the table below). === Oh my gosh -- thank you for pointing that out. Although I saw the host name displayed as a web link in the table at the bottom of the job invocation page, I just figured that would take me back to the host's All Hosts page since that was where I had clicked the Schedule Remote Job button. Clicking on the host's name in that table *does* show the plain, understandable reason why the job failed. I was expecting that in the Host Detail button or the Host Task selection from that button's drop-down menu. That link really needs changed from just the host name to "job output for host.example.com" or perhaps that table's column header needs changed from "Host" to "Job Result Details From Host".
Moving this bug to POST for triage into Satellite since the upstream issue https://projects.theforeman.org/issues/34363 has been resolved.
smart_proxy_remote_execution_ssh-0.6.0 containing the fix for this just went out.
Verified on Satellite 6.12 sn3, the messages in host job output are now more telling