Description of problem: After a successful sync of all repos one subtask of the Deploy task fails with "ActiveRecord::RecordInvalid: Validation failed: MAC address can't be blank". I kicked-off second deployment, the error persisted. Version-Release number of selected component (if applicable): RHCI-6.0-RHEL-7-20150723.t.0-RHCI-x86_64-dvd1.iso How reproducible: Reproduced twice Steps to Reproduce: 1. Start the installer, wait for repo sync 2. After that, one subtask fails Actual results: Installation fails with non-resumable error. Expected results: Installation continues (error is resumable). Additional info: Attaching output from foreman-debug and tarball of /var/log.
Created attachment 1057253 [details] foreman-debug
Created attachment 1057254 [details] tarball of /var/log
When this was encountered, the host was powered ON and discovered. Later, the host was powered OFF.
production.log excerpt: 2015-07-29 15:32:01 [W] XXX ip address is 192.168.11.101 2015-07-29 15:32:01 [W] XXX ================ Finished converting discovered host ==================== 2015-07-29 15:32:01 [E] Validation failed: MAC address can't be blank, MAC address can't be blank (ActiveRecord::RecordInvalid) /opt/rh/ruby193/root/usr/share/gems/gems/activerecord-3.2.8/lib/active_record/validations.rb:56:in `save!' /opt/rh/ruby193/root/usr/share/gems/gems/activerecord-3.2.8/lib/active_record/attribute_methods/dirty.rb:33:in `save!' /opt/rh/ruby193/root/usr/share/gems/gems/activerecord-3.2.8/lib/active_record/transactions.rb:246:in `block in save!' /opt/rh/ruby193/root/usr/share/gems/gems/activerecord-3.2.8/lib/active_record/transactions.rb:295:in `block in with_transaction_returning_status' /opt/rh/ruby193/root/usr/share/gems/gems/activerecord-3.2.8/lib/active_record/connection_adapters/abstract/database_statements.rb:192:in `transaction' /opt/rh/ruby193/root/usr/share/gems/gems/activerecord-3.2.8/lib/active_record/transactions.rb:208:in `transaction' /opt/rh/ruby193/root/usr/share/gems/gems/activerecord-3.2.8/lib/active_record/transactions.rb:293:in `with_transaction_returning_status' /opt/rh/ruby193/root/usr/share/gems/gems/activerecord-3.2.8/lib/active_record/transactions.rb:246:in `save!' /opt/rh/ruby193/root/usr/share/gems/gems/fusor_server-0.0.1/app/lib/actions/fusor/host/trigger_provisioning.rb:103:in `block in assign_host_to_hostgroup' /opt/rh/ruby193/root/usr/share/gems/gems/activerecord-3.2.8/lib/active_record/associations/collection_proxy.rb:89:in `each' /opt/rh/ruby193/root/usr/share/gems/gems/activerecord-3.2.8/lib/active_record/associations/collection_proxy.rb:89:in `method_missing' /opt/rh/ruby193/root/usr/share/gems/gems/fusor_server-0.0.1/app/lib/actions/fusor/host/trigger_provisioning.rb:101:in `assign_host_to_hostgroup' opt/rh/ruby193/root/usr/share/gems/gems/fusor_server-0.0.1/app/lib/actions/fusor/host/trigger_provisioning.rb:30:in `run' /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.7/lib/dynflow/action.rb:481:in `block (3 levels) in execute_run' /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.7/lib/dynflow/middleware/stack.rb:26:in `call' /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.7/lib/dynflow/middleware/stack.rb:26:in `pass' /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.7/lib/dynflow/middleware.rb:16:in `pass' /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.7/lib/dynflow/action/progress.rb:30:in `with_progress_calculation' /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.7/lib/dynflow/action/progress.rb:16:in `run' /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.7/lib/dynflow/middleware/stack.rb:22:in `call' /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.7/lib/dynflow/middleware/stack.rb:26:in `pass' /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.7/lib/dynflow/middleware.rb:16:in `pass' /opt/rh/ruby193/root/usr/share/gems/gems/katello-2.2.0.51/app/lib/actions/middleware/keep_locale.rb:23:in `block in run' /opt/rh/ruby193/root/usr/share/gems/gems/katello-2.2.0.51/app/lib/actions/middleware/keep_locale.rb:34:in `with_locale' /opt/rh/ruby193/root/usr/share/gems/gems/katello-2.2.0.51/app/lib/actions/middleware/keep_locale.rb:23:in `run' /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.7/lib/dynflow/middleware/stack.rb:22:in `call' /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.7/lib/dynflow/middleware/world.rb:30:in `execute' /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.7/lib/dynflow/action.rb:480:in `block (2 levels) in execute_run' /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.7/lib/dynflow/action.rb:479:in `catch' /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.7/lib/dynflow/action.rb:479:in `block in execute_run' /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.7/lib/dynflow/action.rb:396:in `call' /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.7/lib/dynflow/action.rb:396:in `block in with_error_handling' /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.7/lib/dynflow/action.rb:396:in `catch' /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.7/lib/dynflow/action.rb:396:in `with_error_handling' /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.7/lib/dynflow/action.rb:474:in `execute_run' /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.7/lib/dynflow/action.rb:256:in `execute' /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.7/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:9:in `block (2 levels) in execute' /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.7/lib/dynflow/execution_plan/steps/abstract.rb:155:in `call' /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.7/lib/dynflow/execution_plan/steps/abstract.rb:155:in `with_meta_calculation' /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.7/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:8:in `block in execute' /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.7/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:22:in `open_action' /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.7/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:7:in `execute' /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.7/lib/dynflow/executors/parallel/worker.rb:20:in `block in on_message' /opt/rh/ruby193/root/usr/share/gems/gems/algebrick-0.4.0/lib/algebrick.rb:859:in `block in assigns' /opt/rh/ruby193/root/usr/share/gems/gems/algebrick-0.4.0/lib/algebrick.rb:858:in `tap' /opt/rh/ruby193/root/usr/share/gems/gems/algebrick-0.4.0/lib/algebrick.rb:858:in `assigns' /opt/rh/ruby193/root/usr/share/gems/gems/algebrick-0.4.0/lib/algebrick.rb:138:in `match_value' /opt/rh/ruby193/root/usr/share/gems/gems/algebrick-0.4.0/lib/algebrick.rb:116:in `block in match' /opt/rh/ruby193/root/usr/share/gems/gems/algebrick-0.4.0/lib/algebrick.rb:115:in `each' /opt/rh/ruby193/root/usr/share/gems/gems/algebrick-0.4.0/lib/algebrick.rb:115:in `match' /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.7/lib/dynflow/executors/parallel/worker.rb:17:in `on_message' /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.7/lib/dynflow/micro_actor.rb:82:in `on_envelope' /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.7/lib/dynflow/micro_actor.rb:72:in `receive' /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.7/lib/dynflow/micro_actor.rb:99:in `block (2 levels) in run' /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.7/lib/dynflow/micro_actor.rb:99:in `loop' /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.7/lib/dynflow/micro_actor.rb:99:in `block in run' /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.7/lib/dynflow/micro_actor.rb:99:in `catch' /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.7/lib/dynflow/micro_actor.rb:99:in `run' /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.7/lib/dynflow/micro_actor.rb:13:in `block in initialize' /opt/rh/ruby193/root/usr/share/gems/gems/logging-1.8.1/lib/logging/diagnostic_context.rb:323:in `call' /opt/rh/ruby193/root/usr/share/gems/gems/logging-1.8.1/lib/logging/diagnostic_context.rb:323:in `block in create_with_logging_context' 2015-07-29 15:32:01 [E] A sub task failed (RuntimeError) /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.7/lib/dynflow/action/with_sub_plans.rb:158:in `check_for_errors!' /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.7/lib/dynflow/action/with_sub_plans.rb:80:in `try_to_finish' /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.7/lib/dynflow/action/with_sub_plans.rb:19:in `block in run' /opt/rh/ruby193/root/usr/share/gems/gems/algebrick-0.4.0/lib/algebrick.rb:859:in `block in assigns' /opt/rh/ruby193/root/usr/share/gems/gems/algebrick-0.4.0/lib/algebrick.rb:858:in `tap' /opt/rh/ruby193/root/usr/share/gems/gems/algebrick-0.4.0/lib/algebrick.rb:858:in `assigns' /opt/rh/ruby193/root/usr/share/gems/gems/algebrick-0.4.0/lib/algebrick.rb:138:in `match_value' /opt/rh/ruby193/root/usr/share/gems/gems/algebrick-0.4.0/lib/algebrick.rb:116:in `block in match' /opt/rh/ruby193/root/usr/share/gems/gems/algebrick-0.4.0/lib/algebrick.rb:115:in `each' /opt/rh/ruby193/root/usr/share/gems/gems/algebrick-0.4.0/lib/algebrick.rb:115:in `match' /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.7/lib/dynflow/action/with_sub_plans.rb:9:in `run' /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.7/lib/dynflow/action.rb:481:in `block (3 levels) in execute_run' /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.7/lib/dynflow/middleware/stack.rb:26:in `call' /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.7/lib/dynflow/middleware/stack.rb:26:in `pass' /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.7/lib/dynflow/middleware.rb:16:in `pass' /opt/rh/ruby193/root/usr/share/gems/gems/foreman-tasks-0.6.12.8/app/lib/actions/middleware/keep_current_user.rb:23:in `block in run' /opt/rh/ruby193/root/usr/share/gems/gems/foreman-tasks-0.6.12.8/app/lib/actions/middleware/keep_current_user.rb:34:in `restore_curent_user' /opt/rh/ruby193/root/usr/share/gems/gems/foreman-tasks-0.6.12.8/app/lib/actions/middleware/keep_current_user.rb:23:in `run' /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.7/lib/dynflow/middleware/stack.rb:22:in `call' /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.7/lib/dynflow/middleware/stack.rb:26:in `pass' /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.7/lib/dynflow/middleware.rb:16:in `pass' /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.7/lib/dynflow/action/progress.rb:30:in `with_progress_calculation' /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.7/lib/dynflow/action/progress.rb:16:in `run' /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.7/lib/dynflow/middleware/stack.rb:22:in `call' /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.7/lib/dynflow/middleware/stack.rb:26:in `pass' /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.7/lib/dynflow/middleware.rb:16:in `pass' /opt/rh/ruby193/root/usr/share/gems/gems/katello-2.2.0.51/app/lib/actions/middleware/keep_locale.rb:23:in `block in run' /opt/rh/ruby193/root/usr/share/gems/gems/katello-2.2.0.51/app/lib/actions/middleware/keep_locale.rb:34:in `with_locale' /opt/rh/ruby193/root/usr/share/gems/gems/katello-2.2.0.51/app/lib/actions/middleware/keep_locale.rb:23:in `run' /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.7/lib/dynflow/middleware/stack.rb:22:in `call' /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.7/lib/dynflow/middleware/world.rb:30:in `execute' /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.7/lib/dynflow/action.rb:480:in `block (2 levels) in execute_run' /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.7/lib/dynflow/action.rb:479:in `catch' /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.7/lib/dynflow/action.rb:479:in `block in execute_run' /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.7/lib/dynflow/action.rb:396:in `call' /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.7/lib/dynflow/action.rb:396:in `block in with_error_handling' /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.7/lib/dynflow/action.rb:396:in `catch' /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.7/lib/dynflow/action.rb:396:in `with_error_handling' /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.7/lib/dynflow/action.rb:474:in `execute_run' /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.7/lib/dynflow/action.rb:256:in `execute' /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.7/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:9:in `block (2 levels) in execute' /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.7/lib/dynflow/execution_plan/steps/abstract.rb:155:in `call' /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.7/lib/dynflow/execution_plan/steps/abstract.rb:155:in `with_meta_calculation' /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.7/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:8:in `block in execute' /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.7/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:22:in `open_action' /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.7/lib/dynflow/execution_plan/steps/abstract_flow_step.rb:7:in `execute' /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.7/lib/dynflow/executors/parallel/worker.rb:20:in `block in on_message' /opt/rh/ruby193/root/usr/share/gems/gems/algebrick-0.4.0/lib/algebrick.rb:859:in `block in assigns' /opt/rh/ruby193/root/usr/share/gems/gems/algebrick-0.4.0/lib/algebrick.rb:858:in `tap' /opt/rh/ruby193/root/usr/share/gems/gems/algebrick-0.4.0/lib/algebrick.rb:858:in `assigns' /opt/rh/ruby193/root/usr/share/gems/gems/algebrick-0.4.0/lib/algebrick.rb:138:in `match_value' /opt/rh/ruby193/root/usr/share/gems/gems/algebrick-0.4.0/lib/algebrick.rb:116:in `block in match' /opt/rh/ruby193/root/usr/share/gems/gems/algebrick-0.4.0/lib/algebrick.rb:115:in `each' /opt/rh/ruby193/root/usr/share/gems/gems/algebrick-0.4.0/lib/algebrick.rb:115:in `match' /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.7/lib/dynflow/executors/parallel/worker.rb:17:in `on_message' /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.7/lib/dynflow/micro_actor.rb:82:in `on_envelope' /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.7/lib/dynflow/micro_actor.rb:72:in `receive' /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.7/lib/dynflow/micro_actor.rb:99:in `block (2 levels) in run' /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.7/lib/dynflow/micro_actor.rb:99:in `loop' /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.7/lib/dynflow/micro_actor.rb:99:in `block in run' /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.7/lib/dynflow/micro_actor.rb:99:in `catch' /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.7/lib/dynflow/micro_actor.rb:99:in `run' /opt/rh/ruby193/root/usr/share/gems/gems/dynflow-0.7.7.7/lib/dynflow/micro_actor.rb:13:in `block in initialize' /opt/rh/ruby193/root/usr/share/gems/gems/logging-1.8.1/lib/logging/diagnostic_context.rb:323:in `call' /opt/rh/ruby193/root/usr/share/gems/gems/logging-1.8.1/lib/logging/diagnostic_context.rb:323:in `block in create_with_logging_context' 2015-07-29 15:32:01 [W] message could not be sent Dynflow::Listeners::Serialization::Protocol::Done[request_id: 10] because Broken pipe
*** Bug 1250625 has been marked as a duplicate of this bug. ***
It's not clear to me why we're doing this, but it is probably the cause. I want to do some testing to see if there is any reason at all for doing it that I can find. If not then commenting this out will likely fix it; otherwise modifying the behavior. https://github.com/fusor/fusor/blob/master/server/app/lib/actions/fusor/host/trigger_provisioning.rb#L101
This seems to be tied to systems with BMC's.
https://github.com/fusor/fusor/pull/339
In RHCI-6.0-RHEL-7-20150828.2
good 2 go in 0829