Description of problem: If the thread running the setup playbook is killed because it takes too long, we recognize the installation as "configured" but because the setup playbook didn't get to finish we don't have a /var/lib/awx/.tower_version file. This leads to the following errors when we try to start the worker: [----] I, [2017-07-24T11:05:14.619382 #1789:aa356d8] INFO -- : MIQ(EmbeddedAnsibleWorker::Runner#setup_ansible) calling EmbeddedAnsible.start [----] E, [2017-07-24T11:05:14.623501 #1789:aa356d8] ERROR -- : [Errno::ENOENT]: No such file or directory @ rb_sysopen - /var/lib/awx/.tower_version Method:[rescue in do_before_work_loop] [----] E, [2017-07-24T11:05:14.623561 #1789:aa356d8] ERROR -- : /var/www/miq/vmdb/lib/embedded_ansible.rb:204:in `read' /var/www/miq/vmdb/lib/embedded_ansible.rb:204:in `local_tower_version' /var/www/miq/vmdb/lib/embedded_ansible.rb:38:in `upgrade?' /var/www/miq/vmdb/lib/embedded_ansible.rb:52:in `start' /var/www/miq/vmdb/app/models/embedded_ansible_worker/runner.rb:36:in `setup_ansible' /var/www/miq/vmdb/app/models/embedded_ansible_worker/runner.rb:13:in `do_before_work_loop' /var/www/miq/vmdb/app/models/embedded_ansible_worker/runner.rb:7:in `prepare' /var/www/miq/vmdb/app/models/miq_worker/runner.rb:133:in `start' /var/www/miq/vmdb/app/models/miq_worker/runner.rb:21:in `start_worker' /var/www/miq/vmdb/app/models/embedded_ansible_worker.rb:15:in `block in start_monitor_thread' The net effect is that embedded ansible will never be able to start and the user will have to remove the /etc/tower/SECRET_KEY file to retry to install. Version-Release number of selected component (if applicable): 5.8.1.4 How reproducible: Always if you follow the steps in https://bugzilla.redhat.com/show_bug.cgi?id=1469307#c9 Actual results: Embedded Ansible worker is killed once while retrying the configuration, but will then fail checking the upgrade file. Expected results: The Embedded Ansible worker should retry the setup if it didn't finish successfully.
https://github.com/ManageIQ/manageiq/pull/15642
New commit detected on ManageIQ/manageiq/master: https://github.com/ManageIQ/manageiq/commit/cd863fd86318aa9a0366457b6402f8fa4fca8a92 commit cd863fd86318aa9a0366457b6402f8fa4fca8a92 Author: Nick Carboni <ncarboni> AuthorDate: Mon Jul 24 15:28:31 2017 -0400 Commit: Nick Carboni <ncarboni> CommitDate: Mon Jul 24 16:42:02 2017 -0400 Add a marker file for determining when the ansible setup has been run This ensures that we have a definitive way to determine if we have successfully run the setup for embedded ansible on a particular server Before this change, the thread running the setup could be killed and when we tried to start the service again, we would see #configured? as true even if we didn't run through the whole playbook. Now, we will not write this marker file until we finish running the playbook so the next worker will know to attempt the setup if the previous one was killed. https://bugzilla.redhat.com/show_bug.cgi?id=1474427 lib/embedded_ansible.rb | 21 +++++++++++++++++++++ spec/lib/embedded_ansible_spec.rb | 15 ++++++++++++++- 2 files changed, 35 insertions(+), 1 deletion(-)
Verified in 5.9.0.2