Bug 1474427 - Embedded Ansible role will not start if worker is killed during setup run
Summary: Embedded Ansible role will not start if worker is killed during setup run
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: Red Hat CloudForms Management Engine
Classification: Red Hat
Component: Appliance
Version: 5.8.0
Hardware: Unspecified
OS: Unspecified
unspecified
high
Target Milestone: GA
: 5.9.0
Assignee: Nick Carboni
QA Contact: luke couzens
URL:
Whiteboard: ansible_embed:black
Depends On:
Blocks: 1474835
TreeView+ depends on / blocked
 
Reported: 2017-07-24 15:08 UTC by Nick Carboni
Modified: 2018-03-06 15:18 UTC (History)
5 users (show)

Fixed In Version: 5.9.0.1
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
: 1474835 (view as bug list)
Environment:
Last Closed: 2018-03-06 15:18:50 UTC
Category: ---
Cloudforms Team: ---
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)

Description Nick Carboni 2017-07-24 15:08:29 UTC
Description of problem:
If the thread running the setup playbook is killed because it takes too long, we recognize the installation as "configured" but because the setup playbook didn't get to finish we don't have a /var/lib/awx/.tower_version file.

This leads to the following errors when we try to start the worker:

[----] I, [2017-07-24T11:05:14.619382 #1789:aa356d8]  INFO -- : MIQ(EmbeddedAnsibleWorker::Runner#setup_ansible) calling EmbeddedAnsible.start
[----] E, [2017-07-24T11:05:14.623501 #1789:aa356d8] ERROR -- : [Errno::ENOENT]: No such file or directory @ rb_sysopen - /var/lib/awx/.tower_version  Method:[rescue in do_before_work_loop]
[----] E, [2017-07-24T11:05:14.623561 #1789:aa356d8] ERROR -- : /var/www/miq/vmdb/lib/embedded_ansible.rb:204:in `read'
/var/www/miq/vmdb/lib/embedded_ansible.rb:204:in `local_tower_version'
/var/www/miq/vmdb/lib/embedded_ansible.rb:38:in `upgrade?'
/var/www/miq/vmdb/lib/embedded_ansible.rb:52:in `start'
/var/www/miq/vmdb/app/models/embedded_ansible_worker/runner.rb:36:in `setup_ansible'
/var/www/miq/vmdb/app/models/embedded_ansible_worker/runner.rb:13:in `do_before_work_loop'
/var/www/miq/vmdb/app/models/embedded_ansible_worker/runner.rb:7:in `prepare'
/var/www/miq/vmdb/app/models/miq_worker/runner.rb:133:in `start'
/var/www/miq/vmdb/app/models/miq_worker/runner.rb:21:in `start_worker'
/var/www/miq/vmdb/app/models/embedded_ansible_worker.rb:15:in `block in start_monitor_thread'

The net effect is that embedded ansible will never be able to start and the user will have to remove the /etc/tower/SECRET_KEY file to retry to install.

Version-Release number of selected component (if applicable):
5.8.1.4

How reproducible:
Always if you follow the steps in https://bugzilla.redhat.com/show_bug.cgi?id=1469307#c9


Actual results:
Embedded Ansible worker is killed once while retrying the configuration, but will then fail checking the upgrade file.

Expected results:
The Embedded Ansible worker should retry the setup if it didn't finish successfully.

Comment 3 CFME Bot 2017-07-24 21:46:32 UTC
New commit detected on ManageIQ/manageiq/master:
https://github.com/ManageIQ/manageiq/commit/cd863fd86318aa9a0366457b6402f8fa4fca8a92

commit cd863fd86318aa9a0366457b6402f8fa4fca8a92
Author:     Nick Carboni <ncarboni>
AuthorDate: Mon Jul 24 15:28:31 2017 -0400
Commit:     Nick Carboni <ncarboni>
CommitDate: Mon Jul 24 16:42:02 2017 -0400

    Add a marker file for determining when the ansible setup has been run
    
    This ensures that we have a definitive way to determine if we have
    successfully run the setup for embedded ansible on a particular server
    
    Before this change, the thread running the setup could be killed
    and when we tried to start the service again, we would see #configured?
    as true even if we didn't run through the whole playbook.
    
    Now, we will not write this marker file until we finish running
    the playbook so the next worker will know to attempt the setup if
    the previous one was killed.
    
    https://bugzilla.redhat.com/show_bug.cgi?id=1474427

 lib/embedded_ansible.rb           | 21 +++++++++++++++++++++
 spec/lib/embedded_ansible_spec.rb | 15 ++++++++++++++-
 2 files changed, 35 insertions(+), 1 deletion(-)

Comment 5 luke couzens 2017-10-12 16:17:14 UTC
Verified in 5.9.0.2


Note You need to log in before you can comment on or make changes to this bug.