Description of problem:

Various connection related errors when provisioning an Ansible Playbook service before embedded role has finished starting.

How reproducible:

Always

Steps to Reproduce:
1. Reboot CF appliance
2. Provision playbook service
3. See different errors depending on how fast you were

Actual results:

See errors appearing to be connection errors, timeouts, but are really "tower hasn't started".

Expected results:

Things work.

Additional info:

Did you run this even once?



Example log snippets:


[----] E, [2017-07-31T16:35:54.405350 #4126:b64c20] ERROR -- : Q-task_id([service_template_provision_task_1000000000047]) MIQ(ManageIQ::Providers::EmbeddedAnsible::AutomationManager::Job.raw
_create_stack) Failed to create job from template(ManageIQ::Providers::EmbeddedAnsible::AutomationManager::Job), error: Net::ReadTimeout
[----] E, [2017-07-31T16:35:54.405949 #4126:b64c20] ERROR -- : Q-task_id([service_template_provision_task_1000000000047]) <AutomationEngine> The following error occurred during instance meth
od <execute> for AR object <#<ServiceAnsiblePlaybook id: 1000000000033, name: "Ticket Monster App", description: "Ticket Monster App", guid: "287d4f70-760e-11e7-99d4-525400d639c8", type: "Se
rviceAnsiblePlaybook", service_template_id: 1000000000003, options: {:config_info=>{:provision=>{:repository_id=>1000000000001, :playbook_id=>1000000000018, :credential_id=>1000000000008, :h
osts=>"localhost", :extra_vars=>{}, :become_enabled=>false, :new_dialog_name=>"Ticket Monster App", :fqname=>"/Service/Generic/StateMachines/GenericLifecycle/provision", :dialog_id=>10000000
00020}, :retirement=>{:remove_resources=>"yes_without_playbook", :fqname=>"/Service/Generic/StateMachines/GenericLifecycle/Retire_Basic_Resource"}}, :dialog=>{"dialog_credential"=>"100000000
0008", "dialog_hosts"=>"localhost"}, :provision_job_options=>{"hosts"=>"localhost", "extra_vars"=>{}, "credential"=>"2"}}, display: true, created_at: "2017-07-31 16:34:43", updated_at: "2017
-07-31 16:34:51", evm_owner_id: 1000000000001, miq_group_id: 1000000000002, retired: false, retires_on: nil, retirement_warn: nil, retirement_last_warn: nil, retirement_state: nil, retiremen
t_requester: nil, tenant_id: 1000000000001, ancestry: nil, initiator: "user">>
[----] E, [2017-07-31T16:35:54.406053 #4126:b64c20] ERROR -- : Q-task_id([service_template_provision_task_1000000000047]) <AutomationEngine> MiqAeServiceModelBase.ar_method raised: <MiqExcep
tion::MiqOrchestrationProvisionError>: <Net::ReadTimeout>
[----] E, [2017-07-31T16:35:54.406123 #4126:b64c20] ERROR -- : Q-task_id([service_template_provision_task_1000000000047]) <AutomationEngine> /opt/rh/cfme-gemset/gems/ansible_tower_client-0.1
2.2/lib/ansible_tower_client/api.rb:102:in `rescue in method_missing'
/opt/rh/cfme-gemset/gems/ansible_tower_client-0.12.2/lib/ansible_tower_client/api.rb:87:in `method_missing'
/opt/rh/cfme-gemset/gems/ansible_tower_client-0.12.2/lib/ansible_tower_client/base_models/job_template.rb:5:in `launch'
/var/www/miq/vmdb/app/models/manageiq/providers/ansible_tower/shared/automation_manager/configuration_script.rb:86:in `block in run'
/var/www/miq/vmdb/app/models/mixins/provider_object_mixin.rb:15:in `block in with_provider_object'
/var/www/miq/vmdb/app/models/provider.rb:49:in `with_provider_connection'
/var/www/miq/vmdb/app/models/manageiq/providers/ansible_tower/shared/automation_manager.rb:5:in `with_provider_connection'
/var/www/miq/vmdb/app/models/mixins/provider_object_mixin.rb:12:in `with_provider_object'
/var/www/miq/vmdb/app/models/manageiq/providers/ansible_tower/shared/automation_manager/configuration_script.rb:85:in `run'
/var/www/miq/vmdb/app/models/manageiq/providers/ansible_tower/shared/automation_manager/job.rb:20:in `raw_create_stack'
/var/www/miq/vmdb/app/models/manageiq/providers/ansible_tower/shared/automation_manager/job.rb:15:in `block in create_stack'
/var/www/miq/vmdb/app/models/manageiq/providers/ansible_tower/shared/automation_manager/job.rb:14:in `tap'
/var/www/miq/vmdb/app/models/manageiq/providers/ansible_tower/shared/automation_manager/job.rb:14:in `create_stack'
/var/www/miq/vmdb/app/models/service_ansible_playbook.rb:16:in `execute'
/var/www/miq/vmdb/lib/miq_automation_engine/engine/miq_ae_method_service/miq_ae_service_model_base.rb:266:in `public_send'
/var/www/miq/vmdb/lib/miq_automation_engine/engine/miq_ae_method_service/miq_ae_service_model_base.rb:266:in `block in object_send'
/var/www/miq/vmdb/lib/miq_automation_engine/engine/miq_ae_method_service/miq_ae_service_model_base.rb:285:in `ar_method'
/var/www/miq/vmdb/lib/miq_automation_engine/engine/miq_ae_method_service/miq_ae_service_model_base.rb:295:in `ar_method'
/var/www/miq/vmdb/lib/miq_automation_engine/engine/miq_ae_method_service/miq_ae_service_model_base.rb:264:in `object_send'
/var/www/miq/vmdb/lib/miq_automation_engine/engine/miq_ae_method_service/miq_ae_service_model_base.rb:119:in `block (2 levels) in expose'
/opt/rh/rh-ruby23/root/usr/share/ruby/drb/drb.rb:1624:in `perform_without_block'
/opt/rh/rh-ruby23/root/usr/share/ruby/drb/drb.rb:1584:in `perform'
/opt/rh/rh-ruby23/root/usr/share/ruby/drb/drb.rb:1657:in `block (2 levels) in main_loop'
/opt/rh/rh-ruby23/root/usr/share/ruby/drb/drb.rb:1653:in `loop'
.......