Fedora Account System
Red Hat Associate
Red Hat Customer
Description of problem: I am unable to retry an Embedded Ansible playbook method in an Automate state machine. Using the manageiq-automate role to set_retry in the automate_workspace, CFME invokes the playbook method again, however the following error is returned: [----] E, [2018-09-04T13:31:53.779587 #15017:91f118] ERROR -- : Q-task_id([service_template_provision_task_1000000001112]) State=<playbook1> running raised exception: <undefined method `reload' for nil:NilClass> I have expanded the workflow below using the on_entry and on_exit methods to log what is happening. Version-Release number of selected component (if applicable): CFME 5.9.3.4-1 How reproducible: Always Steps to Reproduce: 1. Create a playbook that sets automate_workspace root/ae_result to retry 2. Create a state machine with a single method and add that playbook 3. Create a generic catalog item that executes that state machine 4. Order the catalog item Actual results: -> on_entry ae_state_retries: 0 ae_result: ok -> invoking playbook method passing in correct options/extra_vars Calling playbook.run Setting State Machine Auto Retry Interval: 60 Processed State=[playbook1] with Result=[async_launch] -> playbook runs log available under /var/lib/awx/job_status sets retry using manageiq-automate successful completion -> on_entry ae_state_retries: 1 ae_result: ok -> invoking playbook method no playbook run Processed State=[playbook1] with Result=[retry] -> on_exit ae_state_retries: 2 ae_result: retry -> on_entry ae_state_retries: 2 ae_result: ok -> invoking playbook method no playbook run ERROR => State=<playbook1> running raised exception: <undefined method `reload' for nil:NilClass> -> on_error ae_state_retries: 2 ae_result: error Expected results: -> on_entry ae_state_retries: 0 ae_result: ok -> invoking playbook method passing in correct options/extra_vars Calling playbook.run Setting State Machine Auto Retry Interval: 60 Processed State=[playbook1] with Result=[async_launch] -> playbook runs log available under /var/lib/awx/job_status sets retry using manageiq-automate successful completion -> on_entry ae_state_retries: 1 ae_result: ok -> invoking playbook method no playbook run Processed State=[playbook1] with Result=[retry] -> on_exit ae_state_retries: 2 ae_result: retry -> on_entry ae_state_retries: 2 ae_result: ok -> invoking playbook method passing in correct options/extra_vars Calling playbook.run Setting State Machine Auto Retry Interval: 60 Processed State=[playbook1] with Result=[async_launch] -> keep retrying until max_retries reached -> on_error ae_state_retries: 10 ae_result: error Additional info: Setting ae_result to 'async_launch' instead of 'retry' results in a similar outcome.
Matt, Is there a reproducer environment? Thanks Billy
(In reply to William Fitzgerald from comment #3) > Matt, > > Is there a reproducer environment? > > Thanks > > Billy Hi Billy I have sent you environment information and access details directly Thanks, Matt
Matt, I am seeing the same results on a 5.9.4.5 appliance. Billy
https://github.com/ManageIQ/manageiq-automation_engine/pull/222
New commit detected on ManageIQ/manageiq-automation_engine/master: https://github.com/ManageIQ/manageiq-automation_engine/commit/4eb6a937ed1bda6e6b278fb027292283b847be5f commit 4eb6a937ed1bda6e6b278fb027292283b847be5f Author: Madhu Kanoor <mkanoor> AuthorDate: Thu Sep 6 11:49:55 2018 -0400 Commit: Madhu Kanoor <mkanoor> CommitDate: Thu Sep 6 11:49:55 2018 -0400 Clear stale data from ae_state_data and ae_state_previous https://bugzilla.redhat.com/show_bug.cgi?id=1625047 The ansible playbook was one of the first times we have cared about what is the ae_state_data, the playbook method clears out the key ae_state_data fields regarding the automate_workspace and the taskid that was used to track the playbook on a completion of the playbook (either with 'ok' or 'retry'). Since the stale state data wasn't being cleared the playbook wasn't getting successfully launched on a retry and would bail looking for stale automate workspaces. lib/miq_automation_engine/engine/miq_ae_engine.rb | 2 + spec/engine/miq_ae_state_machine_retry_spec.rb | 13 + 2 files changed, 15 insertions(+)
Satyajit, In case Matt doesn't send you his playbook, here is the one I tested with. https://github.com/billfitzgerald0120/ansible_playbooks/blob/master/set_retry_60_seconds.yml Billy
Hi Satyajit I have uploaded the playbook I used for testing to https://github.com/mattparko/manageiq-playbooks/blob/master/retry.yml It uses the same manageiq-automate role and set_retry task as Billy's example. The only difference is that it only attempts to set the state result to retry for the first 3 runs, after which it exits successfully with the normal "ok" result. This could be helpful for testing both success/failure scenarios, by simply adjusting max number of retries in the state machine. Thanks, Matt
Playbook has an error: In evm.log: ERROR! the role 'syncrou.manageiq-automate' was not found in /var/lib/awx/projects/_7__testing_bz/roles:/etc/ansible/roles:/usr/share/ansible/roles:/var/lib/awx/projects/_7__testing_bz The error appears to have been in '/var/lib/awx/projects/_7__testing_bz/retry.yml': line 14, column 7, but may be elsewhere in the file depending on the exact syntax problem. The offending line appears to be: roles: - syncrou.manageiq-automate ^ here Billy
Satyajit, You could try one of my retry playbooks. I like playbook set_retry_with_input.yml in https://github.com/billfitzgerald0120/ansible_playbooks. 1. Add this to input Parameters (Extra vars): manageiq_validate_certs as false 2. I recommend setting logging output to "Always" 3. I recommend setting Verbosity to 3 You will get more information in the evm.log Adding a screenshot Hope this helps ... Billy
Created attachment 1490691 [details] Ansible logging and certs
With the Help of William followed Verification Steps: 1. Enable Embedded Ansible 2. Add repository - https://github.com/billfitzgerald0120/ansible_playbooks 3. Import Ansible_StateMachine_Set_Retry 4. Enable domain 5. Verify the Playbook method (set_retry_4_times) or use your own playbook 6. Add a dummy dialog 7. Add a catalog 8. Add a new Catalog item (Generic Type) 9. Order service 10. Check automation.log to make sure the playbook retried 3 times and then ended OK. Verified Version: 5.10.0.31.20190108221820_a0968c8