Bug 2053404

Created attachment 1860549 [details]
path proposal for fix in current version

Description of problem:
'tripleo_nodes_validation' role fails on tasks with ping if any of them fails on first try. Ansible task code contains 'retries'+'delay' parameters, but they are ignored as there is no 'until' statement.

https://docs.ansible.com/ansible/latest/user_guide/playbooks_loops.html#retrying-a-task-until-a-condition-is-met
...
You must set the `until` parameter if you want a task to retry. If until is not defined, the value for the `retries` parameter is forced to `1`.
...

Version-Release number of selected component (if applicable):
tripleo-ansible-0.7.1-2.20210603175844.el8ost.9
ansible-2.9.27-1.el8ae.noarch

How reproducible:
Always when there is compute node that fails on first attempt to ping any of the controller nodes or gateway.

Steps to Reproduce:
1. deploy the overcloud*

Actual results:
Deployment fails at step 'Check controllers availability' for some of the compute nodes.

Expected results:
Deployment should retry task 'Check controllers availability' up to 10 times with delay 60 seconds as defined in tripleo_ansible/roles/tripleo_nodes_validation/tasks/main.yml playbook task and only fail if ping fails continuously.

Additional info:
I have noticed this in one deployment where we have a nodes that will manage to ping most of the IPs from the task in tripleo_ansible/roles/tripleo_nodes_validation/tasks/main.yml but fail (on first attempt) at one of them. I have assumed that it maybe needed somehow longer timeout, but then I have noticed on the compute node that once this tasks fails it is never retried and whole deploy just fails.

I have added the 'until' and 'register' parameters to task and re-running the deployment I have noticed that now the one failing ping was retried and the deployment has progressed.

Therefore I'm attaching a simple patch (`tripleo-ansible-fix-ping-0.7.1-2.20210603175844.el8ost.9.patch`) that fixes the 2 tasks (ping GW, ping controller nodes) in way that if they fail on first attempt they are retried as many times as defined by 'retries' parameter (10 times).

To see the effect you may try out following ansible playbook and try uncommenting the 'until' and 'register' parameters to see the difference.
~~~
---
- hosts: localhost
  vars:
    test_ips:
      - 192.168.4.1
      - 192.168.4.2
      - 192.168.4.3
  tasks:
    - name: some ping
      command: "ping -w 10 -c 1 {{ controller }}"
      retries: 10
      delay: 60
      loop_control:
        loop_var: controller
      loop: "{{ test_ips }}"
      changed_when: false
      #register: result
      #until: result is not failed
~~~
If the 'ping' fails persistently the whole task fails as before, therefore the change proposed here should affect only scenarios where we failed to receive reply temporarily.