Bug 1377619

I was able to reproduce this error in a 1 master 1 node setup. I renamed redhat.repo to redhat.repo.orig after the playbooks started running.

> Play 22/29 (Configure nodes)
> ......................................................................
> fatal: [n01.example.com]: FAILED! => {"changed": false, "failed": true,
> "msg": "No package matching 'atomic-openshift-3.3.0.35' found available,
> installed or updated", "rc": 126,
> "results": ["No package matching 'atomic-openshift-3.3.0.35' found available, installed or updated"]}
> .....................................................................................................

Which was then followed by 

> Play 23/29 (Additional node config)

That play finished and then we get to the "Set node schedulability" play. After a few tasks we see the "Wait for Node Registration" task run and retry over and over.

This should be failing much earlier. I'll see what I can do to make that happen. I'm honestly confused as to why it **isn't** causing the install to fail already. Intuitively you'd think it would be. Our task that it failed on doesn't even have "ignore_errors: yes" set on it. So why does the playbook continue running?

I can see after the schedulability task fails that no more tasks run on my example **node**. However, tasks still run on the master at that point. Per the documentation [1] this is normal behavior, "Generally playbooks will stop executing any more steps on a host that has a failure" note that it says "on a host", not "on all hosts".

However, we're in luck. Ansible does provide us with some nobs we can frob to tune the error handling behavior. For example there are the "max_fail_percentage" [2] and "any_errors_fatal" [3] options.

I think the "max fail %" option isn't quite what we're looking for. But the "any errors fatal" option may be more relevant. I'm going to try playing around with that and see how the behavior changes.


[1] http://docs.ansible.com/ansible/playbooks_error_handling.html#ignoring-failed-commands 
[2] http://docs.ansible.com/ansible/playbooks_delegation.html#maximum-failure-percentage
[3] http://docs.ansible.com/ansible/playbooks_error_handling.html#aborting-the-play

https://github.com/openshift/openshift-ansible/pull/2666

After adding the "any_errors_fatal" param to the node configuration plays the installation aborts immediately.

> Play 22/29 (Configure nodes)
> ......................................................................
> fatal: [n01.example.com]: FAILED! => {"changed": false, "failed": true, "msg":
> "No package matching 'atomic-openshift-3.3.0.35' found available
> , installed or updated", "rc": 126, "results": ["No package matching 
> 'atomic-openshift-3.3.0.35' found available, installed or updated"]}
>
> NO MORE HOSTS LEFT
> *************************************************************
>
> localhost                  : ok=14   changed=9    unreachable=0    failed=0
> m01.example.com            : ok=343  changed=28   unreachable=0    failed=0
> n01.example.com            : ok=78   changed=3    unreachable=0    failed=1

Proposed fix: https://github.com/openshift/openshift-ansible/pull/2827

Code has been merged.

Verified with version openshift-ansible-3.4.43-1.git.0.a9dbe87.el7.noarch.

Prepare two host, remove one host's yum repo file to make sure installation will be failed. Installation stop when got error.

[root@ansible ~]# ansible-playbook -i hosts -v /usr/share/ansible/openshift-ansible/playbooks/byo/config.yml
...
TASK [openshift_facts : Ensure yum-utils is installed] *************************
ok: [node.example.com] => {
    "changed": false, 
    "rc": 0, 
    "results": [
        "yum-utils-1.1.31-40.el7.noarch providing yum-utils is already installed"
    ]
}
fatal: [master.example.com]: FAILED! => {
    "changed": false, 
    "failed": true, 
    "rc": 126, 
    "results": [
        "No package matching 'yum-utils' found available, installed or updated"
    ]
}

MSG:

No package matching 'yum-utils' found available, installed or updated
...
PLAY RECAP *********************************************************************
localhost                  : ok=8    changed=0    unreachable=0    failed=0   
node.example.com : ok=10   changed=0    unreachable=0    failed=1   
master.example.com : ok=7    changed=0    unreachable=0    failed=1

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2017:0224