Bug 1456093

Summary: Unable to determine available OpenShift RPM version on dedicated node during upgrade master
Product: OpenShift Container Platform Reporter: liujia <jiajliu>
Component: Cluster Version OperatorAssignee: Jan Chaloupka <jchaloup>
Status: CLOSED ERRATA QA Contact: liujia <jiajliu>
Severity: medium Docs Contact:
Priority: medium    
Version: 3.6.0CC: anli, aos-bugs, jiajliu, jokerman, mmccomas
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: openshift-ansible-3.6.89.1-1 Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2017-08-10 05:26:47 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1440167    

Description liujia 2017-05-27 02:22:08 UTC
Description of problem:
When upgrade ocp3.5 masters with openshift_excluder and docker_excluder enabled on all hosts, upgrade_control_plane playbook will finish with one failed task caused by fatal error that no available openshift package found on dedicated node. 

TASK [Fail when unable to determine available OpenShift RPM version] ***********
task path: /usr/share/ansible/openshift-ansible/playbooks/common/openshift-cluster/upgrades/pre/verify_upgrade_targets.yml:27
skipping: [openshift-119.x.x.x] => {
    "changed": false,
    "skip_reason": "Conditional check failed",
    "skipped": true
}
fatal: [openshift-118.x.x.x]: FAILED! => {
    "changed": false,
    "failed": true,
    "invocation": {
        "module_args": {
            "msg": "Unable to determine available OpenShift RPM version"
        },
        "module_name": "fail"
    }
}

MSG:

Unable to determine available OpenShift RPM version

<--snip-->
PLAY RECAP *********************************************************************
localhost                  : ok=25   changed=0    unreachable=0    failed=0
openshift-118.x.x.x : ok=60   changed=4    unreachable=0    failed=1
openshift-119.x.x.x : ok=515  changed=51   unreachable=0    failed=0


Version-Release number of selected component (if applicable):
atomic-openshift-utils-3.6.80-1.git.0.807fc98.el7.noarch

How reproducible:
always

Steps to Reproduce:
1. install ocp3.5(one master/node + one node)
2. install excluders on all hosts
3. run upgrade_control_plane.yml to upgrade masters first
# ansible-playbook -i hosts /usr/share/ansible/openshift-ansible/playbooks/byo/openshift-cluster/upgrades/v3_6/upgrade_control_plane.yml

Actual results:
Upgrade finished with one failed error on dedicated node.

Expected results:
Upgrade should succeed with no failure.

Additional info:
Disable_master_excluders.yml only run on master host, but verify_upgrade_targets.yml targets on both masters and hosts.

Comment 4 Jan Chaloupka 2017-05-31 10:49:28 UTC
Assuming following inventory file:
```ini
...
[nodes]
openshift-118.lab.eng.nay.redhat.com"
openshift-119.lab.eng.nay.redhat.com

[masters]
openshift-119.lab.eng.nay.redhat.com
...
```

the failure is expected. Unfortunately, playbooks/common/openshift-cluster/upgrades/pre/verify_upgrade_targets.yml checks both masters and nodes for OCP rpm version even if only masters are selected. Given the excluders are disabled only on masters, the OCP version on node can not be determined since the openshift-excluder is still active.

Comment 5 Jan Chaloupka 2017-05-31 13:21:47 UTC
For the same reason `verify_docker_upgrade_targets.yml` will fail as well

Comment 6 Jan Chaloupka 2017-05-31 14:00:21 UTC
Upstream PR https://github.com/openshift/openshift-ansible/pull/4321

Comment 7 Anping Li 2017-06-06 03:26:25 UTC
*** Bug 1458854 has been marked as a duplicate of this bug. ***

Comment 9 liujia 2017-06-09 08:29:08 UTC
Version:
atomic-openshift-utils-3.6.97-1.git.0.bc7d3b8.el7.noarch

Step:
1. install ocp3.5(one master/node + one node)
2. run upgrade_control_plane.yml to upgrade masters first
# ansible-playbook -i hosts /usr/share/ansible/openshift-ansible/playbooks/byo/openshift-cluster/upgrades/v3_6/upgrade_control_plane.yml

Result:
Upgrade master succeed with no failure. Excluders are upgraded only on master host.

Comment 11 errata-xmlrpc 2017-08-10 05:26:47 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2017:1716