Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

Bug 1542855

Summary: upgrade control plane failed at play "Ensure the requested version packages are available" when specify openshift_additional_repos
Product: OpenShift Container Platform Reporter: Weihua Meng <wmeng>
Component: Cluster Version OperatorAssignee: Michael Gugino <mgugino>
Status: CLOSED ERRATA QA Contact: Weihua Meng <wmeng>
Severity: medium Docs Contact:
Priority: medium    
Version: 3.9.0CC: aos-bugs, jokerman, mmccomas, rteague, sdodson, wmeng
Target Milestone: ---   
Target Release: 3.9.0   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2018-03-28 14:26:32 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Weihua Meng 2018-02-07 07:57:09 UTC
Description of problem:
upgrade control plane failed at play "Ensure the requested version packages are available" when specify openshift_additional_repos in hosts file.

The check is targetedd for all hosts(masters+nodes) in task [openshift_version : Get available atomic-openshift version], but only masters group ran task [Update repos on upgrade hosts] during upgrade master phase.

Version-Release number of the following components:
openshift-ansible-3.9.0-0.38.0.git.0.57e1184.el7.noarch
ansible-2.4.2.0-2.el7.noarch

How reproducible:
Always

Steps to Reproduce:
1. specify openshift_additional_repos=[...] in hosts file
2. upgrade OCP v3.7 to v3.9 by openshift-ansible
# ansible-playbook /usr/share/ansible/openshift-ansible/playbooks/byo/openshift-cluster/upgrades/v3_9/upgrade_control_plane.yml

Actual results:

TASK [include_role] ***********************************************************************************************************************************************************************************************
task path: /usr/share/ansible/openshift-ansible/playbooks/init/version.yml:31

TASK [openshift_version : Check openshift_version for rpm installation] *******************************************************************************************************************************************
task path: /usr/share/ansible/openshift-ansible/roles/openshift_version/tasks/masters_and_nodes.yml:5
included: /usr/share/ansible/openshift-ansible/roles/openshift_version/tasks/check_available_rpms.yml for hostxxx.redhat.com, hostxxx.redhat.com, hostxxx.redhat.com, hostxxx.redhat.com, hostxxx.redhat.com, hostxxx.redhat.com

TASK [openshift_version : Get available atomic-openshift version]

TASK [openshift_version : fail] ***********************************************************************************************************************************************************************************
task path: /usr/share/ansible/openshift-ansible/roles/openshift_version/tasks/check_available_rpms.yml:8
skipping: [hostxxx.redhat.com] => {
    "changed": false, 
    "skip_reason": "Conditional result was False"
}
skipping: [hostxxx.redhat.com] => {
    "changed": false, 
    "skip_reason": "Conditional result was False"
}
fatal: [hostxxx.redhat.com]: FAILED! => {
    "changed": false, 
    "msg": "Package atomic-openshift not found"
}
fatal: [hostxxx.redhat.com]: FAILED! => {
    "changed": false, 
    "msg": "Package atomic-openshift not found"
}
fatal: [hostxxx.redhat.com]: FAILED! => {
    "changed": false, 
    "msg": "Package atomic-openshift not found"
}
fatal: [hostxxx.redhat.com]: FAILED! => {
    "changed": false, 
    "msg": "Package atomic-openshift not found"
}


Failure summary:


  1. Hosts:    hostxxx.redhat.com, hostxxx.redhat.com, hostxxx.redhat.com, hostxxx.redhat.com
     Play:     Ensure the requested version packages are available.
     Task:     openshift_version : fail
     Message:  Package atomic-openshift not found


Expected results:
Upgrade succeeds

Comment 4 Michael Gugino 2018-02-08 20:12:22 UTC
I don't believe the situation as reported is possible.

0 non-master nodes should have tasks run against them during upgrade_control_plane.

The logging provided shows the following playflow:

'Examine etcd serving certificate SAN'

Followed by

'Initialization Checkpoint Start'

This is not possible.

I suspect the installer host is corrupted in some way.

Comment 7 Weihua Meng 2018-02-09 02:38:40 UTC
Hi Michael
I set up a new ansible host, did the same thing with openshift-ansible-3.9.0-0.38.0, with non-ha this time.
Meet the same issue.
inventory file and upgrade log attached.

Comment 8 Michael Gugino 2018-02-09 15:38:38 UTC
@Weihua,

Thanks for uploading the new log.  It looks like a change was made recently, it was really hard to find.  Thank you for your patience.

PR Created: https://github.com/openshift/openshift-ansible/pull/7088

Comment 9 Weihua Meng 2018-02-17 02:16:30 UTC
Fixed.
openshift-ansible-3.9.0-0.45.0.git.0.05f6826.el7.noarch

Comment 11 Weihua Meng 2018-02-21 02:26:48 UTC
Fixed.
openshift-ansible-3.9.0-0.45.0.git.0.05f6826.el7.noarch

Comment 14 errata-xmlrpc 2018-03-28 14:26:32 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2018:0489