Bug 1574879

Summary: Fail to upgrade ha containerized ocp to v3.9 due to 3.8 rpm version is different from 3.8 image version
Product: OpenShift Container Platform Reporter: liujia <jiajliu>
Component: Cluster Version OperatorAssignee: Scott Dodson <sdodson>
Status: CLOSED DEFERRED QA Contact: liujia <jiajliu>
Severity: medium Docs Contact:
Priority: medium    
Version: 3.9.0CC: aos-bugs, jokerman, mgugino, mmccomas, wmeng
Target Milestone: ---   
Target Release: 3.9.z   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2018-11-29 21:20:16 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description liujia 2018-05-04 09:04:49 UTC
Description of problem:
Run upgrade against containerized ocp cluster on rhel. When v3.8 rpm version is different with v3.8 image version. Upgrade will fail because the check on masters(not first master) will fail. 

This check(3.8 rpm version should be the same with 3.8 image version) should be useless for an container upgrade from 3.7 to 3.9 because 3.8 pkgs will be not installed/updated.

And user can not workaround through setting openshift_image_tag because this variable is only for targeted version(3.9) but not for middle-stage version(3.8).

TASK [openshift_version : Fail if rpm version and docker image version are different] ***
task path: /usr/share/ansible/openshift-ansible/roles/openshift_version/tasks/masters_and_nodes.yml:7
fatal: [qe-jliu-ha37c-master-container-2]: FAILED! => {"changed": false, "msg": "OCP rpm version 3.8.37 is different from OCP image version 3.8.36"}
fatal: [qe-jliu-ha37c-master-container-3]: FAILED! => {"changed": false, "msg": "OCP rpm version 3.8.37 is different from OCP image version 3.8.36"}

# docker images |grep v3.8
registry.reg-aws.openshift.com:443/openshift3/ose           v3.8                95094e89e1b5        2 weeks ago         1.287 GB
registry.reg-aws.openshift.com:443/openshift3/ose           v3.8.36             95094e89e1b5        2 weeks ago         1.287 GB
registry.reg-aws.openshift.com:443/openshift3/ose           v3.8.37             b38fd31ea4ea        3 weeks ago         1.287 GB

The check for 3.8 image use tag v3.8 and v3.8 image is not the same with latest v3.8.37. 

# cat playbooks/init/version.yml
# NOTE: These steps should only be run against masters and nodes.
- name: Ensure the requested version packages are available.
  hosts: "{{ l_openshift_version_check_hosts | default('oo_nodes_to_config:oo_masters_to_config:!oo_first_master') }}"
  tasks:
  - include_role:
      name: openshift_version
      tasks_from: masters_and_nodes.yml


Version-Release number of the following components:
openshift-ansible-3.9.27-1.git.0.52e35b5.el7.noarch

How reproducible:
always

Steps to Reproduce:
1. HA container install ocp on rhel.
2. Enable ose 3.8 and 3.9 repos.(Ensure the rpm version in 3.8 repo is different from 3.8 image version with tag v3.8)
3. Upgrade above ocp

Actual results:
Upgrade failed.

Expected results:
upgrade succeed.

Additional info:
Please attach logs from ansible-playbook with the -vvv flag

Comment 1 Michael Gugino 2018-11-29 21:20:16 UTC
There appear to be no active cases related to this bug. As such we're closing this bug in order to focus on bugs that are still tied to active customer cases. Please re-open this bug if you feel it was closed in error or a new active case is attached.