Bug 1426070

Summary: excluders are not upgraded to corresponding version when upgrade ocp
Product: OpenShift Container Platform Reporter: liujia <jiajliu>
Component: Cluster Version OperatorAssignee: Jan Chaloupka <jchaloup>
Status: CLOSED ERRATA QA Contact: liujia <jiajliu>
Severity: medium Docs Contact:
Priority: medium    
Version: 3.5.0CC: aos-bugs, jiajliu, jokerman, mmccomas
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: openshift-ansible-3.5.27-1.git.0.08cda99.el7 Doc Type: No Doc Update
Doc Text:
Cause: excluders not updated to the latest rpm available Consequence: excluders are not up-to-date Fix: when installing/updating excluders, install the latest available Result: excluders are up-to-date
Story Points: ---
Clone Of: Environment:
Last Closed: 2017-04-11 21:18:58 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description liujia 2017-02-23 07:42:31 UTC
Description of problem:
Upgrade ocp3.4 to ocp3.5 against rpm installed cluster, which installed and enabled atomic-openshift-excluder and atomic-openshift-docker-excluder. After upgrade succesfully, both of excluders are not upgraded together.

==before upgrade
openshift-119.x.x.x | SUCCESS | rc=0 >>
atomic-openshift-excluder-3.4.1.8-1.git.0.46178be.el7.noarch
atomic-openshift-docker-excluder-3.4.1.8-1.git.0.46178be.el7.noarch

openshift-129.x.x.x | SUCCESS | rc=0 >>
atomic-openshift-docker-excluder-3.4.1.8-1.git.0.46178be.el7.noarch
atomic-openshift-excluder-3.4.1.8-1.git.0.46178be.el7.noarch

==after upgrade
openshift-119.x.x.x | SUCCESS | rc=0 >>
atomic-openshift-excluder-3.4.1.8-1.git.0.46178be.el7.noarch
atomic-openshift-docker-excluder-3.4.1.8-1.git.0.46178be.el7.noarch

openshift-129.x.x.x | SUCCESS | rc=0 >>
atomic-openshift-docker-excluder-3.4.1.8-1.git.0.46178be.el7.noarch
atomic-openshift-excluder-3.4.1.8-1.git.0.46178be.el7.noarch


openshift v3.5.0.32-1+4f84c83
kubernetes v1.5.2+43a9be4
etcd 3.1.0


Version-Release number of selected component (if applicable):
atomic-openshift-utils-3.5.13-1.git.0.562e91d.el7.noarch

How reproducible:
always

Steps to Reproduce:
1. rpm install ocp 3.4
2. install atomic-openshift-excluder and atomic-openshift-docker-excluder(enabled by default)
3. prepare 3.5 ose repo and upgrade ocp3.4 to 3.5

Actual results:
Both of excluders are not upgraded together.

Expected results:
Excluders should be upgraded.

Additional info:
It seems that there is no "latest" command in related tasks. 
<--snip-->
- name: Update to latest excluder packages
  package:
    name: "{{ openshift.common.service_type }}-excluder"
  when:
  - "{{ openshift_excluder_installed.installed_versions | default([]) | length > 0 }}"
  - not openshift.common.is_containerized | bool

- name: Update to the latest docker-excluder packages
  package:
    name: "{{ openshift.common.service_type }}-docker-excluder"
  when:
  - "{{ docker_excluder_installed.installed_versions | default([]) | length > 0 }}"
  - not openshift.common.is_containerized | bool
<--snip-->

Comment 1 Jan Chaloupka 2017-02-27 14:59:39 UTC
I was able to reproduce it. And yes, both tasks are missing "state: latest".

Comment 2 Jan Chaloupka 2017-02-27 15:11:17 UTC
Upstream fix: https://github.com/openshift/openshift-ansible/pull/3502

Fixed. Tested. Rpms after upgrade:

atomic-openshift-excluder-3.5.0.34-1.git.0.9bd77cf.el7.noarch
atomic-openshift-docker-excluder-3.5.0.34-1.git.0.9bd77cf.el7.noarch

Comment 4 liujia 2017-03-08 06:09:10 UTC
blocked verify by bug1430231

Comment 5 Jan Chaloupka 2017-03-08 11:44:23 UTC
Upstream PR: https://github.com/openshift/openshift-ansible/pull/3529

When the upgrade is run, an excluder is updated to the latest available version there is in available enabled repos. Before the excluders are even use, an excluder version is validated and check it if is not higher than the upgrade target version. So if e.g. the current version of OCP is 3.4 and the upgrade target is 3.5, the upgrade fails if 3.6 or higher version of excluders is available.

Comment 6 Anping Li 2017-03-08 14:10:18 UTC
The excluder packages still weren't upgraded to v3.5.x. I was using atomic-openshift-utils-3.5.26-1.git.0.d88f597.el7.noarch

[root@openshift-182 ~]# rpm -qa|grep atomic
atomic-openshift-3.5.0.45-1.git.0.4a06220.el7.x86_64
atomic-openshift-clients-3.5.0.45-1.git.0.4a06220.el7.x86_64
atomic-openshift-node-3.5.0.45-1.git.0.4a06220.el7.x86_64
atomic-openshift-master-3.5.0.45-1.git.0.4a06220.el7.x86_64
atomic-openshift-docker-excluder-3.4.1.10-1.git.0.c96aed3.el7.noarch
atomic-openshift-excluder-3.4.1.10-1.git.0.c96aed3.el7.noarch
tuned-profiles-atomic-openshift-node-3.5.0.45-1.git.0.4a06220.el7.x86_64
atomic-openshift-sdn-ovs-3.5.0.45-1.git.0.4a06220.el7.x86_64


[root@openshift-182 ~]# yum search atomic-openshift-excluder --showdup
Loaded plugins: product-id, search-disabled-repos, subscription-manager
This system is not registered to Red Hat Subscription Management. You can use subscription-manager to register.
===================================================================================== N/S matched: atomic-openshift-excluder ======================================================================================
atomic-openshift-excluder-3.4.1.10-1.git.0.c96aed3.el7.noarch : Exclude openshift packages from updates
atomic-openshift-excluder-3.5.0.45-1.git.0.4a06220.el7.noarch : Exclude openshift packages from updates

Comment 7 Scott Dodson 2017-03-08 17:27:56 UTC
Fixed in openshift-ansible-3.5.27-1.git.0.08cda99.el7

Comment 8 Anping Li 2017-03-09 06:05:12 UTC
The excluder was upgraded to correct version  with openshift-ansible-3.5.27-1.git.0.08cda99.el7


[root@openshift-214 ~]# rpm -qa|grep atomic-
atomic-openshift-docker-excluder-3.5.0.49-1.git.0.c8e072a.el7.noarch
atomic-openshift-3.5.0.49-1.git.0.c8e072a.el7.x86_64
atomic-openshift-master-3.5.0.49-1.git.0.c8e072a.el7.x86_64
atomic-openshift-excluder-3.5.0.49-1.git.0.c8e072a.el7.noarch
atomic-openshift-clients-3.5.0.49-1.git.0.c8e072a.el7.x86_64
tuned-profiles-atomic-openshift-node-3.5.0.49-1.git.0.c8e072a.el7.x86_64
atomic-openshift-sdn-ovs-3.5.0.49-1.git.0.c8e072a.el7.x86_64
atomic-openshift-node-3.5.0.49-1.git.0.c8e072a.el7.x86_64

Comment 9 liujia 2017-03-09 11:03:36 UTC
Version:
atomic-openshift-utils-3.5.28-1.git.0.103513e.el7.noarch

Steps:
1. rpm install ocp 3.4
2. install atomic-openshift-excluder and atomic-openshift-docker-excluder(enabled by default)
3. prepare 3.5 ose repo and upgrade ocp3.4 to 3.5

Result:
before upgrade:
openshift-118.x.x.x | SUCCESS | rc=0 >>
atomic-openshift-docker-excluder-3.4.1.10-1.git.0.c96aed3.el7.noarch
atomic-openshift-excluder-3.4.1.10-1.git.0.c96aed3.el7.noarch

openshift-119.x.x.x | SUCCESS | rc=0 >>
atomic-openshift-docker-excluder-3.4.1.10-1.git.0.c96aed3.el7.noarch
atomic-openshift-excluder-3.4.1.10-1.git.0.c96aed3.el7.noarch

after upgrade:
openshift-118.x.x.x | SUCCESS | rc=0 >>
atomic-openshift-docker-excluder-3.4.1.10-1.git.0.c96aed3.el7.noarch
atomic-openshift-excluder-3.5.0.49-1.git.0.c8e072a.el7.noarch

openshift-119.x.x.x | SUCCESS | rc=0 >>
atomic-openshift-docker-excluder-3.4.1.10-1.git.0.c96aed3.el7.noarch
atomic-openshift-excluder-3.5.0.49-1.git.0.c8e072a.el7.noarch

Atomic-openshift-docker-excluder is not updated.

Comment 10 Scott Dodson 2017-03-09 15:09:17 UTC
Upstream PR: https://github.com/openshift/openshift-ansible/pull/3607

Comment 11 Jan Chaloupka 2017-03-09 15:23:23 UTC
liujia, there is no difference between 3.5.27-1 and 3.5.28-1 wrt. excluders. Can you share your inventory file?

Comment 13 Jan Chaloupka 2017-03-10 16:14:48 UTC
I am not able to reproduce it with the latest commit in master branch with [1] applied. I will run another installation and upgrade scenario later on again once [1] and its dependent PR [2] are merged.

[1] https://github.com/openshift/openshift-ansible/pull/3620
[2] https://github.com/openshift/openshift-ansible/pull/3610

Comment 14 Scott Dodson 2017-03-14 00:14:28 UTC
additional changes merged into openshift-ansible-3.5.31, ON_QA

Comment 15 liujia 2017-03-14 02:30:21 UTC
Version:
atomic-openshift-utils-3.5.32-1.git.0.42cf266.el7.noarch

Steps:
1. rpm install ocp 3.4
2. install atomic-openshift-excluder and atomic-openshift-docker-excluder(enabled by default)
3. prepare 3.5 ose repo and upgrade ocp3.4 to 3.5

Result:

before upgrade
# rpm -qa | grep excluder
atomic-openshift-docker-excluder-3.4.1.10-1.git.0.c96aed3.el7.noarch
atomic-openshift-excluder-3.4.1.10-1.git.0.c96aed3.el7.noarch

after upgrade
# rpm -qa | grep excluder
atomic-openshift-excluder-3.5.0.51-1.git.0.d07622a.el7.noarch
atomic-openshift-docker-excluder-3.4.1.10-1.git.0.c96aed3.el7.noarch

atomic-openshift-docker-excluder is still not upgraded to corresponding version.

Comment 17 liujia 2017-03-21 06:05:25 UTC
Version:
atomic-openshift-utils-3.5.39-1.git.0.cb12cdb.el7.noarch

Steps:
1. rpm install ocp 3.4
2. install atomic-openshift-excluder and atomic-openshift-docker-excluder(enabled by default)
3. prepare 3.5 ose repo and upgrade ocp3.4 to 3.5

Result:
before upgrade
# rpm -qa | grep excluder
atomic-openshift-docker-excluder-3.4.1.10-1.git.0.c96aed3.el7.noarch
atomic-openshift-excluder-3.4.1.10-1.git.0.c96aed3.el7.noarch

after upgrade
# rpm -qa | grep excluder
atomic-openshift-excluder-3.5.0.55-1.git.0.a552679.el7.noarch
atomic-openshift-docker-excluder-3.5.0.55-1.git.0.a552679.el7.noarch

atomic-openshift-docker-excluder is upgraded to corresponding version.