Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

Bug 1721406

Summary: Fail to update cluster to specified version with openshift_pkg_version/openshift_image_tag
Product: OpenShift Container Platform Reporter: liujia <jiajliu>
Component: InstallerAssignee: Russell Teague <rteague>
Installer sub component: openshift-ansible QA Contact: Johnny Liu <jialiu>
Status: CLOSED WONTFIX Docs Contact:
Severity: medium    
Priority: medium CC: aos-bugs, jokerman, mmccomas
Version: 3.10.0   
Target Milestone: ---   
Target Release: 3.10.z   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2019-10-22 19:09:08 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description liujia 2019-06-18 08:42:24 UTC 
Description of problem:

Run upgrade with openshift_pkg_version and openshift_image_tag specified to non-latest version(3.10.148) when both non-latest(3.10.148) and latest repo(3.10.149) enabled. Upgrade succeed, but the cluster is updated to an mixed version status.

[root@ip-172-18-1-45 ~]# oc version
oc v3.10.149
kubernetes v1.10.0+b81c8f8
features: Basic-Auth GSSAPI Kerberos SPNEGO

Server https://ip-172-18-1-45.ec2.internal:8443
openshift v3.10.148
kubernetes v1.10.0+b81c8f8

[root@ip-172-18-1-45 ~]# rpm -qa|grep openshift
atomic-openshift-hyperkube-3.10.149-1.git.0.05de590.el7.x86_64
atomic-openshift-clients-3.10.149-1.git.0.05de590.el7.x86_64
atomic-openshift-docker-excluder-3.10.148-1.git.0.45a3c37.el7.noarch
atomic-openshift-3.10.149-1.git.0.05de590.el7.x86_64
atomic-openshift-excluder-3.10.148-1.git.0.45a3c37.el7.noarch
atomic-openshift-node-3.10.149-1.git.0.05de590.el7.x86_64

[root@ip-172-18-1-45 ~]# docker images
REPOSITORY                                                        TAG                 IMAGE ID            CREATED             SIZE
registry.reg-aws.openshift.com:443/openshift3/ose-deployer        v3.9.83             05455cd23f55        2 days ago          1.27 GB
registry.reg-aws.openshift.com:443/openshift3/ose-web-console     v3.9.83             c89404952fe5        2 days ago          498 MB
registry.reg-aws.openshift.com:443/openshift3/registry-console    v3.9                59195557a9b4        2 days ago          247 MB
registry.reg-aws.openshift.com:443/openshift3/ose-pod             v3.9.83             5d2133f265a3        2 days ago          229 MB
registry.reg-aws.openshift.com:443/openshift3/ose-pod             v3.10.149           593458bfe3a3        6 days ago          229 MB
registry.reg-aws.openshift.com:443/openshift3/ose-node            v3.10.148           446ba368ae99        10 days ago         1.3 GB
registry.reg-aws.openshift.com:443/openshift3/ose-deployer        v3.10.148           93ef0362c097        10 days ago         807 MB
registry.reg-aws.openshift.com:443/openshift3/ose-control-plane   v3.10.148           1baecca0b2cf        10 days ago         807 MB
registry.reg-aws.openshift.com:443/openshift3/ose-web-console     v3.10.148           42180018ddb5        10 days ago         334 MB
registry.reg-aws.openshift.com:443/openshift3/registry-console    v3.10.148           b87801a6e631        10 days ago         246 MB
registry.reg-aws.openshift.com:443/openshift3/ose-pod             v3.10.148           d9b9d26e239e        10 days ago         229 MB
registry.reg-aws.openshift.com:443/rhel7/etcd                     3.2.22              bb2f1d4dd3a7        12 months ago       256 MB

=====================================================
Should be the dependency casue the issue in task [openshift_node : download new node packages].
# yum install --downloadonly atomic-openshift-3.10.148 atomic-openshift-hyperkube-3.10.148 atomic-openshift-node-3.10.148 atomic-openshift-clients-3.10.148
Loaded plugins: amazon-id, search-disabled-repos
aos                                                                                                                                                   | 1.3 kB  00:00:00     
aos310                                                                                                                                                | 1.3 kB  00:00:00     
aos_addon3_10                                                                                                                                         | 1.3 kB  00:00:00     
docker_previous                                                                                                                                       | 2.9 kB  00:00:00     
dockernext                                                                                                                                            | 2.9 kB  00:00:00     
oso-rhui-rhel-server-extras                                                                                                                           | 3.6 kB  00:00:00     
oso-rhui-rhel-server-fast-datapath                                                                                                                    | 3.6 kB  00:00:00     
oso-rhui-rhel-server-releases                                                                                                                         | 3.6 kB  00:00:00     
prerelease-extra                                                                                                                                      | 3.0 kB  00:00:00     
prerelease-rhel7                                                                                                                                      | 2.8 kB  00:00:00     
(1/14): aos/primary                                                                                                                                   | 147 kB  00:00:00     
(2/14): aos310/primary                                                                                                                                | 128 kB  00:00:00     
(3/14): docker_previous/primary_db                                                                                                                    | 112 kB  00:00:00     
(4/14): aos_addon3_10/primary                                                                                                                         | 128 kB  00:00:00     
(5/14): dockernext/primary_db                                                                                                                         |  41 kB  00:00:00     
(6/14): oso-rhui-rhel-server-fast-datapath/group_gz                                                                                                   |  133 B  00:00:00     
(7/14): oso-rhui-rhel-server-extras/group_gz                                                                                                          |  133 B  00:00:00     
(8/14): prerelease-extra/primary_db                                                                                                                   |  50 kB  00:00:00     
(9/14): oso-rhui-rhel-server-extras/primary_db                                                                                                        | 1.0 MB  00:00:00     
(10/14): prerelease-rhel7/group_gz                                                                                                                    | 103 kB  00:00:00     
(11/14): oso-rhui-rhel-server-fast-datapath/primary_db                                                                                                | 434 kB  00:00:00     
(12/14): oso-rhui-rhel-server-releases/group_gz                                                                                                       | 151 kB  00:00:00     
(13/14): prerelease-rhel7/primary                                                                                                                     | 2.0 MB  00:00:00     
(14/14): oso-rhui-rhel-server-releases/primary_db                                                                                                     | 108 MB  00:00:02     
aos                                                                                                                                                                  558/558
aos310                                                                                                                                                               581/581
aos_addon3_10                                                                                                                                                        581/581
prerelease-rhel7                                                                                                                                                   5228/5228
Resolving Dependencies
--> Running transaction check
---> Package atomic-openshift.x86_64 0:3.9.83-1.git.0.78bd535.el7 will be updated
--> Processing Dependency: atomic-openshift = 3.9.83-1.git.0.78bd535.el7 for package: atomic-openshift-master-3.9.83-1.git.0.78bd535.el7.x86_64
---> Package atomic-openshift.x86_64 0:3.10.148-1.git.0.45a3c37.el7 will be an update
---> Package atomic-openshift-clients.x86_64 0:3.9.83-1.git.0.78bd535.el7 will be updated
---> Package atomic-openshift-clients.x86_64 0:3.10.148-1.git.0.45a3c37.el7 will be an update
---> Package atomic-openshift-hyperkube.x86_64 0:3.10.148-1.git.0.45a3c37.el7 will be installed
---> Package atomic-openshift-node.x86_64 0:3.9.83-1.git.0.78bd535.el7 will be updated
--> Processing Dependency: atomic-openshift-node = 3.9.83-1.git.0.78bd535.el7 for package: atomic-openshift-sdn-ovs-3.9.83-1.git.0.78bd535.el7.x86_64
---> Package atomic-openshift-node.x86_64 0:3.10.148-1.git.0.45a3c37.el7 will be an update
--> Running transaction check
---> Package atomic-openshift-master.x86_64 0:3.9.83-1.git.0.78bd535.el7 will be updated
---> Package atomic-openshift-master.x86_64 0:3.10.149-1.git.0.05de590.el7 will be an update
--> Processing Dependency: atomic-openshift = 3.10.149-1.git.0.05de590.el7 for package: atomic-openshift-master-3.10.149-1.git.0.05de590.el7.x86_64
aos310/filelists                                                                                                                                      | 665 kB  00:00:00     
---> Package atomic-openshift-sdn-ovs.x86_64 0:3.9.83-1.git.0.78bd535.el7 will be updated
---> Package atomic-openshift-sdn-ovs.x86_64 0:3.10.149-1.git.0.05de590.el7 will be an update
--> Running transaction check
---> Package atomic-openshift.x86_64 0:3.9.83-1.git.0.78bd535.el7 will be updated
---> Package atomic-openshift.x86_64 0:3.9.83-1.git.0.78bd535.el7 will be updated
---> Package atomic-openshift.x86_64 0:3.10.148-1.git.0.45a3c37.el7 will be an update
---> Package atomic-openshift.x86_64 0:3.10.149-1.git.0.05de590.el7 will be an update
--> Processing Dependency: atomic-openshift-clients = 3.10.149-1.git.0.05de590.el7 for package: atomic-openshift-3.10.149-1.git.0.05de590.el7.x86_64
---> Package atomic-openshift-node.x86_64 0:3.9.83-1.git.0.78bd535.el7 will be updated
---> Package atomic-openshift-node.x86_64 0:3.9.83-1.git.0.78bd535.el7 will be updated
---> Package atomic-openshift-node.x86_64 0:3.10.148-1.git.0.45a3c37.el7 will be an update
---> Package atomic-openshift-node.x86_64 0:3.10.149-1.git.0.05de590.el7 will be an update
--> Processing Dependency: atomic-openshift-hyperkube = 3.10.149-1.git.0.05de590.el7 for package: atomic-openshift-node-3.10.149-1.git.0.05de590.el7.x86_64
--> Running transaction check
---> Package atomic-openshift-clients.x86_64 0:3.9.83-1.git.0.78bd535.el7 will be updated
---> Package atomic-openshift-clients.x86_64 0:3.9.83-1.git.0.78bd535.el7 will be updated
---> Package atomic-openshift-clients.x86_64 0:3.10.148-1.git.0.45a3c37.el7 will be an update
---> Package atomic-openshift-clients.x86_64 0:3.10.149-1.git.0.05de590.el7 will be an update
---> Package atomic-openshift-hyperkube.x86_64 0:3.10.148-1.git.0.45a3c37.el7 will be installed
---> Package atomic-openshift-hyperkube.x86_64 0:3.10.149-1.git.0.05de590.el7 will be installed
--> Finished Dependency Resolution

Dependencies Resolved

=============================================================================================================================================================================
 Package                                          Arch                         Version                                             Repository                           Size
=============================================================================================================================================================================
Updating:
 atomic-openshift-node                            x86_64                       3.10.149-1.git.0.05de590.el7                        aos_addon3_10                       5.2 M
Installing for dependencies:
 atomic-openshift-hyperkube                       x86_64                       3.10.149-1.git.0.05de590.el7                        aos_addon3_10                        33 M
Updating for dependencies:
 atomic-openshift                                 x86_64                       3.10.149-1.git.0.05de590.el7                        aos_addon3_10                        39 M
 atomic-openshift-clients                         x86_64                       3.10.149-1.git.0.05de590.el7                        aos_addon3_10                        20 M
 atomic-openshift-master                          x86_64                       3.10.149-1.git.0.05de590.el7                        aos_addon3_10                       960 k
 atomic-openshift-sdn-ovs                         x86_64                       3.10.149-1.git.0.05de590.el7                        aos_addon3_10                       4.6 M

Transaction Summary
=============================================================================================================================================================================
Install             ( 1 Dependent package)
Upgrade  1 Package  (+4 Dependent packages)

Total download size: 103 M
Background downloading packages, then exiting:
Delta RPMs disabled because /usr/bin/applydeltarpm not installed.
(1/6): atomic-openshift-clients-3.10.149-1.git.0.05de590.el7.x86_64.rpm                                                                               |  20 MB  00:00:00     
(2/6): atomic-openshift-3.10.149-1.git.0.05de590.el7.x86_64.rpm                                                                                       |  39 MB  00:00:01     
(3/6): atomic-openshift-master-3.10.149-1.git.0.05de590.el7.x86_64.rpm                                                                                | 960 kB  00:00:00     
(4/6): atomic-openshift-hyperkube-3.10.149-1.git.0.05de590.el7.x86_64.rpm                                                                             |  33 MB  00:00:00     
(5/6): atomic-openshift-node-3.10.149-1.git.0.05de590.el7.x86_64.rpm                                                                                  | 5.2 MB  00:00:00     
(6/6): atomic-openshift-sdn-ovs-3.10.149-1.git.0.05de590.el7.x86_64.rpm                                                                               | 4.6 MB  00:00:00     
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Total                                                                                                                                         65 MB/s | 103 MB  00:00:01     
exiting because "Download Only" specified



Version-Release number of the following components:
ansible-2.4.6.0-1.el7ae.noarch
openshift-ansible-3.10.149-1.git.0.eb0262c.el7.noarch

How reproducible:
always

Steps to Reproduce:
1.Install ocp v3.9
2. Enable both v3.10.148 and latest v3.10.149 repos on the hosts
3. Specify 
openshift_release=v3.10
openshift_pkg_version=-3.10.148
openshift_image_tag=v3.10.149
4. Run upgrade
/usr/share/ansible/openshift-ansible/playbooks/openshift-master/openshift_node_group.yml
/usr/share/ansible/openshift-ansible/playbooks/byo/openshift-cluster/upgrades/v3_10/upgrade.yml

Actual results:
Cluster is updated to a unexpected version.

Expected results:
Cluster should be updated to specified version.

Additional info:
Please attach logs from ansible-playbook with the -vvv flag
Comment 2 Russell Teague 2019-10-22 19:09:08 UTC 
This issue is only present for 3.9 to 3.10 upgrades due to the package dependency resolution including atomic-openshift-master, which is not applicable to 3.10.  During the upgrade, the atomic-openshift-master package is removed and future upgrades will honor the requested package version.  This issue is not present in upgrades from 3.10 to 3.11.  Due to the limited impact of one time 3.9 to 3.10 upgrade, we will not fix this issue.  If a customer has a specific impact when upgrading while trying to get to 3.11, please open a new bug.