Description of problem: Run upgrade with openshift_pkg_version and openshift_image_tag specified to non-latest version(3.10.148) when both non-latest(3.10.148) and latest repo(3.10.149) enabled. Upgrade succeed, but the cluster is updated to an mixed version status. [root@ip-172-18-1-45 ~]# oc version oc v3.10.149 kubernetes v1.10.0+b81c8f8 features: Basic-Auth GSSAPI Kerberos SPNEGO Server https://ip-172-18-1-45.ec2.internal:8443 openshift v3.10.148 kubernetes v1.10.0+b81c8f8 [root@ip-172-18-1-45 ~]# rpm -qa|grep openshift atomic-openshift-hyperkube-3.10.149-1.git.0.05de590.el7.x86_64 atomic-openshift-clients-3.10.149-1.git.0.05de590.el7.x86_64 atomic-openshift-docker-excluder-3.10.148-1.git.0.45a3c37.el7.noarch atomic-openshift-3.10.149-1.git.0.05de590.el7.x86_64 atomic-openshift-excluder-3.10.148-1.git.0.45a3c37.el7.noarch atomic-openshift-node-3.10.149-1.git.0.05de590.el7.x86_64 [root@ip-172-18-1-45 ~]# docker images REPOSITORY TAG IMAGE ID CREATED SIZE registry.reg-aws.openshift.com:443/openshift3/ose-deployer v3.9.83 05455cd23f55 2 days ago 1.27 GB registry.reg-aws.openshift.com:443/openshift3/ose-web-console v3.9.83 c89404952fe5 2 days ago 498 MB registry.reg-aws.openshift.com:443/openshift3/registry-console v3.9 59195557a9b4 2 days ago 247 MB registry.reg-aws.openshift.com:443/openshift3/ose-pod v3.9.83 5d2133f265a3 2 days ago 229 MB registry.reg-aws.openshift.com:443/openshift3/ose-pod v3.10.149 593458bfe3a3 6 days ago 229 MB registry.reg-aws.openshift.com:443/openshift3/ose-node v3.10.148 446ba368ae99 10 days ago 1.3 GB registry.reg-aws.openshift.com:443/openshift3/ose-deployer v3.10.148 93ef0362c097 10 days ago 807 MB registry.reg-aws.openshift.com:443/openshift3/ose-control-plane v3.10.148 1baecca0b2cf 10 days ago 807 MB registry.reg-aws.openshift.com:443/openshift3/ose-web-console v3.10.148 42180018ddb5 10 days ago 334 MB registry.reg-aws.openshift.com:443/openshift3/registry-console v3.10.148 b87801a6e631 10 days ago 246 MB registry.reg-aws.openshift.com:443/openshift3/ose-pod v3.10.148 d9b9d26e239e 10 days ago 229 MB registry.reg-aws.openshift.com:443/rhel7/etcd 3.2.22 bb2f1d4dd3a7 12 months ago 256 MB ===================================================== Should be the dependency casue the issue in task [openshift_node : download new node packages]. # yum install --downloadonly atomic-openshift-3.10.148 atomic-openshift-hyperkube-3.10.148 atomic-openshift-node-3.10.148 atomic-openshift-clients-3.10.148 Loaded plugins: amazon-id, search-disabled-repos aos | 1.3 kB 00:00:00 aos310 | 1.3 kB 00:00:00 aos_addon3_10 | 1.3 kB 00:00:00 docker_previous | 2.9 kB 00:00:00 dockernext | 2.9 kB 00:00:00 oso-rhui-rhel-server-extras | 3.6 kB 00:00:00 oso-rhui-rhel-server-fast-datapath | 3.6 kB 00:00:00 oso-rhui-rhel-server-releases | 3.6 kB 00:00:00 prerelease-extra | 3.0 kB 00:00:00 prerelease-rhel7 | 2.8 kB 00:00:00 (1/14): aos/primary | 147 kB 00:00:00 (2/14): aos310/primary | 128 kB 00:00:00 (3/14): docker_previous/primary_db | 112 kB 00:00:00 (4/14): aos_addon3_10/primary | 128 kB 00:00:00 (5/14): dockernext/primary_db | 41 kB 00:00:00 (6/14): oso-rhui-rhel-server-fast-datapath/group_gz | 133 B 00:00:00 (7/14): oso-rhui-rhel-server-extras/group_gz | 133 B 00:00:00 (8/14): prerelease-extra/primary_db | 50 kB 00:00:00 (9/14): oso-rhui-rhel-server-extras/primary_db | 1.0 MB 00:00:00 (10/14): prerelease-rhel7/group_gz | 103 kB 00:00:00 (11/14): oso-rhui-rhel-server-fast-datapath/primary_db | 434 kB 00:00:00 (12/14): oso-rhui-rhel-server-releases/group_gz | 151 kB 00:00:00 (13/14): prerelease-rhel7/primary | 2.0 MB 00:00:00 (14/14): oso-rhui-rhel-server-releases/primary_db | 108 MB 00:00:02 aos 558/558 aos310 581/581 aos_addon3_10 581/581 prerelease-rhel7 5228/5228 Resolving Dependencies --> Running transaction check ---> Package atomic-openshift.x86_64 0:3.9.83-1.git.0.78bd535.el7 will be updated --> Processing Dependency: atomic-openshift = 3.9.83-1.git.0.78bd535.el7 for package: atomic-openshift-master-3.9.83-1.git.0.78bd535.el7.x86_64 ---> Package atomic-openshift.x86_64 0:3.10.148-1.git.0.45a3c37.el7 will be an update ---> Package atomic-openshift-clients.x86_64 0:3.9.83-1.git.0.78bd535.el7 will be updated ---> Package atomic-openshift-clients.x86_64 0:3.10.148-1.git.0.45a3c37.el7 will be an update ---> Package atomic-openshift-hyperkube.x86_64 0:3.10.148-1.git.0.45a3c37.el7 will be installed ---> Package atomic-openshift-node.x86_64 0:3.9.83-1.git.0.78bd535.el7 will be updated --> Processing Dependency: atomic-openshift-node = 3.9.83-1.git.0.78bd535.el7 for package: atomic-openshift-sdn-ovs-3.9.83-1.git.0.78bd535.el7.x86_64 ---> Package atomic-openshift-node.x86_64 0:3.10.148-1.git.0.45a3c37.el7 will be an update --> Running transaction check ---> Package atomic-openshift-master.x86_64 0:3.9.83-1.git.0.78bd535.el7 will be updated ---> Package atomic-openshift-master.x86_64 0:3.10.149-1.git.0.05de590.el7 will be an update --> Processing Dependency: atomic-openshift = 3.10.149-1.git.0.05de590.el7 for package: atomic-openshift-master-3.10.149-1.git.0.05de590.el7.x86_64 aos310/filelists | 665 kB 00:00:00 ---> Package atomic-openshift-sdn-ovs.x86_64 0:3.9.83-1.git.0.78bd535.el7 will be updated ---> Package atomic-openshift-sdn-ovs.x86_64 0:3.10.149-1.git.0.05de590.el7 will be an update --> Running transaction check ---> Package atomic-openshift.x86_64 0:3.9.83-1.git.0.78bd535.el7 will be updated ---> Package atomic-openshift.x86_64 0:3.9.83-1.git.0.78bd535.el7 will be updated ---> Package atomic-openshift.x86_64 0:3.10.148-1.git.0.45a3c37.el7 will be an update ---> Package atomic-openshift.x86_64 0:3.10.149-1.git.0.05de590.el7 will be an update --> Processing Dependency: atomic-openshift-clients = 3.10.149-1.git.0.05de590.el7 for package: atomic-openshift-3.10.149-1.git.0.05de590.el7.x86_64 ---> Package atomic-openshift-node.x86_64 0:3.9.83-1.git.0.78bd535.el7 will be updated ---> Package atomic-openshift-node.x86_64 0:3.9.83-1.git.0.78bd535.el7 will be updated ---> Package atomic-openshift-node.x86_64 0:3.10.148-1.git.0.45a3c37.el7 will be an update ---> Package atomic-openshift-node.x86_64 0:3.10.149-1.git.0.05de590.el7 will be an update --> Processing Dependency: atomic-openshift-hyperkube = 3.10.149-1.git.0.05de590.el7 for package: atomic-openshift-node-3.10.149-1.git.0.05de590.el7.x86_64 --> Running transaction check ---> Package atomic-openshift-clients.x86_64 0:3.9.83-1.git.0.78bd535.el7 will be updated ---> Package atomic-openshift-clients.x86_64 0:3.9.83-1.git.0.78bd535.el7 will be updated ---> Package atomic-openshift-clients.x86_64 0:3.10.148-1.git.0.45a3c37.el7 will be an update ---> Package atomic-openshift-clients.x86_64 0:3.10.149-1.git.0.05de590.el7 will be an update ---> Package atomic-openshift-hyperkube.x86_64 0:3.10.148-1.git.0.45a3c37.el7 will be installed ---> Package atomic-openshift-hyperkube.x86_64 0:3.10.149-1.git.0.05de590.el7 will be installed --> Finished Dependency Resolution Dependencies Resolved ============================================================================================================================================================================= Package Arch Version Repository Size ============================================================================================================================================================================= Updating: atomic-openshift-node x86_64 3.10.149-1.git.0.05de590.el7 aos_addon3_10 5.2 M Installing for dependencies: atomic-openshift-hyperkube x86_64 3.10.149-1.git.0.05de590.el7 aos_addon3_10 33 M Updating for dependencies: atomic-openshift x86_64 3.10.149-1.git.0.05de590.el7 aos_addon3_10 39 M atomic-openshift-clients x86_64 3.10.149-1.git.0.05de590.el7 aos_addon3_10 20 M atomic-openshift-master x86_64 3.10.149-1.git.0.05de590.el7 aos_addon3_10 960 k atomic-openshift-sdn-ovs x86_64 3.10.149-1.git.0.05de590.el7 aos_addon3_10 4.6 M Transaction Summary ============================================================================================================================================================================= Install ( 1 Dependent package) Upgrade 1 Package (+4 Dependent packages) Total download size: 103 M Background downloading packages, then exiting: Delta RPMs disabled because /usr/bin/applydeltarpm not installed. (1/6): atomic-openshift-clients-3.10.149-1.git.0.05de590.el7.x86_64.rpm | 20 MB 00:00:00 (2/6): atomic-openshift-3.10.149-1.git.0.05de590.el7.x86_64.rpm | 39 MB 00:00:01 (3/6): atomic-openshift-master-3.10.149-1.git.0.05de590.el7.x86_64.rpm | 960 kB 00:00:00 (4/6): atomic-openshift-hyperkube-3.10.149-1.git.0.05de590.el7.x86_64.rpm | 33 MB 00:00:00 (5/6): atomic-openshift-node-3.10.149-1.git.0.05de590.el7.x86_64.rpm | 5.2 MB 00:00:00 (6/6): atomic-openshift-sdn-ovs-3.10.149-1.git.0.05de590.el7.x86_64.rpm | 4.6 MB 00:00:00 ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------- Total 65 MB/s | 103 MB 00:00:01 exiting because "Download Only" specified Version-Release number of the following components: ansible-2.4.6.0-1.el7ae.noarch openshift-ansible-3.10.149-1.git.0.eb0262c.el7.noarch How reproducible: always Steps to Reproduce: 1.Install ocp v3.9 2. Enable both v3.10.148 and latest v3.10.149 repos on the hosts 3. Specify openshift_release=v3.10 openshift_pkg_version=-3.10.148 openshift_image_tag=v3.10.149 4. Run upgrade /usr/share/ansible/openshift-ansible/playbooks/openshift-master/openshift_node_group.yml /usr/share/ansible/openshift-ansible/playbooks/byo/openshift-cluster/upgrades/v3_10/upgrade.yml Actual results: Cluster is updated to a unexpected version. Expected results: Cluster should be updated to specified version. Additional info: Please attach logs from ansible-playbook with the -vvv flag
This issue is only present for 3.9 to 3.10 upgrades due to the package dependency resolution including atomic-openshift-master, which is not applicable to 3.10. During the upgrade, the atomic-openshift-master package is removed and future upgrades will honor the requested package version. This issue is not present in upgrades from 3.10 to 3.11. Due to the limited impact of one time 3.9 to 3.10 upgrade, we will not fix this issue. If a customer has a specific impact when upgrading while trying to get to 3.11, please open a new bug.