Bug 1692730
| Summary: | Fail to upgrade ocp to non latest version | ||
|---|---|---|---|
| Product: | OpenShift Container Platform | Reporter: | liujia <jiajliu> |
| Component: | Cluster Version Operator | Assignee: | Patrick Dillon <padillon> |
| Status: | CLOSED ERRATA | QA Contact: | liujia <jiajliu> |
| Severity: | high | Docs Contact: | |
| Priority: | high | ||
| Version: | 3.10.0 | CC: | aos-bugs, jokerman, mmccomas, wmeng |
| Target Milestone: | --- | Keywords: | Reopened |
| Target Release: | 3.10.z | ||
| Hardware: | Unspecified | ||
| OS: | Unspecified | ||
| Whiteboard: | |||
| Fixed In Version: | Doc Type: | Bug Fix | |
| Doc Text: |
Cause: a task was inserted early in the upgrade playbooks to install client packages, but the task does not specify explicit versions for all dependencies.
Consequence: circular dependencies within the packages would cause all packages to be updated to the latest version, although a prior version was installed.
Fix: remove the problematic client install task entirely and fix the bug which required that task in another fashion.
Result: install completes with packages installed to the specified version.
|
Story Points: | --- |
| Clone Of: | Environment: | ||
| Last Closed: | 2019-06-27 16:41:12 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
> 2. Enable both v3.10.127 and latest v3.10.129 repos on the hosts
The target host should only have the repos enabled for the appropriate version (in this case, disable v3.10.129). I don't believe this is a bug. Please reopen if you think otherwise or something develops.
(In reply to Patrick Dillon from comment #5) > > 2. Enable both v3.10.127 and latest v3.10.129 repos on the hosts > > The target host should only have the repos enabled for the appropriate > version (in this case, disable v3.10.129). I don't believe this is a bug. > Please reopen if you think otherwise or something develops. Hi Patrick The key point for the issue is not how many repos enabled, but multi versions are avialable during upgrade. QE enabled two repos is just to simulate two pre-release versions avaialable. In customer env, even only one ose z-stream repo enabled in target host, but there will be many versions avialable in this repo, such like[1](there are v3.10.111,v3.10.119, and so on in one repo v3.10). So when they do upgrade to a non-latest version, they will hit the issue. # yum install atomic-openshift-clients-3.10.111 Loaded plugins: product-id, search-disabled-repos, subscription-manager This system is not registered with an entitlement server. You can use subscription-manager to register. aos39-devel-install | 4.0 kB 00:00:00 fast-datapath | 4.0 kB 00:00:00 rhel7 | 3.4 kB 00:00:00 rhel7-extra | 3.4 kB 00:00:00 (1/12): aos39-devel-install/group | 124 B 00:00:00 (2/12): aos39-devel-install/updateinfo | 83 kB 00:00:00 (3/12): fast-datapath/updateinfo | 22 kB 00:00:00 (4/12): fast-datapath/group | 124 B 00:00:00 (5/12): aos39-devel-install/primary_db | 300 kB 00:00:00 (6/12): fast-datapath/primary_db | 108 kB 00:00:00 (7/12): rhel7/group | 774 kB 00:00:00 (8/12): rhel7-extra/group | 124 B 00:00:00 (9/12): rhel7-extra/updateinfo | 169 kB 00:00:00 (10/12): rhel7-extra/primary_db | 519 kB 00:00:00 (11/12): rhel7/updateinfo | 3.0 MB 00:00:01 (12/12): rhel7/primary_db | 54 MB 00:00:28 Resolving Dependencies --> Running transaction check ---> Package atomic-openshift-clients.x86_64 0:3.9.74-1.git.0.78e56ea.el7 will be updated --> Processing Dependency: atomic-openshift-clients = 3.9.74-1.git.0.78e56ea.el7 for package: atomic-openshift-3.9.74-1.git.0.78e56ea.el7.x86_64 ---> Package atomic-openshift-clients.x86_64 0:3.10.111-1.git.0.84b439a.el7 will be an update --> Running transaction check ---> Package atomic-openshift.x86_64 0:3.9.74-1.git.0.78e56ea.el7 will be updated --> Processing Dependency: atomic-openshift = 3.9.74-1.git.0.78e56ea.el7 for package: atomic-openshift-node-3.9.74-1.git.0.78e56ea.el7.x86_64 --> Processing Dependency: atomic-openshift = 3.9.74-1.git.0.78e56ea.el7 for package: atomic-openshift-master-3.9.74-1.git.0.78e56ea.el7.x86_64 ---> Package atomic-openshift.x86_64 0:3.10.119-1.git.0.a1cf669.el7 will be an update --> Processing Dependency: atomic-openshift-clients = 3.10.119-1.git.0.a1cf669.el7 for package: atomic-openshift-3.10.119-1.git.0.a1cf669.el7.x86_64 aos39-devel-install/filelists_db | 987 kB 00:00:00 --> Running transaction check ---> Package atomic-openshift-clients.x86_64 0:3.9.74-1.git.0.78e56ea.el7 will be updated ---> Package atomic-openshift-clients.x86_64 0:3.9.74-1.git.0.78e56ea.el7 will be updated ---> Package atomic-openshift-clients.x86_64 0:3.10.111-1.git.0.84b439a.el7 will be an update ---> Package atomic-openshift-clients.x86_64 0:3.10.119-1.git.0.a1cf669.el7 will be an update ---> Package atomic-openshift-master.x86_64 0:3.9.74-1.git.0.78e56ea.el7 will be updated ---> Package atomic-openshift-master.x86_64 0:3.10.119-1.git.0.a1cf669.el7 will be an update ---> Package atomic-openshift-node.x86_64 0:3.9.74-1.git.0.78e56ea.el7 will be updated --> Processing Dependency: atomic-openshift-node = 3.9.74-1.git.0.78e56ea.el7 for package: atomic-openshift-sdn-ovs-3.9.74-1.git.0.78e56ea.el7.x86_64 ---> Package atomic-openshift-node.x86_64 0:3.10.119-1.git.0.a1cf669.el7 will be an update --> Processing Dependency: atomic-openshift-hyperkube = 3.10.119-1.git.0.a1cf669.el7 for package: atomic-openshift-node-3.10.119-1.git.0.a1cf669.el7.x86_64 --> Running transaction check ---> Package atomic-openshift-hyperkube.x86_64 0:3.10.119-1.git.0.a1cf669.el7 will be installed ---> Package atomic-openshift-sdn-ovs.x86_64 0:3.9.74-1.git.0.78e56ea.el7 will be updated ---> Package atomic-openshift-sdn-ovs.x86_64 0:3.10.119-1.git.0.a1cf669.el7 will be an update --> Finished Dependency Resolution Dependencies Resolved ============================================================================================================================================================================= Package Arch Version Repository Size ============================================================================================================================================================================= Installing for dependencies: atomic-openshift-hyperkube x86_64 3.10.119-1.git.0.a1cf669.el7 aos39-devel-install 33 M Updating for dependencies: atomic-openshift x86_64 3.10.119-1.git.0.a1cf669.el7 aos39-devel-install 39 M atomic-openshift-clients x86_64 3.10.119-1.git.0.a1cf669.el7 aos39-devel-install 20 M atomic-openshift-master x86_64 3.10.119-1.git.0.a1cf669.el7 aos39-devel-install 958 k atomic-openshift-node x86_64 3.10.119-1.git.0.a1cf669.el7 aos39-devel-install 5.2 M atomic-openshift-sdn-ovs x86_64 3.10.119-1.git.0.a1cf669.el7 aos39-devel-install 4.6 M Transaction Summary ============================================================================================================================================================================= Install ( 1 Dependent package) Upgrade ( 5 Dependent packages) Total download size: 103 M Is this ok [y/d/N]: So open the bug again. [1] http://pulp.dist.prod.ext.phx2.redhat.com/content/dist/rhel/server/7/7Server/x86_64/ose/3.10/os/Packages/a/ Patrick, It looks like this is possibly a regression caused by https://github.com/openshift/openshift-ansible/pull/10783 in openshift-ansible-3.10.100-1 and later. There is a circular RPM dependency which causes all packages to get upgraded in task [Install openshift clients on first master]. During upgrades of these packages we must always specify the version of all dependent packages to prevent this circular dependency and is what is done during the regular package upgrade tasks. We need to revisit the fix which introduced this issue to see if we can work around needing to install all dependencies or do the check at a later time when binaries are properly updated. Closed the above PR to take a different approach of attempting to remove the task Install OpenShift Clients on first master. Version: ansible-2.4.6.0-1.el7ae.noarch openshift-ansible-3.10.149-1.git.0.eb0262c.el7.noarch Steps to Reproduce: 1. Install ocp v3.9 2. Enable both v3.10.148 and latest v3.10.149 repos on the hosts 3. Specify openshift_release=v3.10 openshift_pkg_version=-3.10.148 openshift_image_tag=v3.10.149 4. Run upgrade /usr/share/ansible/openshift-ansible/playbooks/openshift-master/openshift_node_group.yml /usr/share/ansible/openshift-ansible/playbooks/byo/openshift-cluster/upgrades/v3_10/upgrade.yml Upgrade succeed. The original issue about oc bianry has been fixed. But the cluster was still not updated to correct version due to another issue. Since it is another cause, so file a new bug to keep track it and close this one for the fix in pr11518. Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2019:1607 |
Description of problem: Upgrade ocp v3.9 to v3.10 when specify openshift_pkg_version to an old version. Upgrade failed at TASK [Install openshift clients on first master] ******************************* task path: /usr/share/ansible/openshift-ansible/playbooks/common/openshift-cluster/upgrades/pre/verify_cluster.yml:47 Monday 25 March 2019 09:21:30 +0000 (0:00:00.035) 0:59:41.307 ********** FAILED - RETRYING: Install openshift clients on first master (3 retries left). FAILED - RETRYING: Install openshift clients on first master (2 retries left). FAILED - RETRYING: Install openshift clients on first master (1 retries left). fatal: [x]: FAILED! => {"attempts": 3, "changed": false, "failed": true, "msg": "No package matching 'atomic-openshift-clients-3.10.127' found available, installed or updated", "rc": 126, "results": ["No package matching 'atomic-openshift-clients-3.10.127' found available, installed or updated"]} Debug info: 1) The first cause should be excluder. # yum search atomic-openshift-clients --showdup Loaded plugins: amazon-id, search-disabled-repos =================================================================== N/S matched: atomic-openshift-clients =================================================================== atomic-openshift-clients-3.10.129-1.git.0.05be54b.el7.x86_64 : Origin Client binaries for Linux Name and summary matches only, use "search all" for everything. Checked that the excluder still enabled, which is not consistant with the task [openshift_excluder : disable openshift excluder] before this failed task. # cat /etc/yum.conf |grep exclude exclude= atomic-openshift-tests atomic-openshift-hyperkube atomic-openshift-sdn-ovs atomic-openshift-recycle atomic-openshift-pod atomic-openshift-node atomic-openshift-master atomic-openshift-clients-redistributable atomic-openshift-clients atomic-openshift docker*1.20* docker*1.19* docker*1.18* docker*1.17* docker*1.16* docker*1.15* docker*1.14* Run unexclude manually, then "yum search" can list avialble atomic-openshift-clients pkg. 2) And another cause should be atomic-openshift-clients-3.10.129(latest version) has been installed in previous TASK [Install openshift clients on first master] as a dependency. # cat /etc/yum.conf |grep exclude exclude= docker*1.20* docker*1.19* docker*1.18* docker*1.17* docker*1.16* docker*1.15* docker*1.14* # yum install atomic-openshift-clients-3.10.127 Loaded plugins: amazon-id, search-disabled-repos aos | 1.3 kB 00:00:00 aos310 | 1.3 kB 00:00:00 aos_addon3_10 | 1.3 kB 00:00:00 Package matching atomic-openshift-clients-3.10.127-1.git.0.dab74c6.el7.x86_64 already installed. Checking for update. Nothing to do # rpm -qa|grep clients openssh-clients-7.4p1-16.el7.x86_64 atomic-openshift-clients-3.10.129-1.git.0.05be54b.el7.x86_64 So i dig more on a fresh installed v3.9 cluster. When enable both latest v3.10.129 and v3.10.127 repos. Try to install atomic-openshift-clients-3.10.127, the result is that atomic-openshift-clients-3.10.129 will be installed. This comply with the cause 2), which showed in upgrade log. # yum install atomic-openshift-clients-3.10.127 Loaded plugins: amazon-id, search-disabled-repos aos | 1.3 kB 00:00:00 aos_addon3_10 | 1.3 kB 00:00:00 oso-rhui-rhel-server-extras | 4.0 kB 00:00:00 oso-rhui-rhel-server-fast-datapath | 4.0 kB 00:00:00 oso-rhui-rhel-server-releases | 4.0 kB 00:00:00 Resolving Dependencies --> Running transaction check ---> Package atomic-openshift-clients.x86_64 0:3.9.71-1.git.0.dba24d1.el7 will be updated --> Processing Dependency: atomic-openshift-clients = 3.9.71-1.git.0.dba24d1.el7 for package: atomic-openshift-3.9.71-1.git.0.dba24d1.el7.x86_64 ---> Package atomic-openshift-clients.x86_64 0:3.10.127-1.git.0.dab74c6.el7 will be an update --> Running transaction check ---> Package atomic-openshift.x86_64 0:3.9.71-1.git.0.dba24d1.el7 will be updated --> Processing Dependency: atomic-openshift = 3.9.71-1.git.0.dba24d1.el7 for package: atomic-openshift-node-3.9.71-1.git.0.dba24d1.el7.x86_64 --> Processing Dependency: atomic-openshift = 3.9.71-1.git.0.dba24d1.el7 for package: atomic-openshift-master-3.9.71-1.git.0.dba24d1.el7.x86_64 ---> Package atomic-openshift.x86_64 0:3.10.129-1.git.0.05be54b.el7 will be an update --> Processing Dependency: atomic-openshift-clients = 3.10.129-1.git.0.05be54b.el7 for package: atomic-openshift-3.10.129-1.git.0.05be54b.el7.x86_64 aos/filelists | 665 kB 00:00:00 --> Running transaction check ---> Package atomic-openshift-clients.x86_64 0:3.9.71-1.git.0.dba24d1.el7 will be updated ---> Package atomic-openshift-clients.x86_64 0:3.9.71-1.git.0.dba24d1.el7 will be updated ---> Package atomic-openshift-clients.x86_64 0:3.10.127-1.git.0.dab74c6.el7 will be an update ---> Package atomic-openshift-clients.x86_64 0:3.10.129-1.git.0.05be54b.el7 will be an update ---> Package atomic-openshift-master.x86_64 0:3.9.71-1.git.0.dba24d1.el7 will be updated ---> Package atomic-openshift-master.x86_64 0:3.10.129-1.git.0.05be54b.el7 will be an update ---> Package atomic-openshift-node.x86_64 0:3.9.71-1.git.0.dba24d1.el7 will be updated --> Processing Dependency: atomic-openshift-node = 3.9.71-1.git.0.dba24d1.el7 for package: atomic-openshift-sdn-ovs-3.9.71-1.git.0.dba24d1.el7.x86_64 ---> Package atomic-openshift-node.x86_64 0:3.10.129-1.git.0.05be54b.el7 will be an update --> Processing Dependency: atomic-openshift-hyperkube = 3.10.129-1.git.0.05be54b.el7 for package: atomic-openshift-node-3.10.129-1.git.0.05be54b.el7.x86_64 --> Running transaction check ---> Package atomic-openshift-hyperkube.x86_64 0:3.10.129-1.git.0.05be54b.el7 will be installed ---> Package atomic-openshift-sdn-ovs.x86_64 0:3.9.71-1.git.0.dba24d1.el7 will be updated ---> Package atomic-openshift-sdn-ovs.x86_64 0:3.10.129-1.git.0.05be54b.el7 will be an update --> Finished Dependency Resolution Dependencies Resolved ============================================================================================================================================================================= Package Arch Version Repository Size ============================================================================================================================================================================= Installing for dependencies: atomic-openshift-hyperkube x86_64 3.10.129-1.git.0.05be54b.el7 aos_addon3_10 33 M Updating for dependencies: atomic-openshift x86_64 3.10.129-1.git.0.05be54b.el7 aos_addon3_10 39 M atomic-openshift-clients x86_64 3.10.129-1.git.0.05be54b.el7 aos_addon3_10 20 M atomic-openshift-master x86_64 3.10.129-1.git.0.05be54b.el7 aos_addon3_10 958 k atomic-openshift-node x86_64 3.10.129-1.git.0.05be54b.el7 aos_addon3_10 5.2 M atomic-openshift-sdn-ovs x86_64 3.10.129-1.git.0.05be54b.el7 aos_addon3_10 4.6 M Transaction Summary ============================================================================================================================================================================= Install ( 1 Dependent package) Upgrade ( 5 Dependent packages) Total download size: 103 M Is this ok [y/d/N]: Version-Release number of the following components: ansible-2.4.6.0-1.el7ae.noarch openshift-ansible-3.10.127-1.git.0.131da09.el7.noarch How reproducible: always Steps to Reproduce: 1. Install ocp v3.9 2. Enable both v3.10.127 and latest v3.10.129 repos on the hosts 3. Specify openshift_release=v3.10 openshift_pkg_version=-3.10.127 openshift_image_tag=v3.10.127 4. Run upgrade /usr/share/ansible/openshift-ansible/playbooks/openshift-master/openshift_node_group.yml /usr/share/ansible/openshift-ansible/playbooks/byo/openshift-cluster/upgrades/v3_10/upgrade.yml Actual results: Upgrade failed. Expected results: Ocp was upgraded to v3.10.127. Additional info: Please attach logs from ansible-playbook with the -vvv flag