Bug 1616439
| Summary: | check_available_rpms.yml: Version checking assumes latest | ||
|---|---|---|---|
| Product: | OpenShift Container Platform | Reporter: | Matthew Barnes <mbarnes> |
| Component: | Installer | Assignee: | Scott Dodson <sdodson> |
| Status: | CLOSED DUPLICATE | QA Contact: | Johnny Liu <jialiu> |
| Severity: | high | Docs Contact: | |
| Priority: | unspecified | ||
| Version: | 3.9.0 | CC: | aos-bugs, jokerman, mbarnes, mmccomas, nraghava |
| Target Milestone: | --- | Keywords: | OnlineDedicated, OpsBlocker |
| Target Release: | 3.9.z | ||
| Hardware: | Unspecified | ||
| OS: | Unspecified | ||
| Whiteboard: | |||
| Fixed In Version: | Doc Type: | If docs needed, set a value | |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2018-08-20 18:30:11 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
Matt, Can you provide your inventory inputs? Mainly interested in the values of openshift_release, openshift_version, openshift_pkg_version, openshift_image_tag. I don't expect all of those to be set, but if they are we need to know what their values are. I think this is a dupe of 1579513 Can you please see if the latest version of openshift-ansible addresses the problem? Any of these errata versions should have the fix. openshift-ansible-3.9.31-1.git.34.154617d.el7 openshift-ansible-3.9.33-1.git.56.19ba16e.el7 openshift-ansible-3.9.40-1.git.0.188c954.el7 Scott: The cluster in my OP has been upgraded already, but an identical cluster's inventory file has: openshift_pkg_version=-3.9.33 The other variables are absent: openshift_release, openshift_version, and openshift_image_tag Note: our inventory file is generated on-the-fly and then immediately handed off to playbooks/openshift-node/scaleup.yml. In case it's relevant, other places in the file where "3.9" appears are: openshift_hosted_logging_deployer_version=v3.9 openshift_logging_image_version=v3.9 openshift_metrics_image_version=v3.9 > Can you please see if the latest version of openshift-ansible addresses the problem? Any of these errata versions should have the fix.
Will do. We're still using openshift-ansible 3.9.14.
Confirmed this is a duplicate of #1579513. Retested as described in comment #1 with openshift-ansible 3.9.40 and was able to successfully scale up nodes. Thanks Scott! *** This bug has been marked as a duplicate of bug 1579513 *** |
Description of problem: Operations had a cluster on 3.9.33 that we wanted to upgrade to 3.9.40. Standard practice is to run the playbooks/openshift-node/scaleup.yml playbook to add at least one extra compute node prior to upgrade to ensure pods have a place to go while draining nodes. The yum repository currently offers RPMs for a variety of releases from 3.9.14 to 3.9.40. However the scale up playbook failed because the openshift_version role is only comparing the docker image version against the _latest available_ RPM version, instead of all available versions. I believe this is either because the repoquery task in check_available_rpms.yml is not using show_duplicates=True and/or the playbooks which evaluate the results only check the first result: {{ rpm_results.results.versions.available_versions.0 }} Consequently, we were unable to scale up the compute nodes prior to upgrade in this case. Version-Release number of the following components: rpm -q openshift-ansible : 3.9.14 rpm -q ansible : 2.4.3.0 ansible --version ansible 2.4.3.0 config file = /etc/ansible/ansible.cfg configured module search path = [u'/home/mbarnes/.ansible/plugins/modules', u'/usr/share/ansible/plugins/modules'] ansible python module location = /usr/lib/python2.7/site-packages/ansible executable location = /usr/bin/ansible python version = 2.7.5 (default, May 31 2018, 09:41:32) [GCC 4.8.5 20150623 (Red Hat 4.8.5-28)] How reproducible: Always Steps to Reproduce: 1. Upgrade a cluster to a less-than-latest version of 3.9. 2. Verify the yum repo contains both the cluster's version of atomic-openshift and a later version. 2. Attempt to run playbooks/openshift-node/scaleup.yml Actual results: Please include the entire output from the last TASK line through the end of output if an error is generated PLAY [Ensure the requested version packages are available.] ************************************************************************************************************* TASK [Gathering Facts] ************************************************************************************************************************************************** Wednesday 15 August 2018 22:14:13 +0000 (0:00:00.134) 0:02:53.951 ****** ok: [18.231.183.197] META: ran handlers TASK [include_role] ***************************************************************************************************************************************************** task path: /home/mbarnes/git/openshift-tools/openshift/installer/atomic-openshift-3.9/playbooks/init/version.yml:31 Wednesday 15 August 2018 22:14:14 +0000 (0:00:01.106) 0:02:55.058 ****** TASK [openshift_version : Check openshift_version for rpm installation] ************************************************************************************************* task path: /home/mbarnes/git/openshift-tools/openshift/installer/vendored/openshift-ansible-3.9.14-1/roles/openshift_version/tasks/masters_and_nodes.yml:5 Wednesday 15 August 2018 22:14:14 +0000 (0:00:00.064) 0:02:55.123 ****** included: /home/mbarnes/git/openshift-tools/openshift/installer/vendored/openshift-ansible-3.9.14-1/roles/openshift_version/tasks/check_available_rpms.yml for 18.231.183 .197 TASK [openshift_version : Get available atomic-openshift version] ******************************************************************************************************* task path: /home/mbarnes/git/openshift-tools/openshift/installer/vendored/openshift-ansible-3.9.14-1/roles/openshift_version/tasks/check_available_rpms.yml:2 Wednesday 15 August 2018 22:14:14 +0000 (0:00:00.050) 0:02:55.173 ****** ok: [18.231.183.197] => {"changed": false, "check_mode": false, "results": {"cmd": "/usr/bin/repoquery --plugins --quiet --pkgnarrow=repos --queryformat=%{version}|%{rel ease}|%{arch}|%{repo}|%{version}-%{release} --config=/tmp/tmpHSn1Jz atomic-openshift", "package_found": true, "package_name": "atomic-openshift", "returncode": 0, "versi ons": {"available_versions": ["3.9.40"], "available_versions_full": ["3.9.40-1.git.0.0c9824a.el7"], "latest": "3.9.40", "latest_full": "3.9.40-1.git.0.0c9824a.el7"}}, "s tate": "list"} TASK [openshift_version : fail] ***************************************************************************************************************************************** task path: /home/mbarnes/git/openshift-tools/openshift/installer/vendored/openshift-ansible-3.9.14-1/roles/openshift_version/tasks/check_available_rpms.yml:8 Wednesday 15 August 2018 22:14:27 +0000 (0:00:12.924) 0:03:08.097 ****** skipping: [18.231.183.197] => {"changed": false, "skip_reason": "Conditional result was False"} TASK [openshift_version : Fail if rpm version and docker image version are different] *********************************************************************************** task path: /home/mbarnes/git/openshift-tools/openshift/installer/vendored/openshift-ansible-3.9.14-1/roles/openshift_version/tasks/masters_and_nodes.yml:7 Wednesday 15 August 2018 22:14:27 +0000 (0:00:00.038) 0:03:08.136 ****** fatal: [18.231.183.197]: FAILED! => {"changed": false, "msg": "OCP rpm version 3.9.40 is different from OCP image version 3.9.33"} Expected results: That the openshift_version role would see that 3.9.33 RPMs are available and use them.