Bug 1539021
| Summary: | Upgrading from 3.5 to 3.6 using upgrade_nodes playbook does not honour openshift_pkg_version | ||||||||
|---|---|---|---|---|---|---|---|---|---|
| Product: | OpenShift Container Platform | Reporter: | Sergi Jimenez Romero <sjr> | ||||||
| Component: | Cluster Version Operator | Assignee: | Russell Teague <rteague> | ||||||
| Status: | CLOSED CURRENTRELEASE | QA Contact: | Weihua Meng <wmeng> | ||||||
| Severity: | medium | Docs Contact: | |||||||
| Priority: | unspecified | ||||||||
| Version: | 3.6.0 | CC: | acomabon, aos-bugs, fcami, jokerman, mmccomas, mrobson, pdwyer, rteague, sdodson, vlaad, vrutkovs, wmeng | ||||||
| Target Milestone: | --- | ||||||||
| Target Release: | 3.6.z | ||||||||
| Hardware: | Unspecified | ||||||||
| OS: | Linux | ||||||||
| Whiteboard: | |||||||||
| Fixed In Version: | Doc Type: | Bug Fix | |||||||
| Doc Text: |
Cause: yum command would install the latest version available of any dependent package which resulted in the latest version of node packages to be installed.
Fix: Ansible task was updated to include all related node packages with the version specified.
Result: Expected version was installed instead of latest.
|
Story Points: | --- | ||||||
| Clone Of: | |||||||||
| : | 1544694 1544783 (view as bug list) | Environment: | |||||||
| Last Closed: | 2018-09-04 18:41:33 UTC | Type: | Bug | ||||||
| Regression: | --- | Mount Type: | --- | ||||||
| Documentation: | --- | CRM: | |||||||
| Verified Versions: | Category: | --- | |||||||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |||||||
| Cloudforms Team: | --- | Target Upstream Version: | |||||||
| Embargoed: | |||||||||
| Bug Depends On: | |||||||||
| Bug Blocks: | 1544694, 1544783 | ||||||||
| Attachments: |
|
||||||||
Created attachment 1386521 [details]
ansible upgrade_control_plane log
Any updates? Since excluders are applied to masters only, whenever node runs on master and gets updated the new package is being pulled in. Scott, what's the rationale of setting excluder on masters only? Should this be applied to nodes as well? Hmm, they should be installed everywhere, if they're not that's yet another bug we have to fix. :-( I was wrong in comment 4 - both masters and nodes have excluders installed, but during upgrade these are disable. In release-3.6 before every upgrade the code verifies we have atomic-openshift installed, which pulls updated atomic-openshift-nodes package. I'll prepare a PR for that Fix is available in openshift-ansible-3.6.173.0.103-1.git.0.dcf417c.el7 Not fixed.
openshift-ansible-3.6.173.0.103-1.git.0.dcf417c.el7.noarch
PR is in this RPM.
task did skip.(before this rpm, result was changed)
TASK [openshift_common : Install the base package for versioning]
skipping: [host-xxxx.redhat.com] => {
"changed": false,
"skip_reason": "Conditional result was False"
}
openshift_pkg_version is set in inventory file.
openshift_pkg_version=-3.6.173.0.21
on node after upgrade
# rpm -qa | grep openshift
atomic-openshift-excluder-3.6.173.0.21-1.git.0.f95b0e7.el7.noarch
atomic-openshift-clients-3.6.173.0.96-1.git.0.8f6ff22.el7.x86_64
atomic-openshift-sdn-ovs-3.6.173.0.96-1.git.0.8f6ff22.el7.x86_64
atomic-openshift-docker-excluder-3.6.173.0.21-1.git.0.f95b0e7.el7.noarch
atomic-openshift-3.6.173.0.96-1.git.0.8f6ff22.el7.x86_64
atomic-openshift-node-3.6.173.0.96-1.git.0.8f6ff22.el7.x86_64
tuned-profiles-atomic-openshift-node-3.6.173.0.96-1.git.0.8f6ff22.el7.x86_64
two phases upgrade and one phase upgrade got the same result.
I suspect this is the same root cause as https://bugzilla.redhat.com/show_bug.cgi?id=1579513 Russ can you check to see if your fix for that bug is relevant to 3.6 as well? I know you forward ported your fixes to 3.10 and 3.9 already. In the attached logs I also found an issue with package installs that was already fixed with https://github.com/openshift/openshift-ansible/pull/7012. In build openshift-ansible-3.6.173.0.125-1.git.0.4275cdb.el7 Fixed. openshift-ansible-3.6.173.0.125-1.git.0.4275cdb.el7.noarch there is enabled repo for openshift-ansible-3.6.173.0.125 set openshift_pkg_version=-3.6.173.0.112 after upgrade on master: [root@host-8-244-128 ~]# rpm -qa | grep openshift atomic-openshift-3.6.173.0.112-1.git.0.17301ab.el7.x86_64 atomic-openshift-node-3.6.173.0.112-1.git.0.17301ab.el7.x86_64 atomic-openshift-docker-excluder-3.6.173.0.112-1.git.0.17301ab.el7.noarch atomic-openshift-clients-3.6.173.0.112-1.git.0.17301ab.el7.x86_64 tuned-profiles-atomic-openshift-node-3.6.173.0.112-1.git.0.17301ab.el7.x86_64 atomic-openshift-sdn-ovs-3.6.173.0.112-1.git.0.17301ab.el7.x86_64 atomic-openshift-excluder-3.6.173.0.112-1.git.0.17301ab.el7.noarch atomic-openshift-master-3.6.173.0.112-1.git.0.17301ab.el7.x86_64 on node: [root@host-8-250-38 ~]# rpm -qa | grep openshift atomic-openshift-excluder-3.6.173.0.112-1.git.0.17301ab.el7.noarch atomic-openshift-docker-excluder-3.6.173.0.112-1.git.0.17301ab.el7.noarch atomic-openshift-clients-3.6.173.0.112-1.git.0.17301ab.el7.x86_64 tuned-profiles-atomic-openshift-node-3.6.173.0.112-1.git.0.17301ab.el7.x86_64 atomic-openshift-sdn-ovs-3.6.173.0.112-1.git.0.17301ab.el7.x86_64 atomic-openshift-3.6.173.0.112-1.git.0.17301ab.el7.x86_64 atomic-openshift-node-3.6.173.0.112-1.git.0.17301ab.el7.x86_64 Kernel Version: 3.10.0-862.el7.x86_64 Operating System: Red Hat Enterprise Linux Server 7.5 (Maipo) |
Created attachment 1386519 [details] ansible upgrade_nodes log Description of problem: While upgrading from 3.5 to 3.6 using upgrade_control_plane and upgrade_nodes method, the nodes end up having a the latest errata available on the repositories and not sticking to openshift_pkg_version, while masters do so. Version-Release number of the following components: openshift-ansible-3.6.173.0.75-1.git.0.0a44128.el7.noarch ansible-2.4.2.0-2.el7.noarch ansible 2.4.2.0 config file = /etc/ansible/ansible.cfg configured module search path = [u'/home/vagrant/.ansible/plugins/modules', u'/usr/share/ansible/plugins/modules'] ansible python module location = /usr/lib/python2.7/site-packages/ansible executable location = /usr/bin/ansible python version = 2.7.5 (default, May 3 2017, 07:55:04) [GCC 4.8.5 20150623 (Red Hat 4.8.5-14)] How reproducible: Always Steps to Reproduce: 1. Set openshift_release=v3.6 openshift_pkg_version=-3.6.173.0.21 2. Configure repos 3. Run /usr/share/ansible/openshift-ansible/playbooks/byo/openshift-cluster/upgrades/v3_6/upgrade_control_plane.yml and then /usr/share/ansible/openshift-ansible/playbooks/byo/openshift-cluster/upgrades/v3_6/upgrade_nodes.yml Actual results: master0 ~$ rpm -q -a | grep atomic atomic-openshift-master-3.6.173.0.21-1.git.0.f95b0e7.el7.x86_64 tuned-profiles-atomic-openshift-node-3.6.173.0.21-1.git.0.f95b0e7.el7.x86_64 atomic-openshift-excluder-3.6.173.0.21-1.git.0.f95b0e7.el7.noarch atomic-openshift-node-3.6.173.0.21-1.git.0.f95b0e7.el7.x86_64 atomic-registries-1.20.1-9.git436cf5d.el7.x86_64 atomic-openshift-clients-3.6.173.0.21-1.git.0.f95b0e7.el7.x86_64 atomic-openshift-sdn-ovs-3.6.173.0.21-1.git.0.f95b0e7.el7.x86_64 atomic-openshift-3.6.173.0.21-1.git.0.f95b0e7.el7.x86_64 atomic-openshift-docker-excluder-3.6.173.0.21-1.git.0.f95b0e7.el7.noarch node0 ~$ rpm -q -a | grep atomic atomic-openshift-clients-3.6.173.0.96-1.git.0.8f6ff22.el7.x86_64 atomic-registries-1.20.1-9.git436cf5d.el7.x86_64 atomic-openshift-docker-excluder-3.6.173.0.21-1.git.0.f95b0e7.el7.noarch tuned-profiles-atomic-openshift-node-3.6.173.0.96-1.git.0.8f6ff22.el7.x86_64 atomic-openshift-excluder-3.6.173.0.21-1.git.0.f95b0e7.el7.noarch atomic-openshift-node-3.6.173.0.96-1.git.0.8f6ff22.el7.x86_64 atomic-openshift-sdn-ovs-3.6.173.0.96-1.git.0.8f6ff22.el7.x86_64 atomic-openshift-3.6.173.0.96-1.git.0.8f6ff22.el7.x86_64 Expected results: All nodes should have same version. Additional info: