Bug 1540537
| Summary: | Upgrade failed against ocp with dedicated etcd due to a missing module when gather facts | |||
|---|---|---|---|---|
| Product: | OpenShift Container Platform | Reporter: | liujia <jiajliu> | |
| Component: | Cluster Version Operator | Assignee: | Russell Teague <rteague> | |
| Status: | CLOSED CURRENTRELEASE | QA Contact: | liujia <jiajliu> | |
| Severity: | high | Docs Contact: | ||
| Priority: | high | |||
| Version: | 3.9.0 | CC: | aos-bugs, bleanhar, cshereme, jokerman, mmccomas, rteague, wmeng | |
| Target Milestone: | --- | |||
| Target Release: | 3.9.0 | |||
| Hardware: | Unspecified | |||
| OS: | Unspecified | |||
| Whiteboard: | ||||
| Fixed In Version: | Doc Type: | Bug Fix | ||
| Doc Text: |
Older node installs may not have newly required packages installed for running openshift-ansible upgrades. The common base package requirements playbook has been added to upgrade initialization to ensure all required packages are installed.
|
Story Points: | --- | |
| Clone Of: | ||||
| : | 1551388 (view as bug list) | Environment: | ||
| Last Closed: | 2018-06-18 16:07:18 UTC | Type: | Bug | |
| Regression: | --- | Mount Type: | --- | |
| Documentation: | --- | CRM: | ||
| Verified Versions: | Category: | --- | ||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | ||
| Cloudforms Team: | --- | Target Upstream Version: | ||
| Embargoed: | ||||
| Bug Depends On: | ||||
| Bug Blocks: | 1536839, 1551388 | |||
Workaround, on your etcd hosts: yum install python-ipaddress The real fix is we need to make sure that these upgrade playbooks include the proper init playbooks in order to ensure pre-requisites are included. *** Bug 1541452 has been marked as a duplicate of this bug. *** *** Bug 1541449 has been marked as a duplicate of this bug. *** *** Bug 1541597 has been marked as a duplicate of this bug. *** Commits pushed to master at https://github.com/openshift/openshift-ansible https://github.com/openshift/openshift-ansible/commit/b683605bd97a798589000d32d1b9ea15bf75666d Add base package installation to upgrade playbooks Hosts will need python ipaddress module installed if it was not installed during initial installation. Bug 1540537 https://bugzilla.redhat.com/show_bug.cgi?id=1540537 https://github.com/openshift/openshift-ansible/commit/52403c8d9ab6e604fb0060250526ae8c1587c269 Merge pull request #7017 from mtnbikenc/fix-1540537 Automatic merge from submit-queue. [1540537] Add base package installation to upgrade playbooks Hosts will need python ipaddress module installed if it was not installed during initial installation. Bug 1540537 https://bugzilla.redhat.com/show_bug.cgi?id=1540537 $ git tag --contains b683605bd97a798589000d32d1b9ea15bf75666d openshift-ansible-3.9.0-0.42.0 openshift-ansible-3.9.0-0.43.0 openshift-ansible-3.9.0-0.44.0 openshift-ansible-3.9.0-0.45.0 Verified on openshift-ansible-3.9.0-0.47.0.git.0.f8847bb.el7.noarch.rpm. |
Description of problem: Upgrade against ocp with dedicated etcd hosts. Upgrade failed when check backup etcd in phase 3.7 to 3.8. It seems that Play [Backup etcd] was not run due to a missing module when gather facts. TASK [Gather Cluster facts] **************************************************** task path: /usr/share/ansible/openshift-ansible/playbooks/init/cluster_facts.yml:9 fatal: [x.x.x.x]: FAILED! => {"changed": false, "module_stderr": "Shared connection to x.x.x.x closed.\r\n", "module_stdout": "Traceback (most recent call last):\r\n File \"/tmp/ansible_FSLoJO/ansible_module_openshift_facts.py\", line 18, in <module>\r\n import ipaddress\r\nImportError: No module named ipaddress\r\n", "msg": "MODULE FAILURE", "rc": 0} <--snip--> PLAY [Backup etcd] ************************************************************* PLAY [Gate on etcd backup] ***************************************************** TASK [Gathering Facts] ********************************************************* ok: [localhost] META: ran handlers TASK [set_fact] **************************************************************** task path: /usr/share/ansible/openshift-ansible/playbooks/openshift-etcd/private/upgrade_backup.yml:18 ok: [localhost] => {"ansible_facts": {"etcd_backup_completed": []}, "changed": false} TASK [set_fact] **************************************************************** task path: /usr/share/ansible/openshift-ansible/playbooks/openshift-etcd/private/upgrade_backup.yml:22 ok: [localhost] => {"ansible_facts": {"etcd_backup_failed": [x.x.x.x]}, "changed": false} TASK [fail] ******************************************************************** task path: /usr/share/ansible/openshift-ansible/playbooks/openshift-etcd/private/upgrade_backup.yml:24 fatal: [localhost]: FAILED! => {"changed": false, "msg": "Upgrade cannot continue. The following hosts did not complete etcd backup: qe-jliu-rp1-etcd-1.0131-gy1.qe.rhcloud.com"} Version-Release number of the following components: ansible-2.4.2.0-2.el7.noarch openshift-ansible-3.9.0-0.34.0.git.0.c7d9585.el7.noarch How reproducible: always Steps to Reproduce: 1. Install ocp with dedicated etcd 2. Run upgrade against above ocp ansible-playbook /usr/share/ansible/openshift-ansible/playbooks/byo/openshift-cluster/upgrades/v3_9/upgrade_control_plane.yml 3. Actual results: Upgrade failed Expected results: Upgrade succeed. Additional info: Please attach logs from ansible-playbook with the -vvv flag