Bug 1540537 - Upgrade failed against ocp with dedicated etcd due to a missing module when gather facts
Summary: Upgrade failed against ocp with dedicated etcd due to a missing module when g...
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Cluster Version Operator
Version: 3.9.0
Hardware: Unspecified
OS: Unspecified
high
high
Target Milestone: ---
: 3.9.0
Assignee: Russell Teague
QA Contact: liujia
URL:
Whiteboard:
: 1541449 1541452 1541597 (view as bug list)
Depends On:
Blocks: 1536839 1551388
TreeView+ depends on / blocked
 
Reported: 2018-01-31 10:49 UTC by liujia
Modified: 2021-06-10 14:24 UTC (History)
7 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Older node installs may not have newly required packages installed for running openshift-ansible upgrades. The common base package requirements playbook has been added to upgrade initialization to ensure all required packages are installed.
Clone Of:
: 1551388 (view as bug list)
Environment:
Last Closed: 2018-06-18 16:07:18 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)

Description liujia 2018-01-31 10:49:36 UTC
Description of problem:
Upgrade against ocp with dedicated etcd hosts. Upgrade failed when check backup etcd in phase 3.7 to 3.8. It seems that Play [Backup etcd] was not run due to a missing module when gather facts.

TASK [Gather Cluster facts] ****************************************************
task path: /usr/share/ansible/openshift-ansible/playbooks/init/cluster_facts.yml:9
fatal: [x.x.x.x]: FAILED! => {"changed": false, "module_stderr": "Shared connection to x.x.x.x closed.\r\n", "module_stdout": "Traceback (most recent call last):\r\n  File \"/tmp/ansible_FSLoJO/ansible_module_openshift_facts.py\", line 18, in <module>\r\n    import ipaddress\r\nImportError: No module named ipaddress\r\n", "msg": "MODULE FAILURE", "rc": 0}

<--snip-->
PLAY [Backup etcd] *************************************************************

PLAY [Gate on etcd backup] *****************************************************

TASK [Gathering Facts] *********************************************************
ok: [localhost]
META: ran handlers

TASK [set_fact] ****************************************************************
task path: /usr/share/ansible/openshift-ansible/playbooks/openshift-etcd/private/upgrade_backup.yml:18
ok: [localhost] => {"ansible_facts": {"etcd_backup_completed": []}, "changed": false}

TASK [set_fact] ****************************************************************
task path: /usr/share/ansible/openshift-ansible/playbooks/openshift-etcd/private/upgrade_backup.yml:22
ok: [localhost] => {"ansible_facts": {"etcd_backup_failed": [x.x.x.x]}, "changed": false}

TASK [fail] ********************************************************************
task path: /usr/share/ansible/openshift-ansible/playbooks/openshift-etcd/private/upgrade_backup.yml:24
fatal: [localhost]: FAILED! => {"changed": false, "msg": "Upgrade cannot continue. The following hosts did not complete etcd backup: qe-jliu-rp1-etcd-1.0131-gy1.qe.rhcloud.com"}

Version-Release number of the following components:
ansible-2.4.2.0-2.el7.noarch
openshift-ansible-3.9.0-0.34.0.git.0.c7d9585.el7.noarch

How reproducible:
always

Steps to Reproduce:
1. Install ocp with dedicated etcd
2. Run upgrade against above ocp
ansible-playbook /usr/share/ansible/openshift-ansible/playbooks/byo/openshift-cluster/upgrades/v3_9/upgrade_control_plane.yml
3.

Actual results:
Upgrade failed

Expected results:
Upgrade succeed.

Additional info:
Please attach logs from ansible-playbook with the -vvv flag

Comment 1 Scott Dodson 2018-01-31 14:22:34 UTC
Workaround, on your etcd hosts:

yum install python-ipaddress


The real fix is we need to make sure that these upgrade playbooks include the proper init playbooks in order to ensure pre-requisites are included.

Comment 3 Scott Dodson 2018-02-05 20:03:03 UTC
*** Bug 1541452 has been marked as a duplicate of this bug. ***

Comment 4 Scott Dodson 2018-02-05 20:03:18 UTC
*** Bug 1541449 has been marked as a duplicate of this bug. ***

Comment 5 Russell Teague 2018-02-05 20:38:11 UTC
Proposed: https://github.com/openshift/openshift-ansible/pull/7017

Comment 6 Russell Teague 2018-02-07 20:55:55 UTC
*** Bug 1541597 has been marked as a duplicate of this bug. ***

Comment 7 openshift-github-bot 2018-02-08 00:25:34 UTC
Commits pushed to master at https://github.com/openshift/openshift-ansible

https://github.com/openshift/openshift-ansible/commit/b683605bd97a798589000d32d1b9ea15bf75666d
Add base package installation to upgrade playbooks

Hosts will need python ipaddress module installed if it was not
installed during initial installation.

Bug 1540537
https://bugzilla.redhat.com/show_bug.cgi?id=1540537

https://github.com/openshift/openshift-ansible/commit/52403c8d9ab6e604fb0060250526ae8c1587c269
Merge pull request #7017 from mtnbikenc/fix-1540537

Automatic merge from submit-queue.

[1540537] Add base package installation to upgrade playbooks

Hosts will need python ipaddress module installed if it was not
installed during initial installation.

Bug 1540537
https://bugzilla.redhat.com/show_bug.cgi?id=1540537

Comment 8 Russell Teague 2018-02-15 20:30:52 UTC
$ git tag --contains b683605bd97a798589000d32d1b9ea15bf75666d
openshift-ansible-3.9.0-0.42.0
openshift-ansible-3.9.0-0.43.0
openshift-ansible-3.9.0-0.44.0
openshift-ansible-3.9.0-0.45.0

Comment 9 liujia 2018-02-22 06:02:14 UTC
Verified on openshift-ansible-3.9.0-0.47.0.git.0.f8847bb.el7.noarch.rpm.


Note You need to log in before you can comment on or make changes to this bug.