Description of problem: Upgrade against ocp with dedicated etcd hosts. Upgrade failed when check backup etcd in phase 3.7 to 3.8. It seems that Play [Backup etcd] was not run due to a missing module when gather facts. TASK [Gather Cluster facts] **************************************************** task path: /usr/share/ansible/openshift-ansible/playbooks/init/cluster_facts.yml:9 fatal: [x.x.x.x]: FAILED! => {"changed": false, "module_stderr": "Shared connection to x.x.x.x closed.\r\n", "module_stdout": "Traceback (most recent call last):\r\n File \"/tmp/ansible_FSLoJO/ansible_module_openshift_facts.py\", line 18, in <module>\r\n import ipaddress\r\nImportError: No module named ipaddress\r\n", "msg": "MODULE FAILURE", "rc": 0} <--snip--> PLAY [Backup etcd] ************************************************************* PLAY [Gate on etcd backup] ***************************************************** TASK [Gathering Facts] ********************************************************* ok: [localhost] META: ran handlers TASK [set_fact] **************************************************************** task path: /usr/share/ansible/openshift-ansible/playbooks/openshift-etcd/private/upgrade_backup.yml:18 ok: [localhost] => {"ansible_facts": {"etcd_backup_completed": []}, "changed": false} TASK [set_fact] **************************************************************** task path: /usr/share/ansible/openshift-ansible/playbooks/openshift-etcd/private/upgrade_backup.yml:22 ok: [localhost] => {"ansible_facts": {"etcd_backup_failed": [x.x.x.x]}, "changed": false} TASK [fail] ******************************************************************** task path: /usr/share/ansible/openshift-ansible/playbooks/openshift-etcd/private/upgrade_backup.yml:24 fatal: [localhost]: FAILED! => {"changed": false, "msg": "Upgrade cannot continue. The following hosts did not complete etcd backup: qe-jliu-rp1-etcd-1.0131-gy1.qe.rhcloud.com"} Version-Release number of the following components: ansible-2.4.2.0-2.el7.noarch openshift-ansible-3.9.0-0.34.0.git.0.c7d9585.el7.noarch How reproducible: always Steps to Reproduce: 1. Install ocp with dedicated etcd 2. Run upgrade against above ocp ansible-playbook /usr/share/ansible/openshift-ansible/playbooks/byo/openshift-cluster/upgrades/v3_9/upgrade_control_plane.yml 3. Actual results: Upgrade failed Expected results: Upgrade succeed. Additional info: Please attach logs from ansible-playbook with the -vvv flag
Workaround, on your etcd hosts: yum install python-ipaddress The real fix is we need to make sure that these upgrade playbooks include the proper init playbooks in order to ensure pre-requisites are included.
*** Bug 1541452 has been marked as a duplicate of this bug. ***
*** Bug 1541449 has been marked as a duplicate of this bug. ***
Proposed: https://github.com/openshift/openshift-ansible/pull/7017
*** Bug 1541597 has been marked as a duplicate of this bug. ***
Commits pushed to master at https://github.com/openshift/openshift-ansible https://github.com/openshift/openshift-ansible/commit/b683605bd97a798589000d32d1b9ea15bf75666d Add base package installation to upgrade playbooks Hosts will need python ipaddress module installed if it was not installed during initial installation. Bug 1540537 https://bugzilla.redhat.com/show_bug.cgi?id=1540537 https://github.com/openshift/openshift-ansible/commit/52403c8d9ab6e604fb0060250526ae8c1587c269 Merge pull request #7017 from mtnbikenc/fix-1540537 Automatic merge from submit-queue. [1540537] Add base package installation to upgrade playbooks Hosts will need python ipaddress module installed if it was not installed during initial installation. Bug 1540537 https://bugzilla.redhat.com/show_bug.cgi?id=1540537
$ git tag --contains b683605bd97a798589000d32d1b9ea15bf75666d openshift-ansible-3.9.0-0.42.0 openshift-ansible-3.9.0-0.43.0 openshift-ansible-3.9.0-0.44.0 openshift-ansible-3.9.0-0.45.0
Verified on openshift-ansible-3.9.0-0.47.0.git.0.f8847bb.el7.noarch.rpm.