Bug 1851914
| Summary: | 13 -> 16.1 upgrade: system_upgrade on bootstrap node fails | ||
|---|---|---|---|
| Product: | Red Hat OpenStack | Reporter: | Sadique Puthen <sputhenp> |
| Component: | openstack-tripleo | Assignee: | Jose Luis Franco <jfrancoa> |
| Status: | CLOSED ERRATA | QA Contact: | nlevinki <nlevinki> |
| Severity: | high | Docs Contact: | |
| Priority: | high | ||
| Version: | 16.1 (Train) | CC: | aschultz, cjeanner, dmacpher, jfrancoa, jpretori, mburns, msufiyan, pkundal, spower, tvignaud |
| Target Milestone: | z1 | Keywords: | Triaged |
| Target Release: | 16.1 (Train on RHEL 8.2) | ||
| Hardware: | Unspecified | ||
| OS: | Unspecified | ||
| Whiteboard: | |||
| Fixed In Version: | openstack-tripleo-heat-templates-11.3.2-0.20200616081539.396affd.el8ost | Doc Type: | Bug Fix |
| Doc Text: |
The overcloud deployment steps included an older Ansible syntax that tagged the `tripleo-bootstrap` and `tripleo-ssh-known-hosts` roles as `common_roles`. This older syntax caused Ansible to run tasks tagged with the `common_roles` when Ansible did not use the `common_roles` tag. This syntax resulted in errors during the 13 to 16.1 `system_upgrade` process.
+
This update uses a newer syntax to tag the `tripleo-bootstrap` and `tripleo-ssh-known-hosts` roles as `common_roles`. Errors do not appear during the 13 to 16.1 `system_upgrade` process and you no longer include the `--playbook upgrade_steps_playbook.yaml` option to the `system_upgrade` process as a workaround.
|
Story Points: | --- |
| Clone Of: | Environment: | ||
| Last Closed: | 2020-08-27 15:19:10 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
The user missed to pass specific UpgradeLeapp parameters during upgrade prepare step: parameter_defaults: UpgradeLeappEnabled: true UpgradeLeappCommandOptions: "--enablerepo rhel-8-for-x86_64-baseos-eus-rpms --enablerepo rhel-8-for-x86_64-appstream-eus-rpms --enablerepo rhel-8-for-x86_64-highavailability-eus-rpms --enablerepo advanced-virt-for-rhel-8-x86_64-rpms --enablerepo ansible-2.9-for-rhel-8-x86_64-rpms --enablerepo fast-datapath-for-rhel-8-x86_64-rpms" CephAnsibleRepo: rhceph-4-tools-for-rhel-8-x86_64-rpms Therefore, the upgrade just moved over upgrade tasks but didn't run the system_upgrade tags tasks as UpgradeLeappEnabled was set to false (default). I'll leave it open meanwhile, in case somebody else faces the same issue and close it after the hackfest. So, after the user reverted to a previous step. He managed to perform the leapp upgrade successfully in the controller but this task failed again. Therefore, this seems to be a valid BZ. Not a blocker issue thought, because it fails in a deployment step which doesn't have any system_upgrade tag. But this can be a problem when running deployment_steps taks passing a specific tag. "openstack overcloud upgrade run --stack overcloud --tags system_upgrade --limit controller-1" - this step leapp upgraded controller successfully.
PLAY RECAP *********************************************************************
controller-1 : ok=30 changed=8 unreachable=0 failed=0 skipped=16 rescued=0 ignored=1
Tuesday 30 June 2020 06:45:26 -0400 (0:00:00.060) 0:22:23.198 **********
===============================================================================
Updated nodes - controller-1
Success
But post leapp upgrade steps, failed.
TASK [tripleo-bootstrap : Gather variables for each operating system] **********
Tuesday 30 June 2020 06:45:44 -0400 (0:00:00.097) 0:00:08.109 **********
fatal: [controller-1]: FAILED! => {"msg": "No file was found when using first_found. Use errors='ignore' to allow this task to be skipped if no files are found"}
NO MORE HOSTS LEFT *************************************************************
PLAY RECAP *********************************************************************
controller-1 : ok=4 changed=1 unreachable=0 failed=1 skipped=0 rescued=0 ignored=0
Tuesday 30 June 2020 06:45:44 -0400 (0:00:00.058) 0:00:08.168 **********
===============================================================================
Ansible failed, check log at /var/log/mistral/package_update.log.
2020-06-30 06:45:44.845 714479 ERROR tripleoclient.v1.overcloud_upgrade.MajorUpgradeRun [-] Exception occured while running the command: RuntimeError: Update failed with: Ansible failed, check log at /var/log/mistral/package_update.log.
2020-06-30 06:45:44.845 714479 ERROR tripleoclient.v1.overcloud_upgrade.MajorUpgradeRun Traceback (most recent call last):
2020-06-30 06:45:44.845 714479 ERROR tripleoclient.v1.overcloud_upgrade.MajorUpgradeRun File "/usr/lib/python3.6/site-packages/tripleoclient/command.py", line 32, in run
2020-06-30 06:45:44.845 714479 ERROR tripleoclient.v1.overcloud_upgrade.MajorUpgradeRun super(Command, self).run(parsed_args)
2020-06-30 06:45:44.845 714479 ERROR tripleoclient.v1.overcloud_upgrade.MajorUpgradeRun File "/usr/lib/python3.6/site-packages/osc_lib/command/command.py", line 41, in run
2020-06-30 06:45:44.845 714479 ERROR tripleoclient.v1.overcloud_upgrade.MajorUpgradeRun return super(Command, self).run(parsed_args)
2020-06-30 06:45:44.845 714479 ERROR tripleoclient.v1.overcloud_upgrade.MajorUpgradeRun File "/usr/lib/python3.6/site-packages/cliff/command.py", line 185, in run
2020-06-30 06:45:44.845 714479 ERROR tripleoclient.v1.overcloud_upgrade.MajorUpgradeRun return_code = self.take_action(parsed_args) or 0
2020-06-30 06:45:44.845 714479 ERROR tripleoclient.v1.overcloud_upgrade.MajorUpgradeRun File "/usr/lib/python3.6/site-packages/tripleoclient/v1/overcloud_upgrade.py", line 236, in take_action
2020-06-30 06:45:44.845 714479 ERROR tripleoclient.v1.overcloud_upgrade.MajorUpgradeRun priv_key=key)
2020-06-30 06:45:44.845 714479 ERROR tripleoclient.v1.overcloud_upgrade.MajorUpgradeRun File "/usr/lib/python3.6/site-packages/tripleoclient/utils.py", line 1245, in run_update_ansible_action
2020-06-30 06:45:44.845 714479 ERROR tripleoclient.v1.overcloud_upgrade.MajorUpgradeRun verbosity=verbosity, extra_vars=extra_vars)
2020-06-30 06:45:44.845 714479 ERROR tripleoclient.v1.overcloud_upgrade.MajorUpgradeRun File "/usr/lib/python3.6/site-packages/tripleoclient/workflows/package_update.py", line 127, in update_ansible
2020-06-30 06:45:44.845 714479 ERROR tripleoclient.v1.overcloud_upgrade.MajorUpgradeRun raise RuntimeError('Update failed with: {}'.format(payload['message']))
2020-06-30 06:45:44.845 714479 ERROR tripleoclient.v1.overcloud_upgrade.MajorUpgradeRun RuntimeError: Update failed with: Ansible failed, check log at /var/log/mistral/package_update.log.
2020-06-30 06:45:44.845 714479 ERROR tripleoclient.v1.overcloud_upgrade.MajorUpgradeRun
2020-06-30 06:45:44.849 714479 ERROR openstack [-] Update failed with: Ansible failed, check log at /var/log/mistral/package_update.log.: RuntimeError: Update failed with: Ansible failed, check log at /var/log/mistral/package_update.log.
2020-06-30 06:45:44.849 714479 INFO osc_lib.shell [-] END return value: 1
sys:1: ResourceWarning: unclosed <socket.socket fd=4, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=6, laddr=('172.16.0.1', 60582)>
sys:1: ResourceWarning: unclosed <socket.socket fd=6, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=6, laddr=('172.16.0.1', 58376), raddr=('172.16.0.1', 8989)>
I hit this but on all nodes, not just the bootstrap node. I've also found that you can ignore the error and proceed with the rest of upgrade process. Not sure if there wer eany unindented consequences of ignoring the error though. When debugging the issue, I could make it work by changing the following roles section: https://github.com/openstack/tripleo-heat-templates/blob/stable/train/common/deploy-steps.j2#L471-L473 into a tasks: with - include_role for each of the roles: tasks: - include_role: name: tripleo-bootstrap - include_role: name: tripleo-ssh-known-hosts tags: - common_roles When done this way, the --tags mechanism worked correctly and skipped this block when tags was different than common_roles. Also, did run the test changing --tags to common_roles and only this block got executed A way to workaroudn this issue would be running the openstack overcloud upgrade run --tags system_upgrade --limit controller-1 passing the playbook we want to run: openstack overcloud upgrade run --tags system_upgrade --limit controller-1 --playbook upgrade_steps_playbook.yaml this way we never get to run the deploy_steps_playbook, which is the one causing trouble. Command used:
openstack overcloud upgrade run ${RUN_ANSWER} \
--stack qe-Cloud-0 \
--static-inventory upgrade_inventory.yaml \
--tags system_upgrade \
--limit "${host}" 2>&1 | tee -a "RHEL_upgrade_${host}" &
Script-> http://cougar11.scl.lab.tlv.redhat.com/DFG-upgrades-ffu-ffu-upgrade-13-16.1_director-rhel-virthost-3cont_2comp_3ceph-ipv4-vxlan-HA/29/undercloud-0.tar.gz?undercloud-0/home/stack/overcloud_system_upgrade-controller-0,ceph-0.sh
Output from run of deployment playbook:
2020-08-18 19:50:50 | 2020-08-18 19:50:50.566 185800 WARNING tripleoclient.plugin [-] Waiting for messages on queue 'tripleo' with no timeout.[00m
2020-08-18 19:51:12 | Using /var/lib/mistral/ce18c295-2714-474a-ae03-66359279a4ee/ansible.cfg as config file
2020-08-18 19:51:12 | [WARNING]: Invalid characters were found in group names but not replaced, use
2020-08-18 19:51:12 | -vvvv to see details
2020-08-18 19:51:12 | [WARNING]: Could not match supplied host pattern, ignoring: BlockStorage
2020-08-18 19:51:12 | [WARNING]: Could not match supplied host pattern, ignoring: ObjectStorage
2020-08-18 19:51:12 |
2020-08-18 19:51:12 | PLAY [Gather facts from undercloud] ********************************************
2020-08-18 19:51:12 | skipping: no hosts matched
2020-08-18 19:51:12 |
2020-08-18 19:51:12 | PLAY [Gather facts from overcloud] *********************************************
2020-08-18 19:51:12 |
2020-08-18 19:51:12 | PLAY [Load global variables] ***************************************************
2020-08-18 19:51:12 |
2020-08-18 19:51:12 | TASK [include_vars] ************************************************************
2020-08-18 19:51:12 | Tuesday 18 August 2020 19:51:02 -0400 (0:00:00.158) 0:00:00.158 ********
2020-08-18 19:51:12 | [WARNING]: Failure using method (v2_runner_on_start) in callback plugin
2020-08-18 19:51:12 | (<ansible.plugins.callback.tripleo.CallbackModule object at 0x7f5a712f9cc0>):
2020-08-18 19:51:12 | 'show_per_host_start'
2020-08-18 19:51:12 | ok: [controller-0] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false}
2020-08-18 19:51:12 |
2020-08-18 19:51:12 | PLAY [Ensure /var/lib/config-data context] *************************************
2020-08-18 19:51:12 |
2020-08-18 19:51:12 | TASK [Ensure /var/lib/config-data context] *************************************
2020-08-18 19:51:12 | Tuesday 18 August 2020 19:51:02 -0400 (0:00:00.095) 0:00:00.254 ********
2020-08-18 19:51:12 | changed: [controller-0] => {"ansible_facts": {"discovered_interpreter_python": "/usr/libexec/platform-python"}, "changed": true, "cmd": "set -o pipefail\nif [[ -e /var/lib/config-data ]]; then\n chcon -R -t svirt_sandbox_file_t /var/lib/config-data\nexit 2\nfi", "delta": "0:00:04.305510", "end": "2020-08-18 23:51:08.039580", "failed_when_result": false, "msg": "non-zero return code", "rc": 2, "start": "2020-08-18 23:51:03.734070", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []}
2020-08-18 19:51:12 |
2020-08-18 19:51:12 | PLAY [Render all_nodes data as group_vars for overcloud] ***********************
2020-08-18 19:51:12 |
2020-08-18 19:51:12 | TASK [Render all_nodes data as group_vars for overcloud] ***********************
2020-08-18 19:51:12 | Tuesday 18 August 2020 19:51:08 -0400 (0:00:05.473) 0:00:05.728 ********
2020-08-18 19:51:12 | ok: [controller-0] => {"all_nodes": null, "changed": false}
2020-08-18 19:51:12 |
2020-08-18 19:51:12 | PLAY [Set all_nodes data as group_vars for overcloud] **************************
2020-08-18 19:51:12 |
2020-08-18 19:51:12 | TASK [Set all_nodes data as group_vars for overcloud] **************************
2020-08-18 19:51:12 | Tuesday 18 August 2020 19:51:10 -0400 (0:00:02.429) 0:00:08.158 ********
2020-08-18 19:51:12 | ok: [controller-0] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false}
2020-08-18 19:51:12 |
2020-08-18 19:51:12 | PLAY [Manage SELinux] **********************************************************
2020-08-18 19:51:12 |
2020-08-18 19:51:12 | PLAY [Generate /etc/hosts] *****************************************************
2020-08-18 19:51:12 |
2020-08-18 19:51:12 | PLAY [Common roles for TripleO servers] ****************************************
2020-08-18 19:51:12 |
2020-08-18 19:51:12 | PLAY [Overcloud deploy step tasks for step 0] **********************************
2020-08-18 19:51:12 |
2020-08-18 19:51:12 | PLAY [Server pre deployment steps] *********************************************
2020-08-18 19:51:12 |
2020-08-18 19:51:12 | PLAY [Server deployments] ******************************************************
2020-08-18 19:51:12 |
2020-08-18 19:51:12 | PLAY [Controller Host prep steps] **********************************************
2020-08-18 19:51:12 |
2020-08-18 19:51:12 | PLAY [Compute Host prep steps] *************************************************
2020-08-18 19:51:12 | skipping: no hosts matched
2020-08-18 19:51:12 |
2020-08-18 19:51:12 | PLAY [BlockStorage Host prep steps] ********************************************
2020-08-18 19:51:12 | skipping: no hosts matched
2020-08-18 19:51:12 |
2020-08-18 19:51:12 | PLAY [ObjectStorage Host prep steps] *******************************************
2020-08-18 19:51:12 | skipping: no hosts matched
2020-08-18 19:51:12 |
2020-08-18 19:51:12 | PLAY [CephStorage Host prep steps] *********************************************
2020-08-18 19:51:12 | skipping: no hosts matched
2020-08-18 19:51:12 |
2020-08-18 19:51:12 | PLAY [External deployment step 1] **********************************************
2020-08-18 19:51:12 | skipping: no hosts matched
2020-08-18 19:51:12 |
2020-08-18 19:51:12 | PLAY [Overcloud deploy step tasks for 1] ***************************************
2020-08-18 19:51:12 |
2020-08-18 19:51:12 | PLAY [Overcloud common bootstrap tasks for step 1] *****************************
2020-08-18 19:51:12 |
2020-08-18 19:51:12 | PLAY [Overcloud common deploy step tasks 1] ************************************
2020-08-18 19:51:12 |
2020-08-18 19:51:12 | PLAY [External deployment step 2] **********************************************
2020-08-18 19:51:12 | skipping: no hosts matched
2020-08-18 19:51:12 |
2020-08-18 19:51:12 | PLAY [Overcloud deploy step tasks for 2] ***************************************
2020-08-18 19:51:12 |
2020-08-18 19:51:12 | PLAY [Overcloud common deploy step tasks 2] ************************************
2020-08-18 19:51:12 |
2020-08-18 19:51:12 | PLAY [External deployment step 3] **********************************************
2020-08-18 19:51:12 | skipping: no hosts matched
2020-08-18 19:51:12 |
2020-08-18 19:51:12 | PLAY [Overcloud deploy step tasks for 3] ***************************************
2020-08-18 19:51:12 |
2020-08-18 19:51:12 | PLAY [Overcloud common deploy step tasks 3] ************************************
2020-08-18 19:51:12 |
2020-08-18 19:51:12 | PLAY [External deployment step 4] **********************************************
2020-08-18 19:51:12 | skipping: no hosts matched
http://cougar11.scl.lab.tlv.redhat.com/DFG-upgrades-ffu-ffu-upgrade-13-16.1_director-rhel-virthost-3cont_2comp_3ceph-ipv4-vxlan-HA/29/undercloud-0.tar.gz?undercloud-0/home/stack/overcloud_system_upgrade_controller-0,ceph-0.log
openstack-tripleo-heat-templates-11.3.2-0.20200616081539.396affd.el8ost.noarch
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Red Hat OpenStack Platform 16.1 director bug fix advisory), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2020:3542 |
Description of problem: system_upgrade on bootstrap node fails with below error. # openstack overcloud upgrade run --tags system_upgrade --limit controller-1 TASK [include_tasks] *********************************************************** Monday 29 June 2020 07:59:53 -0400 (0:00:00.072) 0:01:12.922 *********** skipping: [controller-1] => {"changed": false, "skip_reason": "Conditional result was False"} TASK [include_tasks] *********************************************************** Monday 29 June 2020 07:59:53 -0400 (0:00:00.078) 0:01:13.000 *********** skipping: [controller-1] => {"changed": false, "skip_reason": "Conditional result was False"} PLAY RECAP ********************************************************************* controller-1 : ok=22 changed=4 unreachable=0 failed=0 skipped=24 rescued=0 ignored=0 Monday 29 June 2020 07:59:53 -0400 (0:00:00.058) 0:01:13.059 *********** =============================================================================== Updated nodes - controller-1 Success 2020-06-29 07:59:54.477 1045725 WARNING tripleoclient.plugin [-] Waiting for messages on queue 'tripleo' with no timeout. Using /var/lib/mistral/5546c481-3cf6-41ad-a8a6-30fe55c001b3/ansible.cfg as config file PLAY [Gather facts from undercloud] ******************************************** skipping: no hosts matched PLAY [Gather facts from overcloud] ********************************************* PLAY [Load global variables] *************************************************** TASK [include_vars] ************************************************************ Monday 29 June 2020 08:00:01 -0400 (0:00:00.145) 0:00:00.145 *********** [WARNING]: log file at /var/log/containers/mistral/package_update.log is not writeable and we cannot create it, aborting [WARNING]: Failure using method (v2_runner_on_start) in callback plugin (<ansible.plugins.callback.tripleo.CallbackModule object at 0x7f15dd2b2ef0>): 'show_per_host_start' ok: [controller-1] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} PLAY [Ensure /var/lib/config-data context] ************************************* TASK [Ensure /var/lib/config-data context] ************************************* Monday 29 June 2020 08:00:01 -0400 (0:00:00.079) 0:00:00.225 *********** changed: [controller-1] => {"ansible_facts": {"discovered_interpreter_python": "/usr/bin/python"}, "changed": true, "cmd": "set -o pipefail\nif [[ -e /var/lib/config-data ]]; then\n chcon -R -t svirt_sandbox_file_t /var/lib/config-data\nexit 2\nfi", "delta": "0:00:02.055794", "end": "2020-06-29 12:00:04.454095", "failed_when_result": false, "msg": "non-zero return code", "rc": 2, "start": "2020-06-29 12:00:02.398301", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []} PLAY [Render all_nodes data as group_vars for overcloud] *********************** TASK [Render all_nodes data as group_vars for overcloud] *********************** Monday 29 June 2020 08:00:04 -0400 (0:00:02.856) 0:00:03.081 *********** ok: [controller-1 -> localhost] => {"all_nodes": null, "changed": false} PLAY [Set all_nodes data as group_vars for overcloud] ************************** TASK [Set all_nodes data as group_vars for overcloud] ************************** Monday 29 June 2020 08:00:08 -0400 (0:00:03.602) 0:00:06.683 *********** ok: [controller-1] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false} PLAY [Manage SELinux] ********************************************************** PLAY [Generate /etc/hosts] ***************************************************** PLAY [Common roles for TripleO servers] **************************************** TASK [tripleo-bootstrap : Gather variables for each operating system] ********** Monday 29 June 2020 08:00:08 -0400 (0:00:00.110) 0:00:06.794 *********** fatal: [controller-1]: FAILED! => {"msg": "No file was found when using first_found. Use errors='ignore' to allow this task to be skipped if no files are found"} NO MORE HOSTS LEFT ************************************************************* PLAY RECAP ********************************************************************* controller-1 : ok=4 changed=1 unreachable=0 failed=1 skipped=0 rescued=0 ignored=0 Monday 29 June 2020 08:00:08 -0400 (0:00:00.059) 0:00:06.854 *********** =============================================================================== Ansible failed, check log at /var/log/mistral/package_update.log. 2020-06-29 08:00:08.977 1045725 ERROR tripleoclient.v1.overcloud_upgrade.MajorUpgradeRun [-] Exception occured while running the command: RuntimeError: Update failed with: Ansible failed, check log at /var/log/mistral/package_update.log. 2020-06-29 08:00:08.977 1045725 ERROR tripleoclient.v1.overcloud_upgrade.MajorUpgradeRun Traceback (most recent call last): 2020-06-29 08:00:08.977 1045725 ERROR tripleoclient.v1.overcloud_upgrade.MajorUpgradeRun File "/usr/lib/python3.6/site-packages/tripleoclient/command.py", line 32, in run 2020-06-29 08:00:08.977 1045725 ERROR tripleoclient.v1.overcloud_upgrade.MajorUpgradeRun super(Command, self).run(parsed_args) 2020-06-29 08:00:08.977 1045725 ERROR tripleoclient.v1.overcloud_upgrade.MajorUpgradeRun File "/usr/lib/python3.6/site-packages/osc_lib/command/command.py", line 41, in run 2020-06-29 08:00:08.977 1045725 ERROR tripleoclient.v1.overcloud_upgrade.MajorUpgradeRun return super(Command, self).run(parsed_args) 2020-06-29 08:00:08.977 1045725 ERROR tripleoclient.v1.overcloud_upgrade.MajorUpgradeRun File "/usr/lib/python3.6/site-packages/cliff/command.py", line 185, in run 2020-06-29 08:00:08.977 1045725 ERROR tripleoclient.v1.overcloud_upgrade.MajorUpgradeRun return_code = self.take_action(parsed_args) or 0 2020-06-29 08:00:08.977 1045725 ERROR tripleoclient.v1.overcloud_upgrade.MajorUpgradeRun File "/usr/lib/python3.6/site-packages/tripleoclient/v1/overcloud_upgrade.py", line 236, in take_action 2020-06-29 08:00:08.977 1045725 ERROR tripleoclient.v1.overcloud_upgrade.MajorUpgradeRun priv_key=key) 2020-06-29 08:00:08.977 1045725 ERROR tripleoclient.v1.overcloud_upgrade.MajorUpgradeRun File "/usr/lib/python3.6/site-packages/tripleoclient/utils.py", line 1245, in run_update_ansible_action 2020-06-29 08:00:08.977 1045725 ERROR tripleoclient.v1.overcloud_upgrade.MajorUpgradeRun verbosity=verbosity, extra_vars=extra_vars) 2020-06-29 08:00:08.977 1045725 ERROR tripleoclient.v1.overcloud_upgrade.MajorUpgradeRun File "/usr/lib/python3.6/site-packages/tripleoclient/workflows/package_update.py", line 127, in update_ansible 2020-06-29 08:00:08.977 1045725 ERROR tripleoclient.v1.overcloud_upgrade.MajorUpgradeRun raise RuntimeError('Update failed with: {}'.format(payload['message'])) 2020-06-29 08:00:08.977 1045725 ERROR tripleoclient.v1.overcloud_upgrade.MajorUpgradeRun RuntimeError: Update failed with: Ansible failed, check log at /var/log/mistral/package_update.log. 2020-06-29 08:00:08.977 1045725 ERROR tripleoclient.v1.overcloud_upgrade.MajorUpgradeRun 2020-06-29 08:00:08.979 1045725 ERROR openstack [-] Update failed with: Ansible failed, check log at /var/log/mistral/package_update.log.: RuntimeError: Update failed with: Ansible failed, check log at /var/log/mistral/package_update.log. 2020-06-29 08:00:08.980 1045725 INFO osc_lib.shell [-] END return value: 1 sys:1: ResourceWarning: unclosed <socket.socket fd=4, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=6, laddr=('172.16.0.1', 59992), raddr=('172.16.0.1', 5000)> sys:1: ResourceWarning: unclosed <socket.socket fd=6, family=AddressFamily.AF_INET, type=SocketKind.SOCK_STREAM, proto=6, laddr=('172.16.0.1', 48172), raddr=('172.16.0.1', 8989)> Version-Release number of selected component (if applicable): How reproducible: Steps to Reproduce: 1. 2. 3. Actual results: Expected results: Additional info: