Bug 1787459
| Summary: | [OSp16] Overcloud Minor update failed when preparing container images overcloud_external_update.ExternalUpdateRun | ||
|---|---|---|---|
| Product: | Red Hat OpenStack | Reporter: | Yuri Obshansky <yobshans> |
| Component: | python-tripleoclient | Assignee: | RHOS Maint <rhos-maint> |
| Status: | CLOSED CURRENTRELEASE | QA Contact: | Jad Haj Yahya <jhajyahy> |
| Severity: | high | Docs Contact: | |
| Priority: | high | ||
| Version: | 16.0 (Train) | CC: | ccamposr, hbrock, jhajyahy, jslagle, mburns, ramishra, rrasouli |
| Target Milestone: | z1 | Keywords: | TestOnly, Triaged |
| Target Release: | 16.0 (Train on RHEL 8.1) | ||
| Hardware: | Unspecified | ||
| OS: | Unspecified | ||
| Whiteboard: | |||
| Fixed In Version: | openstack-tripleo-common-11.3.2-0.20191220201330.c653a6f.el8ost python-tripleoclient-12.3.1-0.20191230195937.585fb28.el8ost | Doc Type: | If docs needed, set a value |
| Doc Text: | Story Points: | --- | |
| Clone Of: | Environment: | ||
| Last Closed: | 2020-02-14 11:49:57 UTC | Type: | Bug |
| Regression: | --- | Mount Type: | --- |
| Documentation: | --- | CRM: | |
| Verified Versions: | Category: | --- | |
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
| Cloudforms Team: | --- | Target Upstream Version: | |
| Embargoed: | |||
This had been fixed with https://review.opendev.org/#/q/topic:bug/1854991-stable/train+(status:open+OR+status:merged), python-tripleoclient patch merged on 30th Dec. It won't be available with RHOS_TRUNK-16.0-RHEL-8-20191224.n.0. Please try a latest compose (ex. TEST-RHOS_TRUNK-16.0-RHEL-8-20200103.n.0/). Thank you, Rabi. Will run job with latest TEST-RHOS_TRUNK-16.0-RHEL-8-20200103.n.0 and update with results. *** Bug 1787592 has been marked as a duplicate of this bug. *** Cannot test compose RHOS_TRUNK-16.0-RHEL-8-20200103.n.0 http://rhos-qe-mirror-tlv.usersys.redhat.com/rcm-guest/puddles/OpenStack/16.0-RHEL-8/RHOS_TRUNK-16.0-RHEL-8-20200103.n.0 There is no container_image_prepare.yaml file Undercloud update failed on TASK [Downloading overcloud container images environment file] ***************** task path: /home/rhos-ci/jenkins/workspace/DFG-edge-day2-16-rhel-virthost-3cont-2comp-2leafs-x-2comp-update/infrared/plugins/tripleo-undercloud/tasks/prepare_uc_images.yml:70 Friday 03 January 2020 19:59:11 +0000 (0:00:00.386) 0:02:36.021 ******** fatal: [site-undercloud-0 -> localhost]: FAILED! => { "changed": false, "dest": "/home/rhos-ci/jenkins/workspace/DFG-edge-day2-16-rhel-virthost-3cont-2comp-2leafs-x-2comp-update/infrared/.workspaces/workspace_2020-01-03_15-20-07/container_image_prepare.yaml", "gid": 1000, "group": "rhos-ci", "mode": "0664", "owner": "rhos-ci", "response": "HTTP Error 404: Not Found", "secontext": "unconfined_u:object_r:user_home_t:s0", "size": 1080, "state": "file", "status_code": 404, "uid": 1000, "url": "http://rhos-qe-mirror-tlv.usersys.redhat.com/rcm-guest/puddles/OpenStack/16.0-RHEL-8/RHOS_TRUNK-16.0-RHEL-8-20200103.n.0/container_image_prepare.yaml" } MSG: Request failed Will wait till new compose with container_image_prepare.yaml file https://rhos-ci-staging-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/view/DFG/view/edge/job/DFG-edge-day2-16-rhel-virthost-3cont-2comp-2leafs-x-2comp-update/36/console Looks similar to https://bugs.launchpad.net/tripleo/+bug/1840691 According to our records, this should be resolved by openstack-tripleo-common-11.3.3-0.20200121231250.3c68b48.el8ost. This build is available now. According to our records, this should be resolved by python-tripleoclient-12.3.2-0.20200130192329.78ac810.el8ost. This build is available now. |
Description of problem: OSP 16 DCN multi stack with Spine-Leaf network topology Overcloud minor update failed when run overcloud_prepare_containers.sh for "central" stack. [stack@site-undercloud-0 ~]$ cat overcloud_prepare_containers.sh #!/bin/env bash # # Prepare conatainer images for overcloud update/upgrade # set -euo pipefail source /home/stack/stackrc echo "[$(date)] Start preparing container images for overcloud update" openstack overcloud external-update run \ --stack central \ --tags container_image_prepare 2>&1 [stack@site-undercloud-0 ~]$ cat overcloud_update_prepare_containers.log 2020-01-02 00:04:43 | [Thu Jan 2 00:04:43 UTC 2020] Start preparing container images for overcloud update 2020-01-02 00:04:44 | 2020-01-02 00:04:44.530 85153 INFO osc_lib.shell [-] command: overcloud external-update run -> tripleoclient.v1.overcloud_external_update.ExternalUpdateRun (auth=True) 2020-01-02 00:04:44 | 2020-01-02 00:04:44.531 85153 INFO osc_lib.clientmanager [-] Using auth plugin: password 2020-01-02 00:04:44 | 2020-01-02 00:04:44.531 85153 DEBUG osc_lib.clientmanager [-] Using parameters {'auth_url': 'https://192.168.24.2:13000', 'project_name': 'admin', 'project_domain_name': 'Default', 'username': 'admin', 'user_domain_name': 'Default', 'password': '***'} setup_auth /usr/lib/python3.6/site-packages/osc_lib/clientmanager.py:157 2020-01-02 00:04:44 | 2020-01-02 00:04:44.533 85153 DEBUG osc_lib.clientmanager [-] Get auth_ref auth_ref /usr/lib/python3.6/site-packages/osc_lib/clientmanager.py:201 2020-01-02 00:04:49 | 2020-01-02 00:04:49.159 85153 WARNING tripleoclient.plugin [-] Waiting for messages on queue 'tripleo' with no timeout. 2020-01-02 00:04:49 | Ansible failed, check log at /var/lib/mistral/2d95f968-2ec3-4137-8a57-4279367ab46d/ansible.log. 2020-01-02 00:04:49 | 2020-01-02 00:04:49.862 85153 ERROR tripleoclient.v1.overcloud_external_update.ExternalUpdateRun [-] Exception occured while running the command: RuntimeError: Update failed with: Ansible failed, check log at /var/lib/mistral/2d95f968-2ec3-4137-8a57-4279367ab46d/ansible.log. 2020-01-02 00:04:49 | 2020-01-02 00:04:49.862 85153 ERROR tripleoclient.v1.overcloud_external_update.ExternalUpdateRun Traceback (most recent call last): 2020-01-02 00:04:49 | 2020-01-02 00:04:49.862 85153 ERROR tripleoclient.v1.overcloud_external_update.ExternalUpdateRun File "/usr/lib/python3.6/site-packages/tripleoclient/command.py", line 32, in run 2020-01-02 00:04:49 | 2020-01-02 00:04:49.862 85153 ERROR tripleoclient.v1.overcloud_external_update.ExternalUpdateRun super(Command, self).run(parsed_args) 2020-01-02 00:04:49 | 2020-01-02 00:04:49.862 85153 ERROR tripleoclient.v1.overcloud_external_update.ExternalUpdateRun File "/usr/lib/python3.6/site-packages/osc_lib/command/command.py", line 41, in run 2020-01-02 00:04:49 | 2020-01-02 00:04:49.862 85153 ERROR tripleoclient.v1.overcloud_external_update.ExternalUpdateRun return super(Command, self).run(parsed_args) 2020-01-02 00:04:49 | 2020-01-02 00:04:49.862 85153 ERROR tripleoclient.v1.overcloud_external_update.ExternalUpdateRun File "/usr/lib/python3.6/site-packages/cliff/command.py", line 185, in run 2020-01-02 00:04:49 | 2020-01-02 00:04:49.862 85153 ERROR tripleoclient.v1.overcloud_external_update.ExternalUpdateRun return_code = self.take_action(parsed_args) or 0 2020-01-02 00:04:49 | 2020-01-02 00:04:49.862 85153 ERROR tripleoclient.v1.overcloud_external_update.ExternalUpdateRun File "/usr/lib/python3.6/site-packages/tripleoclient/v1/overcloud_external_update.py", line 125, in take_action 2020-01-02 00:04:49 | 2020-01-02 00:04:49.862 85153 ERROR tripleoclient.v1.overcloud_external_update.ExternalUpdateRun priv_key=key) 2020-01-02 00:04:49 | 2020-01-02 00:04:49.862 85153 ERROR tripleoclient.v1.overcloud_external_update.ExternalUpdateRun File "/usr/lib/python3.6/site-packages/tripleoclient/utils.py", line 1194, in run_update_ansible_action 2020-01-02 00:04:49 | 2020-01-02 00:04:49.862 85153 ERROR tripleoclient.v1.overcloud_external_update.ExternalUpdateRun verbosity=verbosity, extra_vars=extra_vars) 2020-01-02 00:04:49 | 2020-01-02 00:04:49.862 85153 ERROR tripleoclient.v1.overcloud_external_update.ExternalUpdateRun File "/usr/lib/python3.6/site-packages/tripleoclient/workflows/package_update.py", line 127, in update_ansible 2020-01-02 00:04:49 | 2020-01-02 00:04:49.862 85153 ERROR tripleoclient.v1.overcloud_external_update.ExternalUpdateRun raise RuntimeError('Update failed with: {}'.format(payload['message'])) 2020-01-02 00:04:49 | 2020-01-02 00:04:49.862 85153 ERROR tripleoclient.v1.overcloud_external_update.ExternalUpdateRun RuntimeError: Update failed with: Ansible failed, check log at /var/lib/mistral/2d95f968-2ec3-4137-8a57-4279367ab46d/ansible.log. 2020-01-02 00:04:49 | 2020-01-02 00:04:49.862 85153 ERROR tripleoclient.v1.overcloud_external_update.ExternalUpdateRun 2020-01-02 00:04:49 | 2020-01-02 00:04:49.867 85153 ERROR openstack [-] Update failed with: Ansible failed, check log at /var/lib/mistral/2d95f968-2ec3-4137-8a57-4279367ab46d/ansible.log.: RuntimeError: Update failed with: Ansible failed, check log at /var/lib/mistral/2d95f968-2ec3-4137-8a57-4279367ab46d/ansible.log. 2020-01-02 00:04:49 | 2020-01-02 00:04:49.867 85153 INFO osc_lib.shell [-] END return value: 1 Version-Release number of selected component (if applicable): Minor update from RHOS_TRUNK-16.0-RHEL-8-20191213.n.5 to RHOS_TRUNK-16.0-RHEL-8-20191224.n.0 How reproducible: Jenkins job https://rhos-ci-staging-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/view/DFG/view/edge/job/DFG-edge-day2-16-rhel-virthost-3cont-2comp-2leafs-x-2comp-update/35/consoleFull Steps to Reproduce: 1. 2. 3. Actual results: Expected results: Additional info: I will hold an environment for a while for an investigation. Please, ping me on IIRC