Description of problem: During OSP16 update, the container update prepare container mistral task from: openstack overcloud external-update run \ --stack qe-Cloud-0 \ --tags container_image_prepare 2>&1 fails with: 2020-01-07 01:14:25 | 2020-01-07 01:14:25.850 66486 ERROR tripleoclient.v1.overcloud_external_update.ExternalUpdateRun [-] Exception occured while running the command: RuntimeError: Update failed with: Ansible failed, check log at /var/lib/mistral/63de7343-1ffe-4625-b1b7-abc5fce1ff6d/ansible.log. That message is useless and we can only find what happen in the mistral executor logs 2020-01-07 01:14:25.213 8 WARNING mistral.executors.default_executor [req-0468ec87-59d6-47b7-8d6c-833261b89b4f 8897281a0cc5408aa6786dfdc3867e9a 399874a6a68341828a1538e88a0f0d50 - default default] The action raised an exception [action_ex_id=63a4c835-f41e-48f0-9091-a235b3dbc9f7, msg='Container GET failed: https://192.168.24.2:13808/v1/AUTH_399874a6a68341828a1538e88a0f0d50/overcloud-config?format=json 404 Not Found [first 60 chars of response] b'<html><h1>Not Found</h1><p>The resource could not be found.<'', action_cls='<class 'mistral.actions.action_factory.DownloadConfigAction'>', attributes='{}', params='{'work_dir': '/var/lib/mistral/63de7343-1ffe-4625-b1b7-abc5fce1ff6d', 'container_config': 'overcloud-config'}']: swiftclient.exceptions.ClientException: Container GET failed: https://192.168.24.2:13808/v1/AUTH_399874a6a68341828a1538e88a0f0d50/overcloud-config?format=json 404 Not Found [first 60 chars of response] b'<html><h1>Not Found</h1><p>The resource could not be found.<' 2020-01-07 01:14:25.213 8 ERROR mistral.executors.default_executor Traceback (most recent call last): 2020-01-07 01:14:25.213 8 ERROR mistral.executors.default_executor File "/usr/lib/python3.6/site-packages/mistral/executors/default_executor.py", line 133, in _do_run_action 2020-01-07 01:14:25.213 8 ERROR mistral.executors.default_executor result = action.run(action_ctx) 2020-01-07 01:14:25.213 8 ERROR mistral.executors.default_executor File "/usr/lib/python3.6/site-packages/tripleo_common/actions/config.py", line 124, in run 2020-01-07 01:14:25.213 8 ERROR mistral.executors.default_executor self.work_dir) 2020-01-07 01:14:25.213 8 ERROR mistral.executors.default_executor File "/usr/lib/python3.6/site-packages/tripleo_common/utils/swift.py", line 61, in download_container 2020-01-07 01:14:25.213 8 ERROR mistral.executors.default_executor objects = swiftclient.get_container(container)[1] 2020-01-07 01:14:25.213 8 ERROR mistral.executors.default_executor File "/usr/lib/python3.6/site-packages/swiftclient/client.py", line 1820, in get_container 2020-01-07 01:14:25.213 8 ERROR mistral.executors.default_executor query_string=query_string) 2020-01-07 01:14:25.213 8 ERROR mistral.executors.default_executor File "/usr/lib/python3.6/site-packages/swiftclient/client.py", line 1739, in _retry 2020-01-07 01:14:25.213 8 ERROR mistral.executors.default_executor service_token=self.service_token, **kwargs) 2020-01-07 01:14:25.213 8 ERROR mistral.executors.default_executor File "/usr/lib/python3.6/site-packages/swiftclient/client.py", line 993, in get_container 2020-01-07 01:14:25.213 8 ERROR mistral.executors.default_executor raise ClientException.from_response(resp, 'Container GET failed', body) 2020-01-07 01:14:25.213 8 ERROR mistral.executors.default_executor swiftclient.exceptions.ClientException: Container GET failed: https://192.168.24.2:13808/v1/AUTH_399874a6a68341828a1538e88a0f0d50/overcloud-config?format=json 404 Not Found [first 60 chars of response] b'<html><h1>Not Found</h1><p>The resource could not be found.<' 2020-01-07 01:14:25.213 8 ERROR mistral.executors.default_executor This indicate a problem to access the overcloud config on the undercloud in swift. Looking at the swift log: Jan 7 01:14:25 undercloud-0 container-server[6565]: 192.168.24.1 - - [07/Jan/2020:01:14:25 +0000] "HEAD /d1/641/AUTH_399874a6a68341828a1538e88a0f0d50/overcloud-config" 404 - "HEAD http://192.168.24.2:13808/v1/AUTH_399874a6a68341828a1538e88a0f0d50/overcloud-config?format=json&states=listing" "tx6748947592414104a11be-005e13db71" "proxy-server 22" 0.0007 "-" 20 0 Jan 7 01:14:25 undercloud-0 proxy-server[8974]: - - 07/Jan/2020/01/14/25 HEAD /v1/AUTH_399874a6a68341828a1538e88a0f0d50/overcloud-config%3Fformat%3Djson%26states%3Dlisting HTTP/1.0 404 - Swift - - - - tx6748947592414104a11be-005e13db71 - 0.0059 RL - 1578359665.129089117 1578359665.134943962 0 Jan 7 01:14:25 undercloud-0 container-server[6565]: 192.168.24.1 - - [07/Jan/2020:01:14:25 +0000] "GET /d1/641/AUTH_399874a6a68341828a1538e88a0f0d50/overcloud-config" 404 - "GET http://192.168.24.2:13808/v1/AUTH_399874a6a68341828a1538e88a0f0d50/overcloud-config?format=json&states=listing" "tx6748947592414104a11be-005e13db71" "proxy-server 22" 0.0008 "-" 20 0 Jan 7 01:14:25 undercloud-0 proxy-server[8974]: 192.168.24.1 192.168.24.1 07/Jan/2020/01/14/25 GET /v1/AUTH_399874a6a68341828a1538e88a0f0d50/overcloud-config%3Fformat%3Djson%26states%3Dlisting HTTP/1.0 404 - python-swiftclient-3.8.1 gAAAAABeE9twkGEo... - 70 - tx6748947592414104a11be-005e13db71 - 0.0828 - - 1578359665.127118826 1578359665.209952831 0 We can see that those request are met with 404. Looking around we see that the right path is: /v1/AUTH_399874a6a68341828a1538e88a0f0d50/qe-Cloud-0-config/ not ../overcloud-config. So it looks like the name of the overcloud is not correctly used there as the overcloud's name is qe-Cloud-0, not overcloud. Certainly linked to https://review.opendev.org/#/q/change:Ib25d8f8fbfde8fb13bd6518fd2383b7e3ae7abee See my comment there: https://review.opendev.org/#/c/700051/1/tripleoclient/v1/overcloud_external_update.py@120 Version-Release number of selected component (if applicable): See full job there https://rhos-qe-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/view/DFG/view/upgrades/view/update/job/DFG-upgrades-updates-16-from-passed_phase1-HA-ipv4/37/ "Dummy" update from RHOS_TRUNK-16.0-RHEL-8-20200103.n.1 to RHOS_TRUNK-16.0-RHEL-8-20200103.n.1 How reproducible: always, as long as the overcloud's name is not "overcloud"
Hi, after a quick discussion with James (thanks), it appears that the puddle RHOS_TRUNK-16.0-RHEL-8-20200103.n.1 has the tripleo-common[1] part of the patch but not the python-tripleoclient part[2]. We need python-tripleoclient-12.3.1-0.20191230195937.585fb28.el8ost to match the change in tripleo-common after 1a8ec6fe. This is also the case for RHOS_TRUNK-16.0-RHEL-8-20200107.n.3. Basically we need a new puddle with that version of tripleoclient. [1] https://review.opendev.org/#/c/700050/ [2] https://review.opendev.org/#/c/700051/
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHEA-2020:0283