Description of problem: When upgrading from OSP15 to OSP16, more specifically, during the Controller upgrade step. The upgrade fails when trying to start the placement_wait_for_service container in the bootstrap Controller. The output error from the container is the following: 2020-03-25T17:25:01.586984244+00:00 stderr F + command -v python3 2020-03-25T17:25:01.586984244+00:00 stderr F + python3 /container-config-scripts/placement_wait_for_service.py 2020-03-25T17:25:02.438304557+00:00 stdout F DEBUG:keystoneauth.identity.v3.base:Making authentication request to http://172.17.1.90:5000/v3/auth/tokens 2020-03-25T17:25:02.443745403+00:00 stdout F DEBUG:urllib3.connectionpool:Starting new HTTP connection (1): 172.17.1.90:5000 2020-03-25T17:25:02.940318452+00:00 stdout F DEBUG:urllib3.connectionpool:http://172.17.1.90:5000 "POST /v3/auth/tokens HTTP/1.1" 401 109 2020-03-25T17:25:02.940318452+00:00 stdout F DEBUG:keystoneauth.session:Request returned failure status: 401 2020-03-25T17:25:02.944164975+00:00 stdout F ERROR:placement_wait_for_service:Retry - Failed to get placement service endpoint: 2020-03-25T17:25:02.944164975+00:00 stdout F Traceback (most recent call last): 2020-03-25T17:25:02.944164975+00:00 stdout F File "/container-config-scripts/placement_wait_for_service.py", line 91, in <module> 2020-03-25T17:25:02.944164975+00:00 stdout F name='placement')[0].id 2020-03-25T17:25:02.944164975+00:00 stdout F File "/usr/lib/python3.6/site-packages/keystoneclient/v3/services.py", line 93, in list 2020-03-25T17:25:02.944164975+00:00 stdout F **kwargs) 2020-03-25T17:25:02.944164975+00:00 stdout F File "/usr/lib/python3.6/site-packages/keystoneclient/base.py", line 86, in func 2020-03-25T17:25:02.944164975+00:00 stdout F return f(*args, **new_kwargs) 2020-03-25T17:25:02.944164975+00:00 stdout F File "/usr/lib/python3.6/site-packages/keystoneclient/base.py", line 448, in list 2020-03-25T17:25:02.944164975+00:00 stdout F list_resp = self._list(url_query, self.collection_key) 2020-03-25T17:25:02.944164975+00:00 stdout F File "/usr/lib/python3.6/site-packages/keystoneclient/base.py", line 141, in _list 2020-03-25T17:25:02.944164975+00:00 stdout F resp, body = self.client.get(url, **kwargs) 2020-03-25T17:25:02.944164975+00:00 stdout F File "/usr/lib/python3.6/site-packages/keystoneauth1/adapter.py", line 386, in get 2020-03-25T17:25:02.944164975+00:00 stdout F return self.request(url, 'GET', **kwargs) 2020-03-25T17:25:02.944164975+00:00 stdout F File "/usr/lib/python3.6/site-packages/keystoneauth1/adapter.py", line 545, in request 2020-03-25T17:25:02.944164975+00:00 stdout F resp = super(LegacyJsonAdapter, self).request(*args, **kwargs) 2020-03-25T17:25:02.944164975+00:00 stdout F File "/usr/lib/python3.6/site-packages/keystoneauth1/adapter.py", line 248, in request 2020-03-25T17:25:02.944164975+00:00 stdout F return self.session.request(url, method, **kwargs) 2020-03-25T17:25:02.944164975+00:00 stdout F File "/usr/lib/python3.6/site-packages/keystoneauth1/session.py", line 747, in request 2020-03-25T17:25:02.944164975+00:00 stdout F auth_headers = self.get_auth_headers(auth) 2020-03-25T17:25:02.944164975+00:00 stdout F File "/usr/lib/python3.6/site-packages/keystoneauth1/session.py", line 1158, in get_auth_headers 2020-03-25T17:25:02.944164975+00:00 stdout F return auth.get_headers(self, **kwargs) 2020-03-25T17:25:02.944164975+00:00 stdout F File "/usr/lib/python3.6/site-packages/keystoneauth1/plugin.py", line 95, in get_headers 2020-03-25T17:25:02.944164975+00:00 stdout F token = self.get_token(session) 2020-03-25T17:25:02.944164975+00:00 stdout F File "/usr/lib/python3.6/site-packages/keystoneauth1/identity/base.py", line 88, in get_token 2020-03-25T17:25:02.944164975+00:00 stdout F return self.get_access(session).auth_token 2020-03-25T17:25:02.944164975+00:00 stdout F File "/usr/lib/python3.6/site-packages/keystoneauth1/identity/base.py", line 134, in get_access 2020-03-25T17:25:02.944164975+00:00 stdout F self.auth_ref = self.get_auth_ref(session) 2020-03-25T17:25:02.944164975+00:00 stdout F File "/usr/lib/python3.6/site-packages/keystoneauth1/identity/v3/base.py", line 184, in get_auth_ref 2020-03-25T17:25:02.944164975+00:00 stdout F authenticated=False, log=False, **rkwargs) 2020-03-25T17:25:02.944164975+00:00 stdout F File "/usr/lib/python3.6/site-packages/keystoneauth1/session.py", line 1106, in post 2020-03-25T17:25:02.944164975+00:00 stdout F return self.request(url, 'POST', **kwargs) 2020-03-25T17:25:02.944164975+00:00 stdout F File "/usr/lib/python3.6/site-packages/keystoneauth1/session.py", line 943, in request 2020-03-25T17:25:02.944164975+00:00 stdout F raise exceptions.from_response(resp, method, url) 2020-03-25T17:25:02.944164975+00:00 stdout F keystoneauth1.exceptions.http.Unauthorized: The request you have made requires authentication. (HTTP 401) (Request-ID: req-0efa4b58-8879-438e-9114-8350827f46e8) 2020-03-25T17:25:12.952559392+00:00 stdout F DEBUG:keystoneauth.identity.v3.base:Making authentication request to http://172.17.1.90:5000/v3/auth/tokens 2020-03-25T17:25:13.355553668+00:00 stdout F DEBUG:urllib3.connectionpool:http://172.17.1.90:5000 "POST /v3/auth/tokens HTTP/1.1" 401 109 2020-03-25T17:25:13.355553668+00:00 stdout F DEBUG:keystoneauth.session:Request returned failure status: 401 2020-03-25T17:25:13.355553668+00:00 stdout F ERROR:placement_wait_for_service:Retry - Failed to get placement service endpoint: 2020-03-25T17:25:13.355553668+00:00 stdout F Traceback (most recent call last): 2020-03-25T17:25:13.355553668+00:00 stdout F File "/container-config-scripts/placement_wait_for_service.py", line 91, in <module> 2020-03-25T17:25:13.355553668+00:00 stdout F name='placement')[0].id 2020-03-25T17:25:13.355553668+00:00 stdout F File "/usr/lib/python3.6/site-packages/keystoneclient/v3/services.py", line 93, in list 2020-03-25T17:25:13.355553668+00:00 stdout F **kwargs) And the error appears over and over. The endpoint list for the overcloud looks like this: (qe-Cloud-0) [stack@undercloud-0 ~]$ openstack endpoint list +----------------------------------+-----------+--------------+----------------+---------+-----------+------------------------------------------------+ | ID | Region | Service Name | Service Type | Enabled | Interface | URL | +----------------------------------+-----------+--------------+----------------+---------+-----------+------------------------------------------------+ | 12f1801e7e3240a0b091570c8305b043 | regionOne | nova | compute | True | internal | http://172.17.1.90:8774/v2.1 | | 15f96146d42847129379dafdd1cff59c | regionOne | nova | compute | True | public | https://10.0.0.101:13774/v2.1 | | 1930c606abb94a46bf4b145924e770b6 | regionOne | heat | orchestration | True | internal | http://172.17.1.90:8004/v1/%(tenant_id)s | | 20cf73e1c8c54e899f4311a9cd9525dc | regionOne | neutron | network | True | internal | http://172.17.1.90:9696 | | 2a58d7ef1e4b4bd5a20145c5c0ea15b6 | regionOne | heat-cfn | cloudformation | True | public | https://10.0.0.101:13005/v1 | | 330a0aedf9274d9f99996396c65a084d | regionOne | cinderv3 | volumev3 | True | public | https://10.0.0.101:13776/v3/%(tenant_id)s | | 3fef471de548480a8763fb4c0dc9bb1d | regionOne | gnocchi | metric | True | admin | http://172.17.1.90:8041 | | 41334448e44940f18430ddc2aceb9910 | regionOne | nova | compute | True | admin | http://172.17.1.90:8774/v2.1 | | 42049732473b4345934caa301b7370b4 | regionOne | glance | image | True | public | https://10.0.0.101:13292 | | 52c8b921df224bdd8d1366621cd2fc19 | regionOne | heat | orchestration | True | public | https://10.0.0.101:13004/v1/%(tenant_id)s | | 5790d3612fbc4d3bb1263de50ddafba3 | regionOne | keystone | identity | True | internal | http://172.17.1.90:5000 | | 5828e5e6b23b43ccb2d05afca1c9d4a1 | regionOne | keystone | identity | True | admin | http://192.168.24.28:35357 | | 5da85f019da04059a0f1a41cb42a07f2 | regionOne | aodh | alarming | True | internal | http://172.17.1.90:8042 | | 636b7894b7cb46569eae14a20348cedd | regionOne | swift | object-store | True | internal | http://172.17.3.28:8080/v1/AUTH_%(tenant_id)s | | 674a5b54ae2b4ee782a0f5fdd7ed012b | regionOne | heat | orchestration | True | admin | http://172.17.1.90:8004/v1/%(tenant_id)s | | 7e38825df48c4e78b541a5f24ef13ff8 | regionOne | gnocchi | metric | True | internal | http://172.17.1.90:8041 | | 7fb00ddcc780445792b1eaf2afdc8d75 | regionOne | placement | placement | True | admin | http://172.17.1.90:8778/placement | | 84315e4a85bd47df81b6c1de9a7885de | regionOne | swift | object-store | True | public | https://10.0.0.101:13808/v1/AUTH_%(tenant_id)s | | 85d6b5b320cb4beba0349bfd2d854bc6 | regionOne | aodh | alarming | True | admin | http://172.17.1.90:8042 | | 8804814a42f9405d8c2e9b768c7bb49d | regionOne | cinderv2 | volumev2 | True | admin | http://172.17.1.90:8776/v2/%(tenant_id)s | | 946d5be7f11749fdbd5be1679efa138b | regionOne | heat-cfn | cloudformation | True | internal | http://172.17.1.90:8000/v1 | | 94f4fd8f0d404b98a1c8885cbd296d4d | regionOne | glance | image | True | internal | http://172.17.1.90:9292 | | 99c400d6022d458a87fbc61685996740 | regionOne | aodh | alarming | True | public | https://10.0.0.101:13042 | | 9b1bc2c630d84ad0859d105672bf1f61 | regionOne | placement | placement | True | public | https://10.0.0.101:13778/placement | | 9c62f655aabc4ad887fe8d2d76ebe0ba | regionOne | cinderv2 | volumev2 | True | public | https://10.0.0.101:13776/v2/%(tenant_id)s | | a63d07bb6c9d4d1ca8fcfd49129bfc15 | regionOne | gnocchi | metric | True | public | https://10.0.0.101:13041 | | a846c5cabd55493d812ca1624362382f | regionOne | swift | object-store | True | admin | http://172.17.3.28:8080 | | aa61dc6753844175b927077a1871b180 | regionOne | cinderv3 | volumev3 | True | internal | http://172.17.1.90:8776/v3/%(tenant_id)s | | b181d5d2ade4488ba2939e4a23810d3b | regionOne | panko | event | True | internal | http://172.17.1.90:8977 | | b3cb3904a5814d62b6a01aa9959e45cc | regionOne | cinderv2 | volumev2 | True | internal | http://172.17.1.90:8776/v2/%(tenant_id)s | | b54a232239714f5b8a8ecd9a3a696c6c | regionOne | neutron | network | True | admin | http://172.17.1.90:9696 | | c0536920d5df479884aa15e68407a8c7 | regionOne | neutron | network | True | public | https://10.0.0.101:13696 | | c4d8ec302d1b40fdaa9202a288c73aff | regionOne | placement | placement | True | internal | http://172.17.1.90:8778/placement | | ce481824eae842f49afe9acd52212acd | regionOne | panko | event | True | public | https://10.0.0.101:13977 | | d04f76474e9b4b41bdc8292492589908 | regionOne | glance | image | True | admin | http://172.17.1.90:9292 | | d3b56f1e4dcd4dc39037587180e80ddc | regionOne | heat-cfn | cloudformation | True | admin | http://172.17.1.90:8000/v1 | | e6aa00177b524dad819591d9df806328 | regionOne | panko | event | True | admin | http://172.17.1.90:8977 | | e9d2e8e9e62e4e1bb9cd6b95e8e7953c | regionOne | cinderv3 | volumev3 | True | admin | http://172.17.1.90:8776/v3/%(tenant_id)s | | fbb50519f6a5418d869f46a482fb8b01 | regionOne | keystone | identity | True | public | https://10.0.0.101:13000 | +----------------------------------+-----------+--------------+----------------+---------+-----------+------------------------------------------------+ And this is the keystone log at the same timestamp: 2020-03-26 11:17:23.702 135 DEBUG keystone.server.flask.request_processing.req_logging [req-0f0a411f-ac74-48c1-a619-55a4b69718d6 - - - - -] PATH_INFO: `/v3/auth/tokens` log_request_info /usr/lib/python3.6/site-packages/keystone/server/flask/request_processing/req_logging.py:29 2020-03-26 11:17:23.761 134 DEBUG keystone.server.flask.request_processing.req_logging [req-d8b1958a-faf8-4146-9649-cc62906b17b7 - - - - -] REQUEST_METHOD: `GET` log_request_info /usr/lib/python3.6/site-packages/keystone/server/flask/request_processing/req_logging.py:27 2020-03-26 11:17:23.761 138 DEBUG keystone.server.flask.request_processing.req_logging [req-f9d151ac-b6c6-42cc-84e0-2e4900279608 - - - - -] REQUEST_METHOD: `GET` log_request_info /usr/lib/python3.6/site-packages/keystone/server/flask/request_processing/req_logging.py:27 2020-03-26 11:17:23.762 138 DEBUG keystone.server.flask.request_processing.req_logging [req-f9d151ac-b6c6-42cc-84e0-2e4900279608 - - - - -] SCRIPT_NAME: `` log_request_info /usr/lib/python3.6/site-packages/keystone/server/flask/request_processing/req_logging.py:28 2020-03-26 11:17:23.762 134 DEBUG keystone.server.flask.request_processing.req_logging [req-d8b1958a-faf8-4146-9649-cc62906b17b7 - - - - -] SCRIPT_NAME: `` log_request_info /usr/lib/python3.6/site-packages/keystone/server/flask/request_processing/req_logging.py:28 2020-03-26 11:17:23.762 134 DEBUG keystone.server.flask.request_processing.req_logging [req-d8b1958a-faf8-4146-9649-cc62906b17b7 - - - - -] PATH_INFO: `/v3` log_request_info /usr/lib/python3.6/site-packages/keystone/server/flask/request_processing/req_logging.py:29 2020-03-26 11:17:23.762 138 DEBUG keystone.server.flask.request_processing.req_logging [req-f9d151ac-b6c6-42cc-84e0-2e4900279608 - - - - -] PATH_INFO: `/v3` log_request_info /usr/lib/python3.6/site-packages/keystone/server/flask/request_processing/req_logging.py:29 2020-03-26 11:17:24.076 135 WARNING keystone.server.flask.application [req-0f0a411f-ac74-48c1-a619-55a4b69718d6 - - - - -] Authorization failed. The request you have made requires authentication. from 172.17.1.128: keystone.exception.Unauthorized: The request you have made requires authentication. There is an environment with the issue reproduced in case somebod wants to debug it. Version-Release number of selected component (if applicable): How reproducible: Run CI job: https://rhos-qe-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/view/DFG/view/upgrades/view/upgrade/job/DFG-upgrades-upgrade-upgrade-15-16_director-rhel-virthost-3cont_2comp_3ceph-ipv4-geneve-HA/ Steps to Reproduce: 1. 2. 3. Actual results: The upgrade fails. Expected results: The upgrade succeeds. Additional info: (undercloud) [stack@undercloud-0 ~]$ sudo rpm -qa | grep tripleo-heat-templates openstack-tripleo-heat-templates-11.3.2-0.20200315025718.033aae9.el8ost.noarch
Could this change https://review.opendev.org/#/c/690795/ have anything to do with the failure?
placement password in keystone was not updated
It seems that this task was missing to run during deploy tasks playbook: https://github.com/openstack/tripleo-heat-templates/blob/master/deployment/keystone/keystone-container-puppet.yaml#L721-L755 mostly because the upgrade run command is running with the --limit Controller option and the External deploy tasks are run agains the Undercloud host: https://github.com/openstack/tripleo-heat-templates/blob/master/common/deploy-steps.j2#L654 Therefore, all the external deploy tasks are being skipped: 2020-03-25 16:56:41 | PLAY [External deployment step 1] ********************************************** 2020-03-25 16:56:41 | skipping: no hosts matched 2020-03-25 17:05:18 | PLAY [External deployment step 2] ********************************************** 2020-03-25 17:05:18 | skipping: no hosts matched 2020-03-25 17:15:45 | PLAY [External deployment step 3] ********************************************** 2020-03-25 17:15:45 | skipping: no hosts matched 2020-03-25 17:15:45 | 2020-03-25 17:20:01 | PLAY [External deployment step 4] ********************************************** 2020-03-25 17:20:01 | skipping: no hosts matched One option would be running the deploy_steps_playbook.yaml with --limit Undercloud always if the --limit option is used during the overcloud upgrade run command, but that could run some undesired steps.
(In reply to Ollie Walsh from comment #2) > placement password in keystone was not updated Since you linked [1], which is merged, I'm going to take a gamble and close this as UPSTREAM (there's no customer case to ask for a build with an nvr). If this is incorrect and there's still work to do here, please re-open. [1] https://review.opendev.org/708412
Placement service has now the right password and the service starts correctly: d86d35889b64 undercloud-0.ctlplane.redhat.local:8787/rh-osbs/rhosp16-openstack-nova-api:16.1_20200505.1 /usr/bin/bootstra... 2 hours ago Exited (0) 2 hours ago nova_api_ensure_default_cell 12.4MB (virtual 1.09GB) 40257cb54a97 undercloud-0.ctlplane.redhat.local:8787/rh-osbs/rhosp16-openstack-placement-api:16.1_20200505.1 /usr/bin/bootstra... 2 hours ago Exited (0) 2 hours ago placement_api_db_sync 12.4MB (virtual 590MB) 777c209decb1 undercloud-0.ctlplane.redhat.local:8787/rh-osbs/rhosp16-openstack-nova-api:16.1_20200505.1 /usr/bin/bootstra... 2 hours ago Exited (0) 2 hours ago nova_api_map_cell0 12.4MB (virtual 1.09GB) 008f375745a6 undercloud-0.ctlplane.redhat.local:8787/rh-osbs/rhosp16-openstack-placement-api:16.1_20200505.1 /usr/bin/bootstra... 2 hours ago Exited (0) 2 hours ago placement_api_db_extract_data_from_nova_api 12.4MB (virtual 590MB) Controllers do also get properly upgraded: 2020-05-08 02:56:14 | TASK [include_tasks] *********************************************************** 2020-05-08 02:56:14 | Friday 08 May 2020 02:56:12 +0000 (0:00:00.533) 0:01:11.786 ************ 2020-05-08 02:56:14 | skipping: [controller-0] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-05-08 02:56:14 | skipping: [controller-1] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-05-08 02:56:14 | skipping: [controller-2] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-05-08 02:56:14 | 2020-05-08 02:56:14 | TASK [include_tasks] *********************************************************** 2020-05-08 02:56:14 | Friday 08 May 2020 02:56:13 +0000 (0:00:00.526) 0:01:12.312 ************ 2020-05-08 02:56:14 | skipping: [controller-0] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-05-08 02:56:14 | skipping: [controller-1] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-05-08 02:56:14 | skipping: [controller-2] => {"changed": false, "skip_reason": "Conditional result was False"} 2020-05-08 02:56:14 | 2020-05-08 02:56:14 | PLAY RECAP ********************************************************************* 2020-05-08 02:56:14 | controller-0 : ok=17 changed=5 unreachable=0 failed=0 skipped=35 rescued=0 ignored=0 2020-05-08 02:56:14 | controller-1 : ok=15 changed=4 unreachable=0 failed=0 skipped=36 rescued=0 ignored=0 2020-05-08 02:56:14 | controller-2 : ok=15 changed=4 unreachable=0 failed=0 skipped=36 rescued=0 ignored=0 2020-05-08 02:56:14 | 2020-05-08 02:56:14 | Friday 08 May 2020 02:56:13 +0000 (0:00:00.354) 0:01:12.667 ************ 2020-05-08 02:56:14 | =============================================================================== 2020-05-08 02:56:14 | 2020-05-08 02:56:14 | Updated nodes - Controller 2020-05-08 02:56:14 | Success 2020-05-08 02:56:14 | 2020-05-08 02:56:14.491 75654 INFO tripleoclient.v1.overcloud_upgrade.MajorUpgradeRun [-] Completed Overcloud Upgrade Run for Controller with playbooks ['upgrade_steps_playbook.yaml', 'deploy_steps_playbook.yaml', 'post_upgrade_steps_playbook.yaml'] ^[[00m Tested with t-c package: openstack-tripleo-common-11.3.3-0.20200502033438.30ebd4d.el8ost.noarch Logs: https://rhos-qe-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/view/DFG/view/upgrades/view/upgrade/job/DFG-upgrades-upgrade-upgrade-15-16.1_director-rhel-virthost-3cont_2comp_3ceph-ipv4-geneve-HA/24/
If this bug requires doc text for errata release, please set the 'Doc Type' and provide draft text according to the template in the 'Doc Text' field. The documentation team will review, edit, and approve the text. If this bug does not require doc text, please set the 'requires_doc_text' flag to '-'.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2020:3148