Description of problem: ----------------------- Attempt to run 'openstack overcloud upgrade prepare' failed: openstack overcloud upgrade prepare --stack overcloud \ --templates /usr/share/openstack-tripleo-heat-templates \ -e /usr/share/openstack-tripleo-heat-templates/environments/ceph-ansible/ceph-ansible-external.yaml \ -e /home/stack/puma23-bm/ceph-external-custom.yaml \ -e /usr/share/openstack-tripleo-heat-templates/environments/network-isolation.yaml \ -e /home/stack/puma23-bm/network/network-environment.yaml \ -e /home/stack/puma23-bm/enable-tls.yaml \ -e /home/stack/puma23-bm/inject-trust-anchor.yaml \ -e /home/stack/puma23-bm/public_vip.yaml \ -e /usr/share/openstack-tripleo-heat-templates/environments/ssl/tls-endpoints-public-ip.yaml \ -e /home/stack/puma23-bm/hostnames.yml \ -e /home/stack/puma23-bm/nodes_data.yaml \ -e /home/stack/puma23-bm/debug.yaml \ -e /home/stack/puma23-bm/docker-images.yaml \ -e /home/stack/cli_opts_params.yaml \ -e /home/stack/puma23-bm/docker-images.yaml \ --roles-file /usr/share/openstack-tripleo-heat-templates/roles_data.yaml 2>&1 ... 2018-10-10 14:58:48.427 659310 INFO osc_lib.shell [-] command: overcloud upgrade prepare -> tripleoclient.v1.overcloud_upgrade.UpgradePrepare (auth=True)ESC[00m 2018-10-10 14:58:48.429 659310 INFO osc_lib.clientmanager [-] Using auth plugin: passwordESC[00m 2018-10-10 14:58:48.430 659310 DEBUG osc_lib.clientmanager [-] Using parameters {'username': 'admin', 'project_name': 'admin', 'user_domain_name': 'Default', 'auth_url': 'https://192.168.24.2:13000/', 'password': '***', 'project_domain_name': 'Default'} setup_auth /usr/lib/python2.7/site-packages/osc_lib/clientmanager.py:157ESC[00m 2018-10-10 14:58:48.431 659310 DEBUG osc_lib.clientmanager [-] Get auth_ref auth_ref /usr/lib/python2.7/site-packages/osc_lib/clientmanager.py:201ESC[00m 2018-10-10 14:58:56.276 659310 INFO tripleoclient.v1.overcloud_upgrade.MajorUpgradePrepare [-] Stack found, will be doing a stack updateESC[00m Removing the current plan files Uploading new plan files Plan updated. Processing templates in the directory /tmp/tripleoclient-CHg8Wh/tripleo-heat-templates WARNING: Following parameter(s) are deprecated and still defined. Deprecated parameters will be removed soon! OvercloudControlFlavor WARNING: Following parameter(s) are defined but not used in plan. Could be possible that parameter is valid but currently not used. DockerMysqlClientConfigImage DockerDesignateMDNSImage ... DockerDesignateCentralImage DockerManilaConfigImage 2018-10-10 15:08:00.976 659310 WARNING tripleoclient.plugin [-] Waiting for messages on queue 'tripleo' with no timeout.ESC[00m 2018-10-10 15:09:31.674 659310 ERROR openstack [-] {u'deployment_status': None, u'execution': {u'created_at': u'2018-10-10 12:08:00', u'id': u'02dfd243-c1dd-41f2-ac55-8453c3377394', u'input': {u'config_dir': u'/tmp/', u'container': u'overcloud', u'queue_name': u'tripleo', u'skip_deploy_identifier': False, u'timeout': 240}, u'name': u'tripleo.package_update.v1.package_update_plan', u'params': {u'env': {}, u'namespace': u''}, u'spec': {u'description': u'Take a container and perform a package update with possible breakpoints', u'input': [u'container', {u'timeout': 240}, {u'queue_name': u'tripleo'}, {u'skip_deploy_identifier': False}, {u'config_dir': u'/tmp/'}], u'name': u'package_update_plan', u'tags': [u'tripleo-common-managed'], u'tasks': {u'send_message': {u'input': {u'execution': u'<% execution() %>', u'message': u"<% $.get('message', '') %>", u'queue_name': u'<% $.queue_name %>', u'status': u"<% $.get('status', 'SUCCESS') %>", u'type': u'<% execution().name %>'}, u'name': u'send_message', u'type': u'direct', u'version': u'2.0', u'workflow': u'tripleo.messaging.v1.send'}, u'set_update_failed': {u'name': u'set_update_failed', u'on-success': u'send_message', u'publish': {u'message': u'<% task(update).result %>', u'status': u'FAILED'}, u'type': u'direct', u'version': u'2.0'}, u'update': {u'action': u'tripleo.package_update.update_stack', u'input': {u'container': u'<% $.container %>', u'timeout': u'<% $.timeout %>'}, u'name': u'update', u'on-error': u'set_update_failed', u'on-success': u'send_message', u'type': u'direct', u'version': u'2.0'}}, u'version': u'2.0'}, u'updated_at': u'2018-10-10 12:08:00'}, u'execution_id': u'02dfd243-c1dd-41f2-ac55-8453c3377394', u'message': u"The action raised an exception [action_ex_id=cb329a0d-1fde-46d8-a0ab-80a21454090f, action_cls='<class 'mistral.actions.action_factory.UpdateStackAction'>', attributes='{}', params='{u'container': u'overcloud', u'timeout': 240}']\n ERROR: Internal Error", u'plan_name': None, u'status': u'FAILED'}: AssertionError: {u'deployment_status': None,ESC[00m 2018-10-10 15:09:31.677 659310 INFO osc_lib.shell [-] END return value: 1ESC[00m Version-Release number of selected component (if applicable): ------------------------------------------------------------- python2-keystonemiddleware-5.2.0-0.20180816073303.c46f292.el7ost.noarch puppet-keystone-13.3.1-0.20180831224239.75c7b86.el7ost.noarch python2-keystoneauth1-3.10.0-0.20180809120741.323f4e4.el7ost.noarch python2-keystoneclient-3.17.0-0.20180809173259.234ea50.el7ost.noarch openstack-keystone-14.0.1-0.20180920094320.c5930ab.el7ost.noarch python-keystone-14.0.1-0.20180920094320.c5930ab.el7ost.noarch python-mistral-7.0.3-0.20180919133909.4f3792f.el7ost.noarch openstack-mistral-api-7.0.3-0.20180919133909.4f3792f.el7ost.noarch puppet-mistral-13.3.1-0.20180831192741.bb0e35e.el7ost.noarch python2-mistral-lib-1.0.0-0.20180821152751.d1ccfd0.el7ost.noarch openstack-mistral-executor-7.0.3-0.20180919133909.4f3792f.el7ost.noarch python2-mistralclient-3.7.0-0.20180810140142.f0ee48f.el7ost.noarch openstack-mistral-engine-7.0.3-0.20180919133909.4f3792f.el7ost.noarch openstack-mistral-common-7.0.3-0.20180919133909.4f3792f.el7ost.noarch openstack-tripleo-heat-templates-9.0.0-0.20180919080945.0rc1.0rc1.el7ost.noarch Steps to Reproduce: ------------------- 1. Upgrade undercloud to RHOS-14(2018-10-08.4) 2. Setup repos/containers for overcloud upgrade 3. Run upgrade prepare commands Actual results: --------------- Upgrade prepare command fails, blocking further upgrade Expected results: ----------------- Upgrade prepare succeeds
The issue is with the _member_ role. It's present in the 13 install somehow (maybe added by provisioning?), so the trust created by Heat references it. When we move to containerized undercloud, we don't run instack anymore and don't have the "_member_role_exists" function called. puppet-keystone just knows about the admin role, it applies the roles and remove _member_, and the Heat trust is broken. Workaround: add the role _member_ to admin again. Solutions: * Do the instack bits in tripleoclient in undercloud upgrade * Tell puppet-keystone about the existing roles somehow * Tell puppet-keystone to not remove the roles it doesn't know about.
Hi, So, that commit[1] was supposed to take care of this and is run when --use-heat=false on the cli[2] which is the case. So something isn't working as expected here. Yurii, could you provides the undercloud sos report after the undercloud upgrade, or just provide an undercloud upgrade to osp14 env? I'll deploy one on my side as well. [1] https://github.com/openstack/instack-undercloud/commit/9f6465fe8c6732da7bb4f401b230a17f021f47a8 [2] https://github.com/openstack/python-tripleoclient/blob/stable/queens/tripleoclient/v1/undercloud.py#L81..L97
Actually I don't know which command you used to upgrade the undercloud, could you provide it ?
Adding a review, currently based on code reading only, very early WIP.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHEA-2019:0045