Bug 1610498

Description of problem:

Example:

openstack overcloud deploy --stack Foo --templates \
[...]

Causes this failure:

2018-07-31 17:54:43Z [Foo.AllNodesDeploySteps.ControllerDeployment_Step3.0]: SIGNAL_IN_PROGRESS  Signal: deployment af793c12-e5f8-4e17-896c-dc58e2d809ab failed (2)
2018-07-31 17:54:44Z [Foo.AllNodesDeploySteps.ControllerDeployment_Step3.0]: CREATE_FAILED  Error: resources[0]: Deployment to server failed: deploy_status_code : Deployment exited with non-zero status code: 2
2018-07-31 17:54:44Z [Foo.AllNodesDeploySteps.ControllerDeployment_Step3]: CREATE_FAILED  Resource CREATE failed: Error: resources[0]: Deployment to server failed: deploy_status_code : Deployment exited with non-zero status code: 2
2018-07-31 17:54:45Z [Foo.AllNodesDeploySteps.ControllerDeployment_Step3]: CREATE_FAILED  Error: resources.ControllerDeployment_Step3.resources[0]: Deployment to server failed: deploy_status_code: Deployment exited with non-zero status code: 2
2018-07-31 17:54:45Z [Foo.AllNodesDeploySteps]: CREATE_FAILED  Resource CREATE failed: Error: resources.ControllerDeployment_Step3.resources[0]: Deployment to server failed: deploy_status_code: Deployment exited with non-zero status code: 2
2018-07-31 17:54:46Z [Foo.AllNodesDeploySteps]: CREATE_FAILED  Error: resources.AllNodesDeploySteps.resources.ControllerDeployment_Step3.resources[0]: Deployment to server failed: deploy_status_code: Deployment exited with non-zero status code: 2
2018-07-31 17:54:46Z [Foo]: CREATE_FAILED  Resource CREATE failed: Error: resources.AllNodesDeploySteps.resources.ControllerDeployment_Step3.resources[0]: Deployment to server failed: deploy_status_code: Deployment exited with non-zero status code: 2

 Stack Foo CREATE_FAILED 

Foo.AllNodesDeploySteps.ControllerDeployment_Step3.0:
  resource_type: OS::Heat::StructuredDeployment
  physical_resource_id: af793c12-e5f8-4e17-896c-dc58e2d809ab
  status: CREATE_FAILED
  status_reason: |
    Error: resources[0]: Deployment to server failed: deploy_status_code : Deployment exited with non-zero status code: 2
  deploy_stdout: |
    ...
            "+ exit 1", 
            "2018-07-31 12:54:41,050 INFO: 84167 -- Finished processing puppet configs for keystone_init_tasks", 
            "2018-07-31 12:54:41,051 ERROR: 84165 -- ERROR configuring keystone_init_tasks"
        ]
    }
        to retry, use: --limit @/var/lib/heat-config/heat-config-ansible/dbe7570c-45a3-4332-901d-1c9fe198acbc_playbook.retry
    
    PLAY RECAP *********************************************************************
    localhost                  : ok=9    changed=2    unreachable=0    failed=1   
    
    (truncated, view all with --long)
  deploy_stderr: |

Heat Stack create failed.
Heat Stack create failed.





Version-Release number of selected component (if applicable):
Current OSP 13



How reproducible:
100%


Steps to Reproduce:
1. openstack overcloud deploy --stack Foo --templates \
[...]

Actual results:
Failure


Expected results:
Fail nicer or success

Additional info:

keystone db_sync never runs and is silently skipped.

Seems to be a mismatch here:

[heat-admin@foo-controller-0 ~]$ facter hostname
foo-controller-0
[heat-admin@foo-controller-0 ~]$ sudo hiera -c /etc/puppet/hiera.yaml bootstrap_nodeid
Foo-controller-0

But the code here should take care of that:

From  /etc/puppet/modules/tripleo/manifests/profile/base/keystone.pp

  if $::hostname == downcase($bootstrap_node) {
    $sync_db = true
    $manage_roles = true
    $manage_endpoint = true
    $manage_domain = true
  } else {

heat-admin@foo-controller-0 ~]$ sudo docker exec -ti galera-bundle-docker-0 mysql -D keystone -e 'show tables' 
(nil)

From /var/log/containers/keystone/keystone.log

2018-07-31 12:51:47.878 20 INFO keystone.common.wsgi [req-a03a23b8-7e6c-4f2c-b066-e72c675da806 - - - - -] GET http://foo-controller-0.external.flusche.co:35357/v3/domains
2018-07-31 12:51:47.879 20 WARNING keystone.common.controller [req-a03a23b8-7e6c-4f2c-b066-e72c675da806 - - - - -] RBAC: Bypassing authorization
2018-07-31 12:51:47.965 20 ERROR keystone.common.wsgi [req-a03a23b8-7e6c-4f2c-b066-e72c675da806 - - - - -] (pymysql.err.ProgrammingError) (1146, u"Table 'keystone.project' doesn't exist") [SQL: u'SELECT project.id AS project_id, project.name AS project_name, project.domain_id AS project_domain_id, project.description AS project_description, project.enabled AS project_enabled, project.extra AS project_extra, project.parent_id AS project_parent_id, project.is_domain AS project_is_domain \nFROM project \nWHERE project.is_domain = true'] (Background on this error at: http://sqlalche.me/e/f405): ProgrammingError: (pymysql.err.ProgrammingError) (1146, u"Table 'keystone.project' doesn't exist") [SQL: u'SELECT project.id AS project_id, project.name AS project_name, project.domain_id AS project_domain_id, project.description AS project_description, project.enabled AS project_enabled, project.extra AS project_extra, project.parent_id AS project_parent_id, project.is_domain AS project_is_domain \nFROM project \nWHERE project.is_domain = true'] (Background on this error at: http://sqlalche.me/e/f405)
2018-07-31 12:51:47.965 20 ERROR keystone.common.wsgi Traceback (most recent call last):
2018-07-31 12:51:47.965 20 ERROR keystone.common.wsgi   File "/usr/lib/python2.7/site-packages/keystone/common/wsgi.py", line 226, in __call__
2018-07-31 12:51:47.965 20 ERROR keystone.common.wsgi     result = method(req, **params)
2018-07-31 12:51:47.965 20 ERROR keystone.common.wsgi   File "/usr/lib/python2.7/site-packages/keystone/common/controller.py", line 126, in wrapper
2018-07-31 12:51:47.965 20 ERROR keystone.common.wsgi     return f(self, request, filters, **kwargs)
2018-07-31 12:51:47.965 20 ERROR keystone.common.wsgi   File "/usr/lib/python2.7/site-packages/keystone/resource/controllers.py", line 54, in list_domains
2018-07-31 12:51:47.965 20 ERROR keystone.common.wsgi     refs = PROVIDERS.resource_api.list_domains(hints=hints)
2018-07-31 12:51:47.965 20 ERROR keystone.common.wsgi   File "/usr/lib/python2.7/site-packages/keystone/common/manager.py", line 116, in wrapped
2018-07-31 12:51:47.965 20 ERROR keystone.common.wsgi     __ret_val = __f(*args, **kwargs)
2018-07-31 12:51:47.965 20 ERROR keystone.common.wsgi   File "/usr/lib/python2.7/site-packages/keystone/common/manager.py", line 68, in wrapper
2018-07-31 12:51:47.965 20 ERROR keystone.common.wsgi     return f(self, *args, **kwargs)
2018-07-31 12:51:47.965 20 ERROR keystone.common.wsgi   File "/usr/lib/python2.7/site-packages/keystone/resource/core.py", line 735, in list_domains
2018-07-31 12:51:47.965 20 ERROR keystone.common.wsgi     projects = self.list_projects_acting_as_domain(hints)
2018-07-31 12:51:47.965 20 ERROR keystone.common.wsgi   File "/usr/lib/python2.7/site-packages/keystone/common/manager.py", line 116, in wrapped
2018-07-31 12:51:47.965 20 ERROR keystone.common.wsgi     __ret_val = __f(*args, **kwargs)
2018-07-31 12:51:47.965 20 ERROR keystone.common.wsgi   File "/usr/lib/python2.7/site-packages/keystone/resource/core.py", line 854, in list_projects_acting_as_domain
2018-07-31 12:51:47.965 20 ERROR keystone.common.wsgi     hints or driver_hints.Hints())
2018-07-31 12:51:47.965 20 ERROR keystone.common.wsgi   File "/usr/lib/python2.7/site-packages/keystone/resource/backends/sql.py", line 121, in list_projects_acting_as_domain
2018-07-31 12:51:47.965 20 ERROR keystone.common.wsgi     return self.list_projects(hints)
2018-07-31 12:51:47.965 20 ERROR keystone.common.wsgi   File "/usr/lib/python2.7/site-packages/keystone/common/driver_hints.py", line 42, in wrapper
2018-07-31 12:51:47.965 20 ERROR keystone.common.wsgi     return f(self, hints, *args, **kwargs)
2018-07-31 12:51:47.965 20 ERROR keystone.common.wsgi   File "/usr/lib/python2.7/site-packages/keystone/resource/backends/sql.py", line 85, in list_projects
2018-07-31 12:51:47.965 20 ERROR keystone.common.wsgi     return [project_ref.to_dict() for project_ref in project_refs
2018-07-31 12:51:47.965 20 ERROR keystone.common.wsgi   File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2878, in __iter__
2018-07-31 12:51:47.965 20 ERROR keystone.common.wsgi     return self._execute_and_instances(context)
2018-07-31 12:51:47.965 20 ERROR keystone.common.wsgi   File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2901, in _execute_and_instances
2018-07-31 12:51:47.965 20 ERROR keystone.common.wsgi     result = conn.execute(querycontext.statement, self._params)
2018-07-31 12:51:47.965 20 ERROR keystone.common.wsgi   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 948, in execute
2018-07-31 12:51:47.965 20 ERROR keystone.common.wsgi     return meth(self, multiparams, params)
2018-07-31 12:51:47.965 20 ERROR keystone.common.wsgi   File "/usr/lib64/python2.7/site-packages/sqlalchemy/sql/elements.py", line 269, in _execute_on_connection
2018-07-31 12:51:47.965 20 ERROR keystone.common.wsgi     return connection._execute_clauseelement(self, multiparams, params)
2018-07-31 12:51:47.965 20 ERROR keystone.common.wsgi   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1060, in _execute_clauseelement
2018-07-31 12:51:47.965 20 ERROR keystone.common.wsgi     compiled_sql, distilled_params
2018-07-31 12:51:47.965 20 ERROR keystone.common.wsgi   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1200, in _execute_context
2018-07-31 12:51:47.965 20 ERROR keystone.common.wsgi     context)
2018-07-31 12:51:47.965 20 ERROR keystone.common.wsgi   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1409, in _handle_dbapi_exception
2018-07-31 12:51:47.965 20 ERROR keystone.common.wsgi     util.raise_from_cause(newraise, exc_info)
2018-07-31 12:51:47.965 20 ERROR keystone.common.wsgi   File "/usr/lib64/python2.7/site-packages/sqlalchemy/util/compat.py", line 203, in raise_from_cause
2018-07-31 12:51:47.965 20 ERROR keystone.common.wsgi     reraise(type(exception), exception, tb=exc_tb, cause=cause)
2018-07-31 12:51:47.965 20 ERROR keystone.common.wsgi   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/base.py", line 1193, in _execute_context
2018-07-31 12:51:47.965 20 ERROR keystone.common.wsgi     context)
2018-07-31 12:51:47.965 20 ERROR keystone.common.wsgi   File "/usr/lib64/python2.7/site-packages/sqlalchemy/engine/default.py", line 507, in do_execute
2018-07-31 12:51:47.965 20 ERROR keystone.common.wsgi     cursor.execute(statement, parameters)
2018-07-31 12:51:47.965 20 ERROR keystone.common.wsgi   File "/usr/lib/python2.7/site-packages/pymysql/cursors.py", line 166, in execute
2018-07-31 12:51:47.965 20 ERROR keystone.common.wsgi     result = self._query(query)
2018-07-31 12:51:47.965 20 ERROR keystone.common.wsgi   File "/usr/lib/python2.7/site-packages/pymysql/cursors.py", line 322, in _query
2018-07-31 12:51:47.965 20 ERROR keystone.common.wsgi     conn.query(q)
2018-07-31 12:51:47.965 20 ERROR keystone.common.wsgi   File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 856, in query
2018-07-31 12:51:47.965 20 ERROR keystone.common.wsgi     self._affected_rows = self._read_query_result(unbuffered=unbuffered)
2018-07-31 12:51:47.965 20 ERROR keystone.common.wsgi   File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 1057, in _read_query_result
2018-07-31 12:51:47.965 20 ERROR keystone.common.wsgi     result.read()
2018-07-31 12:51:47.965 20 ERROR keystone.common.wsgi   File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 1340, in read
2018-07-31 12:51:47.965 20 ERROR keystone.common.wsgi     first_packet = self.connection._read_packet()
2018-07-31 12:51:47.965 20 ERROR keystone.common.wsgi   File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 1014, in _read_packet
2018-07-31 12:51:47.965 20 ERROR keystone.common.wsgi     packet.check_error()
2018-07-31 12:51:47.965 20 ERROR keystone.common.wsgi   File "/usr/lib/python2.7/site-packages/pymysql/connections.py", line 393, in check_error
2018-07-31 12:51:47.965 20 ERROR keystone.common.wsgi     err.raise_mysql_exception(self._data)
2018-07-31 12:51:47.965 20 ERROR keystone.common.wsgi   File "/usr/lib/python2.7/site-packages/pymysql/err.py", line 107, in raise_mysql_exception
2018-07-31 12:51:47.965 20 ERROR keystone.common.wsgi     raise errorclass(errno, errval)
2018-07-31 12:51:47.965 20 ERROR keystone.common.wsgi ProgrammingError: (pymysql.err.ProgrammingError) (1146, u"Table 'keystone.project' doesn't exist") [SQL: u'SELECT project.id AS project_id, project.name AS project_name, project.domain_id AS project_domain_id, project.description AS project_description, project.enabled AS project_enabled, project.extra AS project_extra, project.parent_id AS project_parent_id, project.is_domain AS project_is_domain \nFROM project \nWHERE project.is_domain = true'] (Background on this error at: http://sqlalche.me/e/f405)
2018-07-31 12:51:47.965 20 ERROR keystone.common.wsgi