Bug 1626140

Description of problem:

Custom roles deployment with controller pcmk + controller systemd roles deployment fails caused by Keystone project/users not being created

Version-Release number of selected component (if applicable):
openstack-tripleo-heat-templates-8.0.4-20.el7ost.noarch

How reproducible:
100%

Steps to Reproduce:
openstack overcloud deploy \
--timeout 100 \
--templates /usr/share/openstack-tripleo-heat-templates \
--stack overcloud \
--libvirt-type kvm \
--ntp-server 10.35.255.6 \
-r /home/stack/roles_data.yaml \
-e /home/stack/virt/config_lvm.yaml \
-e /usr/share/openstack-tripleo-heat-templates/environments/network-isolation.yaml \
-e /home/stack/virt/network/network-environment.yaml \
-e /home/stack/virt/enable-tls.yaml \
-e /home/stack/virt/inject-trust-anchor.yaml \
-e /home/stack/virt/public_vip.yaml \
-e /usr/share/openstack-tripleo-heat-templates/environments/ssl/tls-endpoints-public-ip.yaml \
-e /home/stack/virt/nodes_data.yaml \
-e /home/stack/virt/docker-images.yaml \

Actual results:
Deployment fails:

            "Error running ['docker', 'run', '--name', 'ceilometer_gnocchi_upgrade', '--label', 'config_id=tripleo_step5', '--label', 'container_name=ceilometer_gnocchi_upgrade', '--label', 'managed_by=paunch', '--label', 'config_data={\"start_order\": 99, \"healthcheck\": {\"test\": \"/openstack/healthcheck\"}, \"image\": \"192.168.24.1:8787/rhosp13/openstack-ceilometer-central:2018-08-30.1\", \"command\": [\"/usr/bin/bootstrap_host_exec\", \"ceilometer_agent_central\", \"su ceilometer -s /bin/bash -c \\'for n in {1..10}; do /usr/bin/ceilometer-upgrade --skip-metering-database && exit 0 || sleep 30; done; exit 1\\'\"], \"user\": \"root\", \"volumes\": [\"/etc/hosts:/etc/hosts:ro\", \"/etc/localtime:/etc/localtime:ro\", \"/etc/pki/ca-trust/extracted:/etc/pki/ca-trust/extracted:ro\", \"/etc/pki/ca-trust/source/anchors:/etc/pki/ca-trust/source/anchors:ro\", \"/etc/pki/tls/certs/ca-bundle.crt:/etc/pki/tls/certs/ca-bundle.crt:ro\", \"/etc/pki/tls/certs/ca-bundle.trust.crt:/etc/pki/tls/certs/ca-bundle.trust.crt:ro\", \"/etc/pki/tls/cert.pem:/etc/pki/tls/cert.pem:ro\", \"/dev/log:/dev/log\", \"/etc/ssh/ssh_known_hosts:/etc/ssh/ssh_known_hosts:ro\", \"/etc/puppet:/etc/puppet:ro\", \"/var/lib/config-data/ceilometer/etc/ceilometer/:/etc/ceilometer/:ro\", \"/var/log/containers/ceilometer:/var/log/ceilometer\"], \"net\": \"host\", \"detach\": false, \"privileged\": false}', '--net=host', '--health-cmd=/openstack/healthcheck', '--privileged=false', '--user=root', '--volume=/etc/hosts:/etc/hosts:ro', '--volume=/etc/localtime:/etc/localtime:ro', '--volume=/etc/pki/ca-trust/extracted:/etc/pki/ca-trust/extracted:ro', '--volume=/etc/pki/ca-trust/source/anchors:/etc/pki/ca-trust/source/anchors:ro', '--volume=/etc/pki/tls/certs/ca-bundle.crt:/etc/pki/tls/certs/ca-bundle.crt:ro', '--volume=/etc/pki/tls/certs/ca-bundle.trust.crt:/etc/pki/tls/certs/ca-bundle.trust.crt:ro', '--volume=/etc/pki/tls/cert.pem:/etc/pki/tls/cert.pem:ro', '--volume=/dev/log:/dev/log', '--volume=/etc/ssh/ssh_known_hosts:/etc/ssh/ssh_known_hosts:ro', '--volume=/etc/puppet:/etc/puppet:ro', '--volume=/var/lib/config-data/ceilometer/etc/ceilometer/:/etc/ceilometer/:ro', '--volume=/var/log/containers/ceilometer:/var/log/ceilometer', '192.168.24.1:8787/rhosp13/openstack-ceilometer-central:2018-08-30.1', '/usr/bin/bootstrap_host_exec', 'ceilometer_agent_central', \"su ceilometer -s /bin/bash -c 'for n in {1..10}; do /usr/bin/ceilometer-upgrade --skip-metering-database && exit 0 || sleep 30; done; exit 1'\"]. [1]", 
            "stdout: "
        ]
/var/log/containers/ceilometer/ceilometer-upgrade.log shows:

2018-09-06 03:42:58.316 210 ERROR ceilometer Unauthorized: The request you have made requires authentication. (HTTP 401) (Request-ID: req-6dcf6459-0e7f-4788-a7f6-5f03a12df827)


Checking the Keystone db:
()[root@overcloud-controller-0 /]# mysql -e 'select * from keystone.project'
+----------------------------------+--------------------------+-------+-----------------------------------------------+---------+--------------------------+-----------+-----------+
| id                               | name                     | extra | description                                   | enabled | domain_id                | parent_id | is_domain |
+----------------------------------+--------------------------+-------+-----------------------------------------------+---------+--------------------------+-----------+-----------+
| 09322b6a09844fe580f44c921626c7c0 | admin                    | {}    | Bootstrap project for initializing the cloud. |       1 | default                  | default   |         0 |
| <<keystone.domain.root>>         | <<keystone.domain.root>> | {}    |                                               |       0 | <<keystone.domain.root>> | NULL      |         1 |
| default                          | Default                  | {}    | The default domain                            |       1 | <<keystone.domain.root>> | NULL      |         1 |
+----------------------------------+--------------------------+-------+-----------------------------------------------+---------+--------------------------+-----------+-----------+
()[root@overcloud-controller-0 /]# mysql -e 'select * from keystone.user'
+----------------------------------+-------+---------+--------------------+---------------------+----------------+-----------+
| id                               | extra | enabled | default_project_id | created_at          | last_active_at | domain_id |
+----------------------------------+-------+---------+--------------------+---------------------+----------------+-----------+
| 4da527c70d9d4edbbbae983bb0f8e9a8 | {}    |       1 | NULL               | 2018-09-06 03:25:23 | NULL           | default   |
+----------------------------------+-------+---------+--------------------+---------------------+----------------+-----------+


Expected results:
Deployment succeds.

Additional info:

Attaching the environment files and sosreport.