Bug 1347063

We are hitting this same problem.  If you "source ~/overcloudrc" and then "keystone service-list", you get:

+----------------------------------+------------+---------------+------------------------------+
|                id                |    name    |      type     |         description          |
+----------------------------------+------------+---------------+------------------------------+
| 44302f2369c64c0698b022903b7a1478 | ceilometer |    metering   |      Ceilometer Service      |
| e275e0f6a5a1491283be2ab68a0cc37e |   cinder   |     volume    |    Cinder Volume Service     |
| c16bba4e36f74e84bbd8694d87a4c852 |  cinderv2  |    volumev2   |   Cinder Volume Service v2   |
| 661172dce77f4ecfb9ec50b0ab65ccfa |   glance   |     image     |     Glance Image Service     |
| 46b244f835294b499466034caaac693e |    heat    | orchestration |         Heat Service         |
| f291026064cb42c2858f39e2da82f5dd |  keystone  |    identity   |  Keystone Identity Service   |
| c78c84745d2b49178de9346489581b75 |  neutron   |    network    |       Neutron Service        |
| 5d189f554a364d5c983b82e125f6e21b |    nova    |    compute    |     Nova Compute Service     |
| 5683d301b7b742bf8b59c1a4265c70aa |   swift    |  object-store | Swift Object Storage Service |
+----------------------------------+------------+---------------+------------------------------+

Note that there is no horizon endpoint in the above output.

We are also having this issue. 

However I don't think the issue is with the heat template but rather the trippleoclient. Downgrading this from python-tripleoclient-0.3.4-5.el7ost to python-tripleoclient-0.3.4-4.el7ost.noarch fixes our deployment.

Looking at a diff between the packages I can see a number of changes have been made to the methods that setup the endpoints specifically in:

/usr/lib/python2.7/site-packages/tripleoclient/utils.py
/usr/lib/python2.7/site-packages/tripleoclient/v1/overcloud_deploy.py

I haven't done any detailed debug but I suspect the issues lies somewhere in this code.

Cheers

We are also having the same issue. 

I did downgrading "python-tripleoclient-0.3.4-5.el7ost" to "python-tripleoclient-0.3.4-4.el7ost.noarch", then redeploy, it does NOT fixes the issue ! 

We tested deploy with 1 Controller + 1 Compute, it went through, changed to 3 controller + 1 Compute node, it failed with no "horizon" fully deployed, there is no "network tab" on "horizon". When login to Gui, it give messages "Error: Invalid service catalog service: compute" on TOP Right Window.

.....
2016-06-29 04:55:14 [overcloud]: CREATE_COMPLETE  Stack CREATE completed successfully
Stack overcloud CREATE_COMPLETE
.....
No handlers could be found for logger "oslo_config.cfg"
2016-06-28 23:54:33.703 11795 WARNING keystone.cmd.cli [-] keystone-manage pki_setup is not recommended for production use.
The following cert files already exist, use --rebuild to remove the existing files before regenerating:
/etc/keystone/ssl/certs/ca.pem already exists
/etc/keystone/ssl/private/signing_key.pem already exists
/etc/keystone/ssl/certs/signing_cert.pem already exists
Connection to 192.0.3.6 closed.
The request you have made requires authentication. (HTTP 401) (Request-ID: req-72b49c48-fc5e-45b0-841f-2a225cf8b0c2)
===========================================================================

$ source ~/overcloudrc
$ keystone endpoint-list
/usr/lib/python2.7/site-packages/keystoneclient/shell.py:64: DeprecationWarning: The keystone CLI is deprecated in favor of python-openstackclient. For a Python library, continue using python-keystoneclient.
  'python-keystoneclient.', DeprecationWarning)
/usr/lib/python2.7/site-packages/keystoneclient/v2_0/client.py:145: DeprecationWarning: Constructing an instance of the keystoneclient.v2_0.client.Client class without a session is deprecated as of the 1.7.0 release and may be removed in the 2.0.0 release.
  'the 2.0.0 release.', DeprecationWarning)
/usr/lib/python2.7/site-packages/keystoneclient/v2_0/client.py:147: DeprecationWarning: Using the 'tenant_name' argument is deprecated in version '1.7.0' and will be removed in version '2.0.0', please use the 'project_name' argument instead
  super(Client, self).__init__(**kwargs)
/usr/lib/python2.7/site-packages/debtcollector/renames.py:43: DeprecationWarning: Using the 'tenant_id' argument is deprecated in version '1.7.0' and will be removed in version '2.0.0', please use the 'project_id' argument instead
  return f(*args, **kwargs)
/usr/lib/python2.7/site-packages/keystoneclient/httpclient.py:376: DeprecationWarning: Constructing an HTTPClient instance without using a session is deprecated as of the 1.7.0 release and may be removed in the 2.0.0 release.
  'the 2.0.0 release.', DeprecationWarning)
+----------------------------------+-----------+-------------------------------+------------------------------+-----------------------------+----------------------------------+
|                id                |   region  |           publicurl           |         internalurl          |           adminurl          |            service_id            |
+----------------------------------+-----------+-------------------------------+------------------------------+-----------------------------+----------------------------------+
| d0b1dbb2a28244309b47ce62833892e6 | regionOne | http://173.37.49.70:5000/v2.0 | http://172.16.1.10:5000/v2.0 | http://192.0.3.6:35357/v2.0 | 37da772860754b00b5b1c2eb2f53e504 |
+----------------------------------+-----------+-------------------------------+------------------------------+-----------------------------+----------------------------------+
$ 
$ neutron agent-list
publicURL endpoint for network service not found

In my case with errors, I find out it cause from one node slow in response within cluster, also NTP services has issues with hostname, I had to change NTP to point to IP address of the NTP server, that fixed all issues, the re-deploy went successfully.

Fix we did:

1). downgrading "python-tripleoclient-0.3.4-5.el7ost" to "python-tripleoclient-0.3.4-4.el7ost.noarch";

2). remove the slow response server out of "ironic node-list" pool.

3). change NTP server point from hostname to IP.

Workaround:

yum downgrade python-tripleoclient-0.3.4-4.el7ost.noarch

KCS workaround verified & published:  https://access.redhat.com/solutions/2446961

Hi,

I confirm that the fix is working. Thanks for that.

Regards,

On a fully patched undercloud with HEAT template package versions:

openstack-tripleo-heat-templates.noarch 0:0.8.14-14.el7ost 
openstack-tripleo-heat-templates-kilo.noarch 0:0.8.14-14.el7ost

downgrading just the tripleoclient to:

python-tripleoclient-0.3.4-4.el7ost.noarch

causes a new error:

2016-07-27 06:38:19 [overcloud]: CREATE_FAILED  Resource CREATE failed: resources.Controller: Property error: resources[1].properties: Property MysqlClustercheckPassword not assigned
Stack overcloud CREATE_FAILED
Heat Stack create failed.

You also need to run:

sudo yum downgrade -y openstack-tripleo-heat-templates-kilo-0.8.14-11.el7ost.noarch openstack-tripleo-heat-templates-0.8.14-11.el7ost.noarch

Can you confirm and update the KB appropriately?

I can confirm Charlie's result. Just downgraded tripleo client only in my OSP8 lab. Now seeing the MysqlClustercheckPassword error on overcloud deploy.

2016-07-27 15:09:47 [overcloud-Controller-2ink7cvzrmb2]: CREATE_IN_PROGRESS  Stack CREATE started
2016-07-27 15:09:47 [overcloud-Controller-2ink7cvzrmb2]: CREATE_COMPLETE  Stack CREATE completed successfully
2016-07-27 15:09:47 [0]: CREATE_IN_PROGRESS  state changed
2016-07-27 15:09:48 [Controller]: CREATE_FAILED  resources.Controller: Property error: resources[1].properties: Property MysqlClustercheckPassword not assigned

Confirm that downgrading tripleo-heat-templates got passed the MysqlClustercheckPassword error.

2016-07-27 15:20:51 [Controller]: CREATE_IN_PROGRESS  state changed
2016-07-27 15:20:51 [overcloud-BlockStorage-ytl4omnx6kmv]: UPDATE_COMPLETE  Stack UPDATE completed successfully
2016-07-27 15:20:51 [overcloud-Controller-fw2hcg2pbqnm]: CREATE_IN_PROGRESS  Stack CREATE started
2016-07-27 15:20:52 [overcloud-Controller-fw2hcg2pbqnm]: CREATE_COMPLETE  Stack CREATE completed successfully
2016-07-27 15:20:54 [Compute]: CREATE_IN_PROGRESS  state changed
2016-07-27 15:20:54 [overcloud-Controller-fw2hcg2pbqnm]: UPDATE_IN_PROGRESS  Stack UPDATE started
2016-07-27 15:20:54 [1]: CREATE_IN_PROGRESS  state changed
2016-07-27 15:20:54 [overcloud-Compute-d2wa47xuvvgf]: CREATE_IN_PROGRESS  Stack CREATE started
2016-07-27 15:20:54 [overcloud-Compute-d2wa47xuvvgf]: CREATE_COMPLETE  Stack CREATE completed successfully
2016-07-27 15:20:55 [0]: CREATE_IN_PROGRESS  state changed
2016-07-27 15:20:55 [overcloud-Compute-d2wa47xuvvgf]: UPDATE_IN_PROGRESS  Stack UPDATE started
2016-07-27 15:20:56 [BlockStorage]: CREATE_COMPLETE  state changed
2016-07-27 15:20:56 [overcloud-Controller-fw2hcg2pbqnm-1-xxknwzz46m4h]: CREATE_IN_PROGRESS  Stack CREATE started

Workaround: 
sudo yum downgrade -y openstack-tripleo-heat-templates-kilo-0.8.14-11.el7ost.noarch openstack-tripleo-heat-templates-0.8.14-11.el7ost.noarch

Verified no message about skipping Horizon PostConfig in my latest deployment after downgrading tripleo-client and tripleo-heat-templates.

Stack overcloud CREATE_COMPLETE
/home/stack/.ssh/known_hosts updated.
Original contents retained as /home/stack/.ssh/known_hosts.old
PKI initialization in init-keystone is deprecated and will be removed.
Warning: Permanently added '192.168.104.100' (ECDSA) to the list of known hosts.
No handlers could be found for logger "oslo_config.cfg"
2016-07-27 15:50:25.784 16828 WARNING keystone.cmd.cli [-] keystone-manage pki_setup is not recommended for production use.
The following cert files already exist, use --rebuild to remove the existing files before regenerating:
/etc/keystone/ssl/certs/ca.pem already exists
/etc/keystone/ssl/private/signing_key.pem already exists
/etc/keystone/ssl/certs/signing_cert.pem already exists
Connection to 192.168.104.100 closed.
Overcloud Endpoint: http://10.22.54.101:5000/v2.0
Overcloud Deployed

Since KCS solution intent is 1 issue, 1 solution; I've created another article to highlight this issue, and related it to the first solution. Customers who run into this issue should see this fix, and the previous fix in the linked solution. Thank you for verifying.

KCS workaround verified & published: https://access.redhat.com/solutions/2472361

Hi,

I can confirm Charlie's comment. Downgrading heat templates works.

Thanks,

This also affects OSP9.

python-tripleoclient-2.0.0-3.el7ost.noarch

Ran into the same issue on OSP 8 with python-tripleoclient-0.3.4-6.el7ost.noarch. 

Downgraded to python-tripleoclient-0.3.4-4.el7ost.noarch. 

Issue resolved.