Can you post the logs for the overcloud's keystone?

(In reply to Juan Antonio Osorio from comment #2)
> Can you post the logs for the overcloud's keystone?

I can do better - please ping me on IRC, I'll let you into the system to poke around

The issue reproduces with and without SSL

https://rhos-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/view/Director/view/9.0/job/backwards_compatibility_UC_9.0_OC_new_8.0_3control_1compute_1ceph_vxlan_noUCSSL_noOCSSL_nonrpm-images-sanity/3/consoleFull

Deployment command: openstack overcloud deploy --templates /home/stack/tht --control-scale 3 --compute-scale 1   --neutron-network-type vxlan --neutron-tunnel-types vxlan  --ntp-server clock.redhat.com --timeout 90 -e /home/stack/tht/environments/storage-environment.yaml -e /home/stack/tht/environments/network-isolation.yaml -e network-environment.yaml --ceph-storage-scale 1


05:05:08 DEBUG::command: timeout 30 sshpass -p 'stack' ssh -q -o UserKnownHostsFile=/dev/null -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no stack.122.89 'source /home/stack/overcloudrc && keystone token-get'
05:05:08 DEBUG::output:  /usr/lib/python2.7/site-packages/keystoneclient/shell.py:64: DeprecationWarning: The keystone CLI is deprecated in favor of python-openstackclient. For a Python library, continue using python-keystoneclient.
05:05:08   'python-keystoneclient.', DeprecationWarning)
05:05:08 Expecting an auth URL via either --os-auth-url or env[OS_AUTH_URL]
05:05:08 The run of timeout 30 sshpass -p 'stack' ssh -q -o UserKnownHostsFile=/dev/null -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no stack.122.89 'source /home/stack/overcloudrc && keystone token-get' failed.
05:05:08 Checking Nova
05:05:08 DEBUG::command: timeout 30 sshpass -p 'stack' ssh -q -o UserKnownHostsFile=/dev/null -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no stack.122.89 'source /home/stack/overcloudrc && nova service-list'
05:05:08 DEBUG::output:  ERROR (CommandError): You must provide a username or user ID via --os-username, --os-user-id, env[OS_USERNAME] or env[OS_USER_ID]
05:05:08 The run of timeout 30 sshpass -p 'stack' ssh -q -o UserKnownHostsFile=/dev/null -o UserKnownHostsFile=/dev/null -o StrictHostKeyChecking=no stack.122.89 'source /home/stack/overcloudrc && nova service-list' failed.
05:05:08 ERROR (CommandError): You must provide a username or user ID via --os-username, --os-user-id, env[OS_USERNAME] or env[OS_USER_ID]

Update:

So it works if you use the admin token
 OK, so it seems it's an issue with the keystone provisioning
 using the admin token
 it worked

[stack@instack ~]$ cat overcloudrc
export OS_NO_CACHE=True
export OS_CLOUDNAME=overcloud
export OS_AUTH_URL=https://192.168.200.180:13000/v2.0
export NOVA_VERSION=1.1
export COMPUTE_API_VERSION=1.1
export OS_USERNAME=admin
export no_proxy=,192.168.200.180,192.0.2.6
export OS_PASSWORD=9FwBqBgeQ3Ks6U9PHAmmVh9Hs
export PYTHONWARNINGS="ignore:Certificate has no, ignore:A true SSLContext object is not available"
export OS_TENANT_NAME=admin
[stack@instack ~]$ openstack --os-token x7vZNWZxJm7qr9akGzEjrsdZs --os-url http://192.0.2.6:35357/v2.0/ user set admin --password 9FwBqBgeQ3Ks6U9PHAmmVh9Hs
[stack@instack ~]$ openstack user list
+----------------------------------+------------------+
| ID                               | Name             |
+----------------------------------+------------------+
| 003f80406d57476899b8d3260e72f519 | admin            |
| b9f0ffa973e643c08dde68ace45aef69 | aodh             |
| 2f1b47eca83847de8cdeea6bc0c59771 | ceilometer       |
| 2e14bd86d8be4191962b5b1097e7ad3e | glance           |
| dff4a1ffc4884a4ca3d472b32f5a26c4 | heat             |
| 6c8cf56f3fd14f438f8fe4e05b10fab3 | ironic           |
| 0ab50d7b72d248c099fa290223ae26f6 | ironic-inspector |
| 7a634731c80546e5a8ba35fd0657c690 | neutron          |
| 529ace28caa44f3db1d25cb72e2b36ac | nova             |
| 161780cbeda5432388099e77a3042b51 | swift            |
+----------------------------------+------------------+

I was able to reproduce this and tested a fix which seemed to work. The problems were due to incompatibility between OSP 8 t-h-t and OSP 9 tripleoclient.

One issue was that fix for bug 1304587 didn't pull necessary hiera with it, resulting in creation of 'openstack' tenant rather than 'admin' tenant. This confuses the OSP 9 tripleoclient initialization of keystone.

Another issue is that we previously created Keystone endpoint via tripleoclient but in OSP 9 that's no longer the case, so we need backport a patch to OSP 8 templates that does the cration via Puppet, to make it compatible with OSP 9 tripleoclient.

I submitted a downstream backport with the rest of the relevant upstream patch.

Steps to verify:
1. Installed OSP9 undercloud using today's puddle:

[stack@instack ~]$ rpm -qa |grep tripleo
openstack-tripleo-0.0.8-0.2.d81bd6dgit.el7ost.noarch
openstack-tripleo-image-elements-0.9.9-6.el7ost.noarch
python-tripleoclient-2.0.0-1.0.6.el7ost.noarch
openstack-tripleo-puppet-elements-2.0.0-3.el7ost.noarch
openstack-tripleo-common-2.0.0-6.el7ost.noarch
openstack-tripleo-heat-templates-2.0.0-16.el7ost.noarch
openstack-tripleo-heat-templates-liberty-2.0.0-16.el7ost.noarch
openstack-tripleo-heat-templates-kilo-2.0.0-16.el7ost.noarch


2. Downloaded the full patch from https://code.engineering.redhat.com/gerrit/changes/79319/revisions/4f0520ac613058e35589697106a30a52abf83165/archive?format=tar

3. on instack, replaced /usr/share/openstack-tripleo-heat-templates with the extracted contents of the patch tar

4. Deployed a new Overcloud 8 from current puddle:
openstack overcloud deploy --templates /home/stack/tht --control-scale 3 --compute-scale 1 --neutron-network-type vxlan --neutron-tunnel-types vxlan --ntp-server clock.redhat.com --timeout 90 -e /home/stack/tht/environments/storage-environment.yaml -e /home/stack/tht/environments/network-isolation.yaml -e network-environment.yaml --ceph-storage-scale 1

/home/stack/tht copied from /usr/share/openstack-tripleo-heat-templates/liberty/

5. Deployment completed successfully
Testing the deployment:

[stack@instack ~]$ . stackrc 
[stack@instack ~]$ keystone endpoint-list
/usr/lib/python2.7/site-packages/keystoneclient/shell.py:64: DeprecationWarning: The keystone CLI is deprecated in favor of python-openstackclient. For a Python library, continue using python-keystoneclient.
  'python-keystoneclient.', DeprecationWarning)
/usr/lib/python2.7/site-packages/keystoneclient/v2_0/client.py:145: DeprecationWarning: Constructing an instance of the keystoneclient.v2_0.client.Client class without a session is deprecated as of the 1.7.0 release and may be removed in the 2.0.0 release.
  'the 2.0.0 release.', DeprecationWarning)
/usr/lib/python2.7/site-packages/keystoneclient/v2_0/client.py:147: DeprecationWarning: Using the 'tenant_name' argument is deprecated in version '1.7.0' and will be removed in version '2.0.0', please use the 'project_name' argument instead
  super(Client, self).__init__(**kwargs)
/usr/lib/python2.7/site-packages/debtcollector/renames.py:45: DeprecationWarning: Using the 'tenant_id' argument is deprecated in version '1.7.0' and will be removed in version '2.0.0', please use the 'project_id' argument instead
  return f(*args, **kwargs)
/usr/lib/python2.7/site-packages/keystoneclient/httpclient.py:371: DeprecationWarning: Constructing an HTTPClient instance without using a session is deprecated as of the 1.7.0 release and may be removed in the 2.0.0 release.
  'the 2.0.0 release.', DeprecationWarning)
/usr/lib/python2.7/site-packages/keystoneclient/session.py:140: DeprecationWarning: keystoneclient.session.Session is deprecated as of the 2.1.0 release in favor of keystoneauth1.session.Session. It will be removed in future releases.
  DeprecationWarning)
/usr/lib/python2.7/site-packages/keystoneclient/auth/identity/base.py:56: DeprecationWarning: keystoneclient auth plugins are deprecated as of the 2.1.0 release in favor of keystoneauth1 plugins. They will be removed in future releases.
  'in future releases.', DeprecationWarning)
+----------------------------------+-----------+---------------------------------------------+---------------------------------------------+----------------------------------------+----------------------------------+
|                id                |   region  |                  publicurl                  |                 internalurl                 |                adminurl                |            service_id            |
+----------------------------------+-----------+---------------------------------------------+---------------------------------------------+----------------------------------------+----------------------------------+
| 13c3a725f8a3456f97b04be059efe202 | regionOne |            http://192.0.2.1:8042            |            http://192.0.2.1:8042            |         http://192.0.2.1:8042          | ee022cbdbe844320943f63ce016f8584 |
| 43ce37bbaa834732ac670d165f062696 | regionOne |            http://192.0.2.1:9292            |            http://192.0.2.1:9292            |         http://192.0.2.1:9292          | be92cd78bf5a479fb7906a2a4c896118 |
| 6319e33c68294543bfce1ee3668642f0 | regionOne |            http://192.0.2.1:9696            |            http://192.0.2.1:9696            |         http://192.0.2.1:9696          | 76b27990b6384e1b912c7f374391527e |
| 8986d729629f42b9ae9a979fea88fdc0 | regionOne |    http://192.0.2.1:8774/v2/%(tenant_id)s   |    http://192.0.2.1:8774/v2/%(tenant_id)s   | http://192.0.2.1:8774/v2/%(tenant_id)s | dad5588feba04f1baeaabb6668f6d292 |
| 9693ab4bcec54e2b814f4bac6cdd3052 | regionOne |            http://192.0.2.1:6385            |            http://192.0.2.1:6385            |         http://192.0.2.1:6385          | 96433a52259049d28c0e21f9a98dc556 |
| a04afb8107694e31afb48b90ad3b05f6 | regionOne |            http://192.0.2.1:8777            |            http://192.0.2.1:8777            |         http://192.0.2.1:8777          | 6614559a0e6747e5b23e8fab45e8b42a |
| a800937456fc42069f56ce8ca4ab5a74 | regionOne |           http://192.0.2.1:8774/v3          |           http://192.0.2.1:8774/v3          |        http://192.0.2.1:8774/v3        | 2d2efd5178e74b7db93cdfea8785ce69 |
| abd10356a5ca47e79cc77f279f2c1991 | regionOne |          http://192.0.2.1:5000/v2.0         |          http://192.0.2.1:5000/v2.0         |      http://192.0.2.1:35357/v2.0       | 27b9a5f04d0949028e8c4f1c70eb4f08 |
| af5c50ad84c14c0b805a2ac66e7e4e17 | regionOne |            http://192.0.2.1:5050            |            http://192.0.2.1:5050            |         http://192.0.2.1:5050          | 9a799712d81744ed9542ff909034a82c |
| dbe26606424846e4854babedb9a605f9 | regionOne | http://192.0.2.1:8080/v1/AUTH_%(tenant_id)s | http://192.0.2.1:8080/v1/AUTH_%(tenant_id)s |         http://192.0.2.1:8080          | d89114d31422468bb7f65b8f6c7cd901 |
| f43f480887e241cbb7f836d4913fa95a | regionOne |    http://192.0.2.1:8004/v1/%(tenant_id)s   |    http://192.0.2.1:8004/v1/%(tenant_id)s   | http://192.0.2.1:8004/v1/%(tenant_id)s | 0ba42f5f6aae4d17a1a8003ba5301a94 |
+----------------------------------+-----------+---------------------------------------------+---------------------------------------------+----------------------------------------+----------------------------------+



[stack@instack ~]$ . overcloudrc 
[stack@instack ~]$ keystone endpoint-list
/usr/lib/python2.7/site-packages/keystoneclient/shell.py:64: DeprecationWarning: The keystone CLI is deprecated in favor of python-openstackclient. For a Python library, continue using python-keystoneclient.
  'python-keystoneclient.', DeprecationWarning)
/usr/lib/python2.7/site-packages/keystoneclient/v2_0/client.py:145: DeprecationWarning: Constructing an instance of the keystoneclient.v2_0.client.Client class without a session is deprecated as of the 1.7.0 release and may be removed in the 2.0.0 release.
  'the 2.0.0 release.', DeprecationWarning)
/usr/lib/python2.7/site-packages/keystoneclient/v2_0/client.py:147: DeprecationWarning: Using the 'tenant_name' argument is deprecated in version '1.7.0' and will be removed in version '2.0.0', please use the 'project_name' argument instead
  super(Client, self).__init__(**kwargs)
/usr/lib/python2.7/site-packages/debtcollector/renames.py:45: DeprecationWarning: Using the 'tenant_id' argument is deprecated in version '1.7.0' and will be removed in version '2.0.0', please use the 'project_id' argument instead
  return f(*args, **kwargs)
/usr/lib/python2.7/site-packages/keystoneclient/httpclient.py:371: DeprecationWarning: Constructing an HTTPClient instance without using a session is deprecated as of the 1.7.0 release and may be removed in the 2.0.0 release.
  'the 2.0.0 release.', DeprecationWarning)
/usr/lib/python2.7/site-packages/keystoneclient/session.py:140: DeprecationWarning: keystoneclient.session.Session is deprecated as of the 2.1.0 release in favor of keystoneauth1.session.Session. It will be removed in future releases.
  DeprecationWarning)
/usr/lib/python2.7/site-packages/keystoneclient/auth/identity/base.py:56: DeprecationWarning: keystoneclient auth plugins are deprecated as of the 2.1.0 release in favor of keystoneauth1 plugins. They will be removed in future releases.
  'in future releases.', DeprecationWarning)
The request you have made requires authentication. (HTTP 401) (Request-ID: req-fdfe45e2-acfc-461a-8120-cf34cabe2cb7)


[stack@instack ~]$ cat overcloudrc
export OS_NO_CACHE=True
export OS_CLOUDNAME=overcloud
export OS_AUTH_URL=http://192.168.200.180:5000/v2.0
export NOVA_VERSION=1.1
export COMPUTE_API_VERSION=1.1
export OS_USERNAME=admin
export no_proxy=,192.168.200.180,192.0.2.6
export OS_PASSWORD=CubWMqjcN3WfPNZkD4czGMd2K
export PYTHONWARNINGS="ignore:Certificate has no, ignore:A true SSLContext object is not available"
export OS_TENANT_NAME=admin

(In reply to Dan Yasny from comment #7)
> 3. on instack, replaced /usr/share/openstack-tripleo-heat-templates with the
> extracted contents of the patch tar
> 

<snip>

> 
> /home/stack/tht copied from
> /usr/share/openstack-tripleo-heat-templates/liberty/
> 

I think the reason it didn't work might be that the downloaded tar contains only the liberty templates, not the full RPM contents.

IOW the tar should be extracted into /usr/share/openstack-tripleo-heat-templates/liberty, not into /usr/share/openstack-tripleo-heat-templates.

Or alternatively, since the liberty templates were then copied to /home/stack/tht and deployed from there anyway, the tar could be extracted into /home/stack/tht directly.

(In reply to Jiri Stransky from comment #8)
> 
> I think the reason it didn't work might be that the downloaded tar contains
> only the liberty templates, not the full RPM contents.
> 
> IOW the tar should be extracted into
> /usr/share/openstack-tripleo-heat-templates/liberty, not into
> /usr/share/openstack-tripleo-heat-templates.
> 
> Or alternatively, since the liberty templates were then copied to
> /home/stack/tht and deployed from there anyway, the tar could be extracted
> into /home/stack/tht directly.

Thanks, I'll start over, and get it done properly this time, then report.

Completed a simple deployment, using the patched directory as my v8 THT. 
source overcloudrc && keystone endpoint-list now works as expected

I suggest we keep the BZ open until the fix is merged and the deployment works ok in CI using a new puddle

The deployment passes the standard sanity tests as well.

Scale tests/OC management tests also pass

Verified with/without ssl and using other feature conjunctions

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://rhn.redhat.com/errata/RHEA-2016-1599.html