Bug 1245169

I am getting convinced this is not UI bug. There is something wrong with keystone roles configuration. Infrastructure dashboard has permissions set to "permissions = ('openstack.roles.admin',)". When I remove this, I get over that error and I am able to display for example service configuration page but I am getting Forbidden 403 a on overview page (dashboard/infrastructure) on ironic node list call, which probably requires the role permission too.

2015-07-21 13:23:25,402 6517 ERROR django.request Internal Server Error: /dashboard/infrastructure/
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/django/core/handlers/base.py", line 132, in get_response
    response = wrapped_callback(request, *callback_args, **callback_kwargs)
  File "/usr/lib/python2.7/site-packages/horizon/decorators.py", line 36, in dec
    return view_func(request, *args, **kwargs)
  File "/usr/lib/python2.7/site-packages/horizon/decorators.py", line 52, in dec
    return view_func(request, *args, **kwargs)
  File "/usr/lib/python2.7/site-packages/horizon/decorators.py", line 36, in dec
    return view_func(request, *args, **kwargs)
  File "/usr/lib/python2.7/site-packages/django/views/generic/base.py", line 71, in view
    return self.dispatch(request, *args, **kwargs)
  File "/usr/lib/python2.7/site-packages/django/views/generic/base.py", line 89, in dispatch
    return handler(request, *args, **kwargs)
  File "/usr/lib/python2.7/site-packages/tuskar_ui/infrastructure/overview/views.py", line 147, in get
    return super(IndexView, self).get(request, *args, **kwargs)
  File "/usr/lib/python2.7/site-packages/django/views/generic/edit.py", line 206, in get
    return self.render_to_response(self.get_context_data(form=form))
  File "/usr/lib/python2.7/site-packages/tuskar_boxes/overview/views.py", line 190, in get_context_data
    context = super(IndexView, self).get_context_data(**kwargs)
  File "/usr/lib/python2.7/site-packages/tuskar_ui/infrastructure/overview/views.py", line 154, in get_context_data
    context.update(self.get_data(self.request, context))
  File "/usr/lib/python2.7/site-packages/tuskar_boxes/overview/views.py", line 121, in get_data
    *args, **kwargs)
  File "/usr/lib/python2.7/site-packages/tuskar_ui/infrastructure/overview/views.py", line 222, in get_data
    messages = forms.validate_plan(request, plan)
  File "/usr/lib/python2.7/site-packages/tuskar_ui/infrastructure/overview/forms.py", line 108, in validate_plan
    maintenance=False))
  File "/usr/lib/python2.7/site-packages/horizon/utils/memoized.py", line 90, in wrapped
    value = cache[key] = func(*args, **kwargs)
  File "/usr/lib/python2.7/site-packages/tuskar_ui/handle_errors.py", line 67, in wrapper
    redirect=_error_redirect)
  File "/usr/lib/python2.7/site-packages/horizon/exceptions.py", line 364, in handle
    six.reraise(exc_type, exc_value, exc_traceback)
  File "/usr/lib/python2.7/site-packages/tuskar_ui/handle_errors.py", line 62, in wrapper
    return func(*args, **kwargs)
  File "/usr/lib/python2.7/site-packages/tuskar_ui/api/node.py", line 226, in list
    maintenance=maintenance)
  File "/usr/lib/python2.7/site-packages/ironicclient/v1/node.py", line 88, in list
    return self._list(self._path(path), "nodes")
  File "/usr/lib/python2.7/site-packages/ironicclient/common/base.py", line 121, in _list
    resp, body = self.api.json_request('GET', url)
  File "/usr/lib/python2.7/site-packages/ironicclient/common/http.py", line 353, in json_request
    resp, body_iter = self._http_request(url, method, **kwargs)
  File "/usr/lib/python2.7/site-packages/ironicclient/common/http.py", line 162, in wrapper
    return func(self, url, method, **kwargs)
  File "/usr/lib/python2.7/site-packages/ironicclient/common/http.py", line 336, in _http_request
    error_json.get('debuginfo'), method, url)
Forbidden: Forbidden (HTTP 403)

Reproduced on my freshly installed environment.
Environment:
instack-undercloud-2.1.2-21.el7ost.noarch
python-tuskarclient-0.1.18-3.el7ost.noarch
openstack-tuskar-ui-extras-0.0.4-1.el7ost.noarch
openstack-tuskar-ui-0.3.0-12.el7ost.noarch
openstack-tuskar-0.4.18-3.el7ost.noarch

this appears to be related to undercloud ssl. I was able to reproduce on my undercloud with ssl and I see the following error in /var/log/horizon/horizon.log:

2015-07-21 18:36:55,715 17836 ERROR openstack_auth.user Unable to retrieve project list.
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/openstack_auth/user.py", line 315, in authorized_tenants
    is_federated=self.is_federated)
  File "/usr/lib/python2.7/site-packages/openstack_auth/utils.py", line 145, in wrapper
    result = func(*args, **kwargs)
  File "/usr/lib/python2.7/site-packages/openstack_auth/utils.py", line 247, in get_project_list
    projects = client.tenants.list()
  File "/usr/lib/python2.7/site-packages/keystoneclient/v2_0/tenants.py", line 123, in list
    tenant_list = self._list('/tenants%s' % query, 'tenants')
  File "/usr/lib/python2.7/site-packages/keystoneclient/base.py", line 113, in _list
    resp, body = self.client.get(url, **kwargs)
  File "/usr/lib/python2.7/site-packages/keystoneclient/adapter.py", line 170, in get
    return self.request(url, 'GET', **kwargs)
  File "/usr/lib/python2.7/site-packages/keystoneclient/adapter.py", line 206, in request
    resp = super(LegacyJsonAdapter, self).request(*args, **kwargs)
  File "/usr/lib/python2.7/site-packages/keystoneclient/adapter.py", line 95, in request
    return self.session.request(url, method, **kwargs)
  File "/usr/lib/python2.7/site-packages/keystoneclient/utils.py", line 318, in inner
    return func(*args, **kwargs)
  File "/usr/lib/python2.7/site-packages/keystoneclient/session.py", line 382, in request
    resp = send(**kwargs)
  File "/usr/lib/python2.7/site-packages/keystoneclient/session.py", line 420, in _send_request
    raise exceptions.SSLError(msg)
SSLError: SSL exception connecting to https://192.0.2.2:13000/v2.0/tenants: [Errno 1] _ssl.c:504: error:14090086:SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed

James, are you saying you can't reproduce above *unless* ssl is turned on on the undercloud?

so i reconfigured Horizon to use the internalURL's instead of the publicURL's. that got me past the SSL error.

However, I'm still getting the same error in the UI.

From what I can tell, it has to do with the admin user having the swiftoperator role. We added the swiftoperator role to the admin user so that the admin user can query the swift bugs for the data saved by ironic-discoverd.

What I see happening is that keystone is only returning the first role associated with the user. I've reproduced this in environments with undercloud SSL and non-SSL in place. If the database id of the service project (associated with the swiftoperator role) comes before (a-z sorted) the admin project, then keystone returns that the admin user is only associated with the swiftoperator role. Likewise, if the admin project is sorted first (such as the db id of the admin project starting with 'a', and the db id of the service project starting with 'b'), then I can login to the UI fine.

If I put a breakpoint in the code and inspect the data returned from Keystone for the admin user at line 81, /usr/lib/python2.7/site-packages/keystoneclient/auth/identity/v2.py, if I pp resp_data, I see:

(Pdb) pp resp_data
{u'metadata': {u'is_admin': 0,
               u'roles': [u'6aeb92e4048e457ab4d6b2c9838badba']},
 u'serviceCatalog': [{u'endpoints': [{u'adminURL': u'http://192.0.2.1:8585/v2',
                                      u'id': u'2aaa5edbf2b44fecae2f17d0aea62186',
                                      u'internalURL': u'http://192.0.2.1:8585/v2',
                                      u'publicURL': u'http://192.0.2.2:8585/v2',
                                      u'region': u'regionOne'}],
                      u'endpoints_links': [],
                      u'name': u'tuskar',
                      u'type': u'management'},
                     {u'endpoints': [{u'adminURL': u'http://192.0.2.1:8774/v2/a59ea87d4a334dcca4192d5d79385775',
                                      u'id': u'0f64bbf9510d435ca3aa670fba8f0e93',
                                      u'internalURL': u'http://192.0.2.1:8774/v2/a59ea87d4a334dcca4192d5d79385775',
                                      u'publicURL': u'https://192.0.2.2:13774/v2/a59ea87d4a334dcca4192d5d79385775',
                                      u'region': u'regionOne'}],
                      u'endpoints_links': [],
                      u'name': u'nova',
                      u'type': u'compute'},
                     {u'endpoints': [{u'adminURL': u'http://192.0.2.1:9696/',
                                      u'id': u'3c05907aec104311ad5dd1b83f253046',
                                      u'internalURL': u'http://192.0.2.1:9696/',
                                      u'publicURL': u'https://192.0.2.2:13696/',
                                      u'region': u'regionOne'}],
                      u'endpoints_links': [],
                      u'name': u'neutron',
                      u'type': u'network'},
                     {u'endpoints': [{u'adminURL': u'http://192.0.2.1:8774/v3',
                                      u'id': u'107a2958dc9d4c598535588729b4f515',
                                      u'internalURL': u'http://192.0.2.1:8774/v3',
                                      u'publicURL': u'https://192.0.2.2:13774/v3',
                                      u'region': u'regionOne'}],
                      u'endpoints_links': [],
                      u'name': u'nova',
                      u'type': u'computev3'},
                     {u'endpoints': [{u'adminURL': u'http://192.0.2.1:9292/',
                                      u'id': u'1313a33da5f949e2ab885b755132f8e5',
                                      u'internalURL': u'http://192.0.2.1:9292/',
                                      u'publicURL': u'https://192.0.2.2:13292/',
                                      u'region': u'regionOne'}],
                      u'endpoints_links': [],
                      u'name': u'glance',
                      u'type': u'image'},
                     {u'endpoints': [{u'adminURL': u'http://192.0.2.1:8777/',
                                      u'id': u'178702419f954e628568e0bb6929d409',
                                      u'internalURL': u'http://192.0.2.1:8777/',
                                      u'publicURL': u'http://192.0.2.2:8777/',
                                      u'region': u'regionOne'}],
                      u'endpoints_links': [],
                      u'name': u'ceilometer',
                      u'type': u'metering'},
                     {u'endpoints': [{u'adminURL': u'http://192.0.2.1:6385/',
                                      u'id': u'2e1134ad6ec34d21b7c2e788d7d351f1',
                                      u'internalURL': u'http://192.0.2.1:6385/',
                                      u'publicURL': u'https://192.0.2.2:13385/',
                                      u'region': u'regionOne'}],
                      u'endpoints_links': [],
                      u'name': u'ironic',
                      u'type': u'baremetal'},
                     {u'endpoints': [{u'adminURL': u'http://192.0.2.1:8004/v1/a59ea87d4a334dcca4192d5d79385775',
                                      u'id': u'b45c4f481f304497a327440e763fd4c8',
                                      u'internalURL': u'http://192.0.2.1:8004/v1/a59ea87d4a334dcca4192d5d79385775',
                                      u'publicURL': u'https://192.0.2.2:13004/v1/a59ea87d4a334dcca4192d5d79385775',
                                      u'region': u'regionOne'}],
                      u'endpoints_links': [],
                      u'name': u'heat',
                      u'type': u'orchestration'},
                     {u'endpoints': [{u'adminURL': u'http://192.0.2.1:8080/v1',
                                      u'id': u'084c77cd28384442a1fba6ff8887bf17',
                                      u'internalURL': u'http://192.0.2.1:8080/v1/AUTH_a59ea87d4a334dcca4192d5d79385775',
                                      u'publicURL': u'https://192.0.2.2:13080/v1/AUTH_a59ea87d4a334dcca4192d5d79385775',
                                      u'region': u'regionOne'}],
                      u'endpoints_links': [],
                      u'name': u'swift',
                      u'type': u'object-store'},
                     {u'endpoints': [{u'adminURL': u'http://192.0.2.1:35357/v2.0',
                                      u'id': u'1ed7c6daa17d4326b1efb0c76cc9aed8',
                                      u'internalURL': u'http://192.0.2.1:5000/v2.0',
                                      u'publicURL': u'https://192.0.2.2:13000/v2.0',
                                      u'region': u'regionOne'}],
                      u'endpoints_links': [],
                      u'name': u'keystone',
                      u'type': u'identity'}],
 u'token': {u'audit_ids': [u'F6-d3YfkRYWXRdojiXi0tw',
                           u'wAN2zGydR2SuXpbsZ2zqhg'],
            u'expires': u'2015-07-22T17:17:42Z',
            u'id': u'1a510e3fde0841e0a914727bdce4bf32',
            u'issued_at': u'2015-07-22T13:17:52.951536',
            u'tenant': {u'description': None,
                        u'enabled': True,
                        u'id': u'a59ea87d4a334dcca4192d5d79385775',
                        u'name': u'service'}},
 u'user': {u'id': u'4288b599513f426bbe3b0d7e3c7b8f84',
           u'name': u'admin',
           u'roles': [{u'name': u'swiftoperator'}],
           u'roles_links': [],
           u'username': u'admin'}}


So you can see in roles that only swiftoperator is returned.

ok, i see why it's taking only the first project now.

if you dont specify a project, it takes the first one. you can see in /usr/lib/python2.7/site-packages/openstack_auth/backend.py around line 55, the loop "for project in projects..." will break as soon as you get a scoped auth token for the first project. So, if the service project is sorted first before admin, you get the service project, where the admin user is only associated with the swiftoperator role.

(In reply to James Slagle from comment #9)
> ok, i see why it's taking only the first project now.
> 
> if you dont specify a project, it takes the first one. you can see in
> /usr/lib/python2.7/site-packages/openstack_auth/backend.py around line 55,

that's actually around line 155

i confirmed that if you give the admin user the admin role in the service project via:

openstack role add --user admin --project service admin

you can login fine

I've ran : 
openstack role add --user admin --project service admin

on Virt HA+isoltaed network env and then was able to access the Undercloud UI

(also have reproduced the issue before it...)

just waiting for a CI pass on the gate for this patch. the first run failed due to the regression caused by https://bugzilla.redhat.com/show_bug.cgi?id=1236136

the revert for that has landed and built into a new poodle. I've rekicked the CI job for this one:
https://rhos-jenkins.rhev-ci-vms.eng.rdu2.redhat.com/view/RDO/view/rdo-manager/job/rdo_manager-gate_instack_undercloud-downstream-rhos-7_director/28/

CI passed, merged the commit, did build: instack-undercloud-2.1.2-22.el7ost

*** Bug 1245835 has been marked as a duplicate of this bug. ***

Verified:
Environment:
instack-undercloud-2.1.2-22.el7ost.noarch

Able to login now.

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2015:1549