Bug 1361746

Summary:	rhel-osp-director: 7.2->7.3Async->8.0 fails: ERROR oslo_messaging.rpc.dispatcher Forbidden: Trustee has no delegated roles.
Product:	Red Hat OpenStack	Reporter:	Alexander Chuzhoy <sasha>
Component:	rhosp-director	Assignee:	Angus Thomas <athomas>
Status:	CLOSED DUPLICATE	QA Contact:	Omri Hochman <ohochman>
Severity:	high	Docs Contact:
Priority:	high
Version:	8.0 (Liberty)	CC:	aschultz, dbecker, mandreou, mburns, morazi, rhel-osp-director-maint, sathlang
Target Milestone:	async
Target Release:	8.0 (Liberty)
Hardware:	x86_64
OS:	Linux
Whiteboard:
Fixed In Version:		Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2017-02-09 17:14:38 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description Alexander Chuzhoy 2016-07-30 00:38:01 UTC

rhel-osp-director:  7.2->7.3Async->8.0 fails: ERROR oslo_messaging.rpc.dispatcher Forbidden: Trustee has no delegated roles.


Environment:
openstack-tripleo-heat-templates-kilo-0.8.14-16.el7ost.noarch
openstack-tripleo-heat-templates-0.8.14-16.el7ost.noarch
instack-undercloud-2.2.7-7.el7ost.noarch
openstack-puppet-modules-7.1.3-1.el7ost.noarch


Steps to reproduce:
1. Deploy 7.2 with:
openstack overcloud deploy --templates --control-scale 3 --compute-scale 1 --neutron-network-type vxlan --neutron-tunnel-types vxlan --ntp-server clock.redhat.com --timeout 90 -e /usr/share/openstack-tripleo-heat-templates/environments/storage-environment.yaml -e /usr/share/openstack-tripleo-heat-templates/environments/network-isolation.yaml -e network-environment.yaml --ceph-storage-scale 1

2. Populate the setup
3. Update to 7.3Async
4. Attempt to upgrade to 8.0

Result:
2016-07-29 20:16:57 [NodeTLSData]: UPDATE_IN_PROGRESS  state changed
2016-07-29 20:17:02 [NodeTLSData]: UPDATE_COMPLETE  state changed
2016-07-29 20:17:02 [ControllerConfig]: UPDATE_IN_PROGRESS  state changed
2016-07-29 20:17:03 [NodeTLSData]: UPDATE_COMPLETE  state changed
2016-07-29 20:17:04 [ControllerConfig]: CREATE_IN_PROGRESS  state changed
2016-07-29 20:17:04 [ControllerConfig]: UPDATE_IN_PROGRESS  state changed
2016-07-29 20:17:05 [ControllerConfig]: CREATE_IN_PROGRESS  state changed
2016-07-29 20:17:06 [ControllerConfig]: CREATE_COMPLETE  state changed
2016-07-29 20:17:07 [ControllerConfig]: CREATE_COMPLETE  state changed
ERROR: Authentication failed: Authentication required


I see repeating error in heat engine log:
2016-07-29 20:36:41.110 10725 ERROR oslo_messaging.rpc.dispatcher [req-56355828-8ffb-4b4f-bc37-cf4b452b5f86 - 1453013919f24b8c9593e1497deecc88] Exception during message handling: Trustee has no delegated roles. (D
isable debug mode to suppress these details.) (HTTP 403) (Request-ID: req-826f6bd9-2fa5-4254-817e-ed43dff6e474) (HTTP 403)
2016-07-29 20:36:41.110 10725 ERROR oslo_messaging.rpc.dispatcher Traceback (most recent call last):
2016-07-29 20:36:41.110 10725 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 142, in _dispatch_and_reply
2016-07-29 20:36:41.110 10725 ERROR oslo_messaging.rpc.dispatcher     executor_callback))
2016-07-29 20:36:41.110 10725 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 186, in _dispatch
2016-07-29 20:36:41.110 10725 ERROR oslo_messaging.rpc.dispatcher     executor_callback)
2016-07-29 20:36:41.110 10725 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/oslo_messaging/rpc/dispatcher.py", line 129, in _do_dispatch
2016-07-29 20:36:41.110 10725 ERROR oslo_messaging.rpc.dispatcher     result = func(ctxt, **new_args)
2016-07-29 20:36:41.110 10725 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/osprofiler/profiler.py", line 105, in wrapper
2016-07-29 20:36:41.110 10725 ERROR oslo_messaging.rpc.dispatcher     return f(*args, **kwargs)
2016-07-29 20:36:41.110 10725 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/heat/common/context.py", line 308, in wrapped
2016-07-29 20:36:41.110 10725 ERROR oslo_messaging.rpc.dispatcher     return func(self, ctx, *args, **kwargs)
2016-07-29 20:36:41.110 10725 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/heat/engine/service.py", line 1404, in resource_signal
2016-07-29 20:36:41.110 10725 ERROR oslo_messaging.rpc.dispatcher     stack = parser.Stack.load(cnxt, stack=s, use_stored_context=True)
2016-07-29 20:36:41.110 10725 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/heat/engine/stack.py", line 408, in load
2016-07-29 20:36:41.110 10725 ERROR oslo_messaging.rpc.dispatcher     cache_data=cache_data)
2016-07-29 20:36:41.110 10725 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/heat/engine/stack.py", line 459, in _from_db
2016-07-29 20:36:41.110 10725 ERROR oslo_messaging.rpc.dispatcher     current_deps=stack.current_deps, cache_data=cache_data)
2016-07-29 20:36:41.110 10725 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/heat/engine/stack.py", line 192, in __init__
2016-07-29 20:36:41.110 10725 ERROR oslo_messaging.rpc.dispatcher     'keystone').auth_ref.role_names
2016-07-29 20:36:41.110 10725 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/heat/engine/clients/__init__.py", line 70, in client
2016-07-29 20:36:41.110 10725 ERROR oslo_messaging.rpc.dispatcher     return client_plugin.client()
2016-07-29 20:36:41.110 10725 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/heat/engine/clients/client_plugin.py", line 78, in client
2016-07-29 20:36:41.110 10725 ERROR oslo_messaging.rpc.dispatcher     self._client = self._create()
2016-07-29 20:36:41.110 10725 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/heat/engine/clients/os/keystone.py", line 29, in _create
2016-07-29 20:36:41.110 10725 ERROR oslo_messaging.rpc.dispatcher     return hkc.KeystoneClient(self.context)
2016-07-29 20:36:41.110 10725 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/heat/common/heat_keystoneclient.py", line 573, in __new__
2016-07-29 20:36:41.110 10725 ERROR oslo_messaging.rpc.dispatcher     return KeystoneClientV3(context)
2016-07-29 20:36:41.110 10725 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/heat/common/heat_keystoneclient.py", line 84, in __init__
2016-07-29 20:36:41.110 10725 ERROR oslo_messaging.rpc.dispatcher     self._client = self._v3_client_init()
2016-07-29 20:36:41.110 10725 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/heat/common/heat_keystoneclient.py", line 155, in _v3_client_init
2016-07-29 20:36:41.110 10725 ERROR oslo_messaging.rpc.dispatcher     auth_ref = self.context.auth_plugin.get_access(self.session)
2016-07-29 20:36:41.110 10725 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/keystoneclient/auth/identity/base.py", line 240, in get_access
2016-07-29 20:36:41.110 10725 ERROR oslo_messaging.rpc.dispatcher     self.auth_ref = self.get_auth_ref(session)
2016-07-29 20:36:41.110 10725 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/keystoneclient/auth/identity/v3/base.py", line 190, in get_auth_ref
2016-07-29 20:36:41.110 10725 ERROR oslo_messaging.rpc.dispatcher     authenticated=False, log=False, **rkwargs)
2016-07-29 20:36:41.110 10725 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/keystoneclient/session.py", line 501, in post
2016-07-29 20:36:41.110 10725 ERROR oslo_messaging.rpc.dispatcher     return self.request(url, 'POST', **kwargs)
2016-07-29 20:36:41.110 10725 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/keystoneclient/utils.py", line 337, in inner
2016-07-29 20:36:41.110 10725 ERROR oslo_messaging.rpc.dispatcher     return func(*args, **kwargs)
2016-07-29 20:36:41.110 10725 ERROR oslo_messaging.rpc.dispatcher   File "/usr/lib/python2.7/site-packages/keystoneclient/session.py", line 401, in request
2016-07-29 20:36:41.110 10725 ERROR oslo_messaging.rpc.dispatcher     raise exceptions.from_response(resp, method, url)
2016-07-29 20:36:41.110 10725 ERROR oslo_messaging.rpc.dispatcher Forbidden: Trustee has no delegated roles. (Disable debug mode to suppress these details.) (HTTP 403) (Request-ID: req-826f6bd9-2fa5-4254-817e-ed43dff6e474)
2016-07-29 20:36:41.110 10725 ERROR oslo_messaging.rpc.dispatcher

Comment 3 Marios Andreou 2016-08-01 08:32:04 UTC

o/ hi @sasha... this looks like it might be a duplicate of https://bugzilla.redhat.com/show_bug.cgi?id=1325475 - afaics is it also about 7->8 upgrade after 7 update. Looking at the error especially "Trustee has no delegated roles" it looks like it may have the same root cause. Look at the doctext there for a workaround... wdyt?

Comment 4 Sofer Athlan-Guyot 2017-02-09 17:14:38 UTC

Hi,

closing this one as the doc text is now available in the documentation (adding the _member_ role to the admin user during upgrade.)

*** This bug has been marked as a duplicate of bug 1325475 ***

Comment 5 Sofer Athlan-Guyot 2017-02-09 17:34:04 UTC

*** Bug 1361795 has been marked as a duplicate of this bug. ***