Bug 1303246

Without knowing for sure, I would guess we're going to need a lot more logs to get an idea of what's going on here. 

What was the result of the token replacement above?

Token replacement did not work. My guess is the db has a new admin password and the rc files didn't get written with the new password.

another test

Tried updating password with no luck

  /etc/keystone/keystone.conf 
  admin_token = 2c7d0ddb1aed4579172cbba4105e28e6a7f69587 
  #admin_token = 3f038ac3099362ac23b53cfc4b38ecae39e78825 

tried both tokens

login as root to your controller
export OS_SERVICE_TOKEN=2c7d0ddb1aed4579172cbba4105e28e6a7f69587
export OS_SERVICE_ENDPOINT=http://10.133.165.20:35357/v2.0/
keystone user-password-update --pass a96d995c317ffe65d7d29157774c1d67f90924e0 admin

didn't work, also didn't get the following warning. I tested this on my machine(s) and it works. 
 
WARNING: Bypassing authentication using a token & endpoint (authentication credentials are being ignored).

We tried:

1) changing the tokens to what was listed in tripleo-overcloud-passwords
2) using token only auth to change the passwords
3) Rerunning the deploy command with only the original computes
4) using heat stack-update-cancel
5) redefining the new node and rerunning the scale up deploy command

all failed.

Current state is:

1) still cannot use the admin account to access the controllers. Other user accounts evidently work fine.
2) the stack is in UPDATE_FAILED state with compute and controller resources marked as failed.

There is newer code. However, I'd rather get engineering feedback before attempting to upgrade since this is a running system integrated with openshift and cloudforms.

I noticed that admin_tenant_name=service in nova.conf and in my nova.conf is services.

If I change it to "service" instead of "services" I get very similar errors but not the same:

# nova list
ERROR (Unauthorized): Unauthorized (HTTP 401) (Request-ID: req-87938873-b5d0-41e9-ab2c-12f506e4c040)



2016-01-31 08:30:37.865 5757 DEBUG keystoneclient.auth.identity.v2 [-] Making authentication request to http://192.168.101.196:35357/v2.0/tokens get_auth_ref /usr/lib/python2.7/site-packages/keystoneclient/auth/identity/v2.py:76
2016-01-31 08:30:37.920 5757 DEBUG keystoneclient.session [-] Request returned failure status: 401 request /usr/lib/python2.7/site-packages/keystoneclient/session.py:396
2016-01-31 08:30:37.921 5757 WARNING keystonemiddleware.auth_token [-] Identity response: {"error": {"message": "Could not find project: service (Disable debug mode to suppress these details.)", "code": 401, "title": "Unauthorized"}}
2016-01-31 08:30:37.921 5757 DEBUG keystoneclient.auth.identity.v2 [-] Making authentication request to http://192.168.101.196:35357/v2.0/tokens get_auth_ref /usr/lib/python2.7/site-packages/keystoneclient/auth/identity/v2.py:76
2016-01-31 08:30:37.972 5757 DEBUG keystoneclient.session [-] Request returned failure status: 401 request /usr/lib/python2.7/site-packages/keystoneclient/session.py:396
2016-01-31 08:30:37.973 5757 WARNING keystonemiddleware.auth_token [-] Identity response: {"error": {"message": "Could not find project: service (Disable debug mode to suppress these details.)", "code": 401, "title": "Unauthorized"}}  <=================
2016-01-31 08:30:37.973 5757 WARNING keystonemiddleware.auth_token [-] Authorization failed for token
2016-01-31 08:30:37.973 5757 INFO nova.osapi_compute.wsgi.server [-] 192.168.101.196 "GET /v2/fb2c259140e4405aad4a3867afc6f95f/servers/detail HTTP/1.1" status: 401 len: 290 time: 0.1090090

let me compare with customer logs:


/site-packages/keystoneclient/auth/identity/v2.py:76
2016-01-29 12:41:28.504 35696 DEBUG keystoneclient.session [-] Request returned failure status: 401 request /usr/lib/python2.7/site-packages/keystoneclient/session.py:396
2016-01-29 12:41:28.504 35696 WARNING keystonemiddleware.auth_token [-] Identity response: {"error": {"message": "Invalid user / password (Disable debug mode to suppress these details.)", "code": 401, "title": "Unauthorized"}}
2016-01-29 12:41:28.505 35696 WARNING keystonemiddleware.auth_token [-] Authorization failed for token
2016-01-29 12:41:28.505 35696 INFO nova.osapi_compute.wsgi.server [-] 10.133.162.9 "GET /v2/ef9184f8a90544c587036d90e9f7c362/servers/detail?limit=21&project_id=ef9184f8a90544c587036d90e9f7c362 HTTP/1.1" status: 401 len: 288 time: 0.1541209
2016-01-29 12:41:28.587 35649 WARNING keystonemiddleware.auth_token [-] Unable to find authentication token in headers
2016-01-29 12:41:28.588 35649 INFO nova.osapi_compute.wsgi.server [-] 10.133.162.9 "GET /v2/ef9184f8a90544c587036d90e9f7c362 HTTP/1.1" status: 401 len: 288 time: 0.0022380

No dice on the services change. 

BTW, my controllers use "service" instead of "services" and it works fine.
# rpm -qa | grep keystone
python-keystoneclient-1.3.0-2.el7ost.noarch
openstack-keystone-2015.1.0-4.el7ost.noarch
python-keystonemiddleware-1.5.1-1.el7ost.noarch
python-keystone-2015.1.0-4.el7ost.noarch

This bug did not make the OSP 8.0 release.  It is being deferred to OSP 10.

Seems outdated, please re-open if the concern is still valid.