Bug 1393435

Summary: Token (fernet or uuid) for admin user is not valid after overcloud redeploy
Product: Red Hat OpenStack Reporter: Rodrigo Duarte <rduartes>
Component: openstack-tripleoAssignee: James Slagle <jslagle>
Status: CLOSED WONTFIX QA Contact: Arik Chernetsky <achernet>
Severity: medium Docs Contact:
Priority: unspecified    
Version: 10.0 (Newton)CC: jdennis, mburns, nkinder, rhel-osp-director-maint, srevivo
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2017-02-17 18:45:29 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:

Description Rodrigo Duarte 2016-11-09 14:40:05 UTC
Description of problem:

Fernet tokens are not persisted, keys that are stored in the file system are used to decrypt the token and then keystone is able to validate it. Overcloud redeploys that don't affect the token provider settings in keystone or the keys should not invalidate the tokens previously issued.

Version-Release number of selected component (if applicable):

openstack-keystone-10.0.0-3.el7ost.noarch

How reproducible:

Every time.

Steps to Reproduce:
1. Configure fernet as token provider
  1.1. Check the keys permissions, content and modification time
2. Issue a token and use it to perform basic operations
3. Redeploy the overcloud and check the fernet settings didn't change neither the keys
  3.1. Verify the keys permissions, content and modification time against the ones from step 1.1.
4. Reuse the token issued in step 2
5. Issue a new token and use it to perform basic operations

Actual results:

In step 4, the token is invalid.

Expected results:

Token is valid in step 4.

Additional info:

- Request response:

{"error": {"message": "The request you have made requires authentication.", "code": 401, "title": "Unauthorized"}}

- Keystone logs:

2016-11-08 22:51:27.966 17730 DEBUG keystone.middleware.auth [req-26c14a0e-9989-449d-9a29-4e32b57dcc3d - - - - -] Authenticating user token process_request /usr/lib/python2.7/site-packages/keystonemiddleware/auth_token/__init__.py:346
2016-11-08 22:51:27.969 17730 DEBUG keystone.common.fernet_utils [req-26c14a0e-9989-449d-9a29-4e32b57dcc3d - - - - -] Loaded 2 Fernet keys from /etc/keystone/fernet-keys, but `[fernet_tokens] max_active_keys = 3`; perhaps there have not been enough key rotations to reach `max_active_keys` yet? load_keys /usr/lib/python2.7/site-packages/keystone/common/fernet_utils.py:269
2016-11-08 22:51:28.134 17730 INFO keystone.middleware.auth [req-26c14a0e-9989-449d-9a29-4e32b57dcc3d - - - - -] Invalid user token
2016-11-08 22:51:28.135 17730 DEBUG keystone.middleware.auth [req-2ba4a754-f72e-4829-a013-789e4fec75f4 - - - - -] There is either no auth token in the request or the certificate issuer is not trusted. No auth context will be set. fill_context /usr/lib/python2.7/site-packages/keystone/middleware/auth.py:210
2016-11-08 22:51:28.137 17730 INFO keystone.common.wsgi [req-2ba4a754-f72e-4829-a013-789e4fec75f4 - - - - -] GET https://10.0.0.101:13000/v3/domains
2016-11-08 22:51:28.137 17730 WARNING keystone.common.wsgi [req-2ba4a754-f72e-4829-a013-789e4fec75f4 - - - - -] Authorization failed. The request you have made requires authentication. from 10.0.0.42

Comment 1 Rodrigo Duarte 2016-11-11 19:29:49 UTC
Confirmed the token is rejected by a revocation event:

- Before redeploy:

MariaDB [keystone]> select * from revocation_event;
+----+-----------+------------+----------------------------------+---------+----------+-------------+-----------------+---------------------+------------+---------------------+----------+----------------+
| id | domain_id | project_id | user_id                          | role_id | trust_id | consumer_id | access_token_id | issued_before       | expires_at | revoked_at          | audit_id | audit_chain_id |
+----+-----------+------------+----------------------------------+---------+----------+-------------+-----------------+---------------------+------------+---------------------+----------+----------------+
| 24 | NULL      | NULL       | a50b3b7ad47f4fe0a473bb5d5c5a378e | NULL    | NULL     | NULL        | NULL            | 2016-11-09 16:01:18 | NULL       | 2016-11-09 16:01:18 | NULL     | NULL           |
| 26 | NULL      | NULL       | a50b3b7ad47f4fe0a473bb5d5c5a378e | NULL    | NULL     | NULL        | NULL            | 2016-11-09 16:01:56 | NULL       | 2016-11-09 16:01:56 | NULL     | NULL           |
+----+-----------+------------+----------------------------------+---------+----------+-------------+-----------------+---------------------+------------+---------------------+----------+----------------+

- After redeploy

MariaDB [keystone]> select * from revocation_event;
+----+-----------+------------+----------------------------------+---------+----------+-------------+-----------------+---------------------+------------+---------------------+----------+----------------+
| id | domain_id | project_id | user_id                          | role_id | trust_id | consumer_id | access_token_id | issued_before       | expires_at | revoked_at          | audit_id | audit_chain_id |
+----+-----------+------------+----------------------------------+---------+----------+-------------+-----------------+---------------------+------------+---------------------+----------+----------------+
| 28 | NULL      | NULL       | a50b3b7ad47f4fe0a473bb5d5c5a378e | NULL    | NULL     | NULL        | NULL            | 2016-11-11 18:44:41 | NULL       | 2016-11-11 18:44:41 | NULL     | NULL           |
| 30 | NULL      | NULL       | a50b3b7ad47f4fe0a473bb5d5c5a378e | NULL    | NULL     | NULL        | NULL            | 2016-11-11 18:45:19 | NULL       | 2016-11-11 18:45:19 | NULL     | NULL           |
+----+-----------+------------+----------------------------------+---------+----------+-------------+-----------------+---------------------+------------+---------------------+----------+----------------+

Also tested with UUID tokens (issue token before the redeploy and try to reuse it afterwards) with the same result.

Comment 2 Rodrigo Duarte 2016-11-11 22:42:47 UTC
Tokens from a custom created user weren't revoked. After the redeploy I was able to reuse a token previously issued.

Comment 3 Nathan Kinder 2017-02-17 18:45:29 UTC
This is an issue that really shouldn't impact anything.  The admin user is the only user affected, and they can simply request a new token after performing a redeploy.  Closing as WONTFIX.