Bug 1438965 - radosgw keystone integration - revoked tokens response is missing signed section
Summary: radosgw keystone integration - revoked tokens response is missing signed section
Status: CLOSED ERRATA
Alias: None
Product: Red Hat Ceph Storage
Classification: Red Hat
Component: RGW
Version: 2.3
Hardware: x86_64
OS: Linux
medium
medium
Target Milestone: rc
: 2.*
Assignee: Marcus Watts
QA Contact: shilpa
Erin Donnelly
URL:
Whiteboard:
Keywords: Reopened
Depends On: 1436386 1455856
Blocks: 1437916
TreeView+ depends on / blocked
 
Reported: 2017-04-04 22:00 UTC by Matt Flusche
Modified: 2019-07-08 17:44 UTC (History)
15 users (show)

(edit)
.Two new parameters have been introduced to cope with the errors caused by modern Keystone token types 

The token revocation API that the Ceph Object Gateway uses no longer works with modern token types in OpenStack and Keystone. This causes errors in the Ceph log and Python backtraces in Keystone.

To cope with these errors, two new parameters `rgw_keystone_token_cache_size` and `rgw_keystone_revocation_interval` have been introduced. Setting the `rgw_keystone_toke_cache_size` parameter to 0 in the Ceph configuration file removes the errors. Setting the `rgw_keystone_revocation_interval` parameter to 0 improves performance, but removes the ability to revoke tokens.
Clone Of:
(edit)
Last Closed: 2018-01-30 17:45:45 UTC


Attachments (Terms of Use)


External Trackers
Tracker ID Priority Status Summary Last Updated
Red Hat Product Errata RHBA-2017:1497 normal SHIPPED_LIVE Red Hat Ceph Storage 2.3 bug fix and enhancement update 2017-06-19 17:24:11 UTC
Ceph Project Bug Tracker 9493 None None None 2017-04-14 01:27 UTC
Ceph Project Bug Tracker 19499 None None None 2017-04-14 01:27 UTC
Red Hat Bugzilla 1142424 None CLOSED Allow Keystone revocation list fetching to be disabled 2019-07-17 10:43 UTC

Internal Trackers: 1142424

Description Matt Flusche 2017-04-04 22:00:49 UTC
Description of problem:
Env: OSP 10 Deployment with ceph storage and radosgw.

radosgw/swift seems to work fine; however the following error is produced every 5 minutes in /var/log/ceph/radosgw.log

2017-04-04 16:23:41.038176 7f54fffdf700  0 revoked tokens response is missing signed section
2017-04-04 16:23:41.038192 7f54fffdf700  0 ERROR: keystone revocation processing returned error r=-22

Keystone is configured to use UUID tokens and produces the following exception.

2017-04-04 16:38:41.061 107009 INFO keystone.common.wsgi [req-4b741f7f-129c-4e64-ada0-7d96678f7f7d - - - - -] GET https://osp10.example.com:13357/v2.0/tokens/revoked
2017-04-04 16:38:41.062 107009 WARNING keystone.common.controller [req-4b741f7f-129c-4e64-ada0-7d96678f7f7d - - - - -] RBAC: Bypassing authorization
2017-04-04 16:38:41.077 107009 ERROR keystoneclient.common.cms [req-4b741f7f-129c-4e64-ada0-7d96678f7f7d - - - - -] Signing error: Unable to load certificate - ensure you have configured PKI with "keystone-manage pki_setup"
2017-04-04 16:38:41.078 107009 ERROR keystone.common.wsgi [req-4b741f7f-129c-4e64-ada0-7d96678f7f7d - - - - -] Command 'openssl' returned non-zero exit status 3
2017-04-04 16:38:41.078 107009 ERROR keystone.common.wsgi Traceback (most recent call last):
2017-04-04 16:38:41.078 107009 ERROR keystone.common.wsgi   File "/usr/lib/python2.7/site-packages/keystone/common/wsgi.py", line 225, in __call__
2017-04-04 16:38:41.078 107009 ERROR keystone.common.wsgi     result = method(req, **params)
2017-04-04 16:38:41.078 107009 ERROR keystone.common.wsgi   File "/usr/lib/python2.7/site-packages/oslo_log/versionutils.py", line 174, in wrapped
2017-04-04 16:38:41.078 107009 ERROR keystone.common.wsgi     return func_or_cls(*args, **kwargs)
2017-04-04 16:38:41.078 107009 ERROR keystone.common.wsgi   File "/usr/lib/python2.7/site-packages/keystone/common/controller.py", line 164, in inner
2017-04-04 16:38:41.078 107009 ERROR keystone.common.wsgi     return f(self, request, *args, **kwargs)
2017-04-04 16:38:41.078 107009 ERROR keystone.common.wsgi   File "/usr/lib/python2.7/site-packages/keystone/token/controllers.py", line 471, in revocation_list
2017-04-04 16:38:41.078 107009 ERROR keystone.common.wsgi     CONF.signing.keyfile)
2017-04-04 16:38:41.078 107009 ERROR keystone.common.wsgi   File "/usr/lib/python2.7/site-packages/keystoneclient/common/cms.py", line 325, in cms_sign_text
2017-04-04 16:38:41.078 107009 ERROR keystone.common.wsgi     signing_key_file_name, message_digest=message_digest)
2017-04-04 16:38:41.078 107009 ERROR keystone.common.wsgi   File "/usr/lib/python2.7/site-packages/keystoneclient/common/cms.py", line 373, in cms_sign_data
2017-04-04 16:38:41.078 107009 ERROR keystone.common.wsgi     raise subprocess.CalledProcessError(retcode, 'openssl')
2017-04-04 16:38:41.078 107009 ERROR keystone.common.wsgi CalledProcessError: Command 'openssl' returned non-zero exit status 3
2017-04-04 16:38:41.078 107009 ERROR keystone.common.wsgi

Seems like there are two issues here:

- radosgw should not request revoked certs from keystone when using UUID tokens.
- keystone should handle this request better when not configured for PKI tokens.


Version-Release number of selected component (if applicable):
Ceph:
python-rados-10.2.3-17.el7cp.x86_64
ceph-radosgw-10.2.3-17.el7cp.x86_64
librados2-10.2.3-17.el7cp.x86_64
python-cephfs-10.2.3-17.el7cp.x86_64
ceph-osd-10.2.3-17.el7cp.x86_64
ceph-selinux-10.2.3-17.el7cp.x86_64
ceph-mon-10.2.3-17.el7cp.x86_64
ceph-base-10.2.3-17.el7cp.x86_64
puppet-ceph-2.2.1-3.el7ost.noarch
libcephfs1-10.2.3-17.el7cp.x86_64
ceph-common-10.2.3-17.el7cp.x86_64

Keystone:
python-keystoneclient-3.5.0-1.el7ost.noarch
python-keystonemiddleware-4.9.0-1.el7ost.noarch
python-keystoneauth1-2.12.2-1.el7ost.noarch
python-keystone-10.0.0-4.el7ost.noarch
openstack-keystone-10.0.0-4.el7ost.noarch
puppet-keystone-9.4.0-2.el7ost.noarch


How reproducible:
100% for the one time I attempted

Steps to Reproduce:
1. Deploy OSP 10 w/ ceph and radosgw
2. 
3.

Actual results:


Expected results:


Additional info:

Comment 3 Vikhyat Umrao 2017-04-05 18:45:26 UTC
Actually, this bug is causing the issue in Openstack 10 and RGW 1.3.x integration because now we do not have ca.pem and signing_cert.pem in /etc/keystone/ssl path because as Matt said OSP 10 is using the UUID tokens not the PKI tokens so we can not run below steps for importing certs in RGW node.

mkdir /var/ceph/nss

openssl x509 -in /etc/keystone/ssl/certs/ca.pem -pubkey | \
        certutil -d /var/ceph/nss -A -n ca -t "TCu,Cu,Tuw"
openssl x509 -in /etc/keystone/ssl/certs/signing_cert.pem -pubkey | \
        certutil -A -d /var/ceph/nss -n signing_cert -t "P,P,P"

and this is causing this issue. In OSP 8 all is working, I had the similar issue in OSP8 but when I imported certs with above steps all good. Same was recommended in the upstream tracker: http://tracker.ceph.com/issues/17186#note-3.

Comment 19 shilpa 2017-05-31 09:40:17 UTC
After setting these values, the errors can no longer be seen:
rgw keystone token cache size = 0
or
rgw keystone revocation interval = 0

Comment 23 errata-xmlrpc 2017-06-19 13:31:32 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2017:1497


Note You need to log in before you can comment on or make changes to this bug.