Bug 1438965

Summary:	radosgw keystone integration - revoked tokens response is missing signed section
Product:	[Red Hat Storage] Red Hat Ceph Storage	Reporter:	Matt Flusche <mflusche>
Component:	RGW	Assignee:	Marcus Watts <mwatts>
Status:	CLOSED ERRATA	QA Contact:	shilpa <smanjara>
Severity:	medium	Docs Contact:	Erin Donnelly <edonnell>
Priority:	medium
Version:	2.3	CC:	cbodley, ceph-eng-bugs, edonnell, gkadam, hnallurv, kbader, kdreyer, mbenjamin, mflusche, mwatts, nchandek, owasserm, smanjara, sweil, vumrao
Target Milestone:	rc	Keywords:	Reopened
Target Release:	2.*
Hardware:	x86_64
OS:	Linux
Whiteboard:
Fixed In Version:	RHEL: ceph-10.2.7-12.el7cp Ubuntu: ceph_10.2.7-14redhat1xenial	Doc Type:	Bug Fix
Doc Text:	.Two new parameters have been introduced to cope with the errors caused by modern Keystone token types The token revocation API that the Ceph Object Gateway uses no longer works with modern token types in OpenStack and Keystone. This causes errors in the Ceph log and Python backtraces in Keystone. To cope with these errors, two new parameters `rgw_keystone_token_cache_size` and `rgw_keystone_revocation_interval` have been introduced. Setting the `rgw_keystone_toke_cache_size` parameter to 0 in the Ceph configuration file removes the errors. Setting the `rgw_keystone_revocation_interval` parameter to 0 improves performance, but removes the ability to revoke tokens.	Story Points:	---
Clone Of:		Environment:
Last Closed:	2018-01-30 17:45:45 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:
Bug Depends On:	1436386, 1455856
Bug Blocks:	1437916

Description Matt Flusche 2017-04-04 22:00:49 UTC

Description of problem:
Env: OSP 10 Deployment with ceph storage and radosgw.

radosgw/swift seems to work fine; however the following error is produced every 5 minutes in /var/log/ceph/radosgw.log

2017-04-04 16:23:41.038176 7f54fffdf700  0 revoked tokens response is missing signed section
2017-04-04 16:23:41.038192 7f54fffdf700  0 ERROR: keystone revocation processing returned error r=-22

Keystone is configured to use UUID tokens and produces the following exception.

2017-04-04 16:38:41.061 107009 INFO keystone.common.wsgi [req-4b741f7f-129c-4e64-ada0-7d96678f7f7d - - - - -] GET https://osp10.example.com:13357/v2.0/tokens/revoked
2017-04-04 16:38:41.062 107009 WARNING keystone.common.controller [req-4b741f7f-129c-4e64-ada0-7d96678f7f7d - - - - -] RBAC: Bypassing authorization
2017-04-04 16:38:41.077 107009 ERROR keystoneclient.common.cms [req-4b741f7f-129c-4e64-ada0-7d96678f7f7d - - - - -] Signing error: Unable to load certificate - ensure you have configured PKI with "keystone-manage pki_setup"
2017-04-04 16:38:41.078 107009 ERROR keystone.common.wsgi [req-4b741f7f-129c-4e64-ada0-7d96678f7f7d - - - - -] Command 'openssl' returned non-zero exit status 3
2017-04-04 16:38:41.078 107009 ERROR keystone.common.wsgi Traceback (most recent call last):
2017-04-04 16:38:41.078 107009 ERROR keystone.common.wsgi   File "/usr/lib/python2.7/site-packages/keystone/common/wsgi.py", line 225, in __call__
2017-04-04 16:38:41.078 107009 ERROR keystone.common.wsgi     result = method(req, **params)
2017-04-04 16:38:41.078 107009 ERROR keystone.common.wsgi   File "/usr/lib/python2.7/site-packages/oslo_log/versionutils.py", line 174, in wrapped
2017-04-04 16:38:41.078 107009 ERROR keystone.common.wsgi     return func_or_cls(*args, **kwargs)
2017-04-04 16:38:41.078 107009 ERROR keystone.common.wsgi   File "/usr/lib/python2.7/site-packages/keystone/common/controller.py", line 164, in inner
2017-04-04 16:38:41.078 107009 ERROR keystone.common.wsgi     return f(self, request, *args, **kwargs)
2017-04-04 16:38:41.078 107009 ERROR keystone.common.wsgi   File "/usr/lib/python2.7/site-packages/keystone/token/controllers.py", line 471, in revocation_list
2017-04-04 16:38:41.078 107009 ERROR keystone.common.wsgi     CONF.signing.keyfile)
2017-04-04 16:38:41.078 107009 ERROR keystone.common.wsgi   File "/usr/lib/python2.7/site-packages/keystoneclient/common/cms.py", line 325, in cms_sign_text
2017-04-04 16:38:41.078 107009 ERROR keystone.common.wsgi     signing_key_file_name, message_digest=message_digest)
2017-04-04 16:38:41.078 107009 ERROR keystone.common.wsgi   File "/usr/lib/python2.7/site-packages/keystoneclient/common/cms.py", line 373, in cms_sign_data
2017-04-04 16:38:41.078 107009 ERROR keystone.common.wsgi     raise subprocess.CalledProcessError(retcode, 'openssl')
2017-04-04 16:38:41.078 107009 ERROR keystone.common.wsgi CalledProcessError: Command 'openssl' returned non-zero exit status 3
2017-04-04 16:38:41.078 107009 ERROR keystone.common.wsgi

Seems like there are two issues here:

- radosgw should not request revoked certs from keystone when using UUID tokens.
- keystone should handle this request better when not configured for PKI tokens.


Version-Release number of selected component (if applicable):
Ceph:
python-rados-10.2.3-17.el7cp.x86_64
ceph-radosgw-10.2.3-17.el7cp.x86_64
librados2-10.2.3-17.el7cp.x86_64
python-cephfs-10.2.3-17.el7cp.x86_64
ceph-osd-10.2.3-17.el7cp.x86_64
ceph-selinux-10.2.3-17.el7cp.x86_64
ceph-mon-10.2.3-17.el7cp.x86_64
ceph-base-10.2.3-17.el7cp.x86_64
puppet-ceph-2.2.1-3.el7ost.noarch
libcephfs1-10.2.3-17.el7cp.x86_64
ceph-common-10.2.3-17.el7cp.x86_64

Keystone:
python-keystoneclient-3.5.0-1.el7ost.noarch
python-keystonemiddleware-4.9.0-1.el7ost.noarch
python-keystoneauth1-2.12.2-1.el7ost.noarch
python-keystone-10.0.0-4.el7ost.noarch
openstack-keystone-10.0.0-4.el7ost.noarch
puppet-keystone-9.4.0-2.el7ost.noarch


How reproducible:
100% for the one time I attempted

Steps to Reproduce:
1. Deploy OSP 10 w/ ceph and radosgw
2. 
3.

Actual results:


Expected results:


Additional info:

Comment 3 Vikhyat Umrao 2017-04-05 18:45:26 UTC

Actually, this bug is causing the issue in Openstack 10 and RGW 1.3.x integration because now we do not have ca.pem and signing_cert.pem in /etc/keystone/ssl path because as Matt said OSP 10 is using the UUID tokens not the PKI tokens so we can not run below steps for importing certs in RGW node.

mkdir /var/ceph/nss

openssl x509 -in /etc/keystone/ssl/certs/ca.pem -pubkey | \
        certutil -d /var/ceph/nss -A -n ca -t "TCu,Cu,Tuw"
openssl x509 -in /etc/keystone/ssl/certs/signing_cert.pem -pubkey | \
        certutil -A -d /var/ceph/nss -n signing_cert -t "P,P,P"

and this is causing this issue. In OSP 8 all is working, I had the similar issue in OSP8 but when I imported certs with above steps all good. Same was recommended in the upstream tracker: http://tracker.ceph.com/issues/17186#note-3.

Comment 19 shilpa 2017-05-31 09:40:17 UTC

After setting these values, the errors can no longer be seen:
rgw keystone token cache size = 0
or
rgw keystone revocation interval = 0

Comment 23 errata-xmlrpc 2017-06-19 13:31:32 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2017:1497