Description of problem: Env: OSP 10 Deployment with ceph storage and radosgw. radosgw/swift seems to work fine; however the following error is produced every 5 minutes in /var/log/ceph/radosgw.log 2017-04-04 16:23:41.038176 7f54fffdf700 0 revoked tokens response is missing signed section 2017-04-04 16:23:41.038192 7f54fffdf700 0 ERROR: keystone revocation processing returned error r=-22 Keystone is configured to use UUID tokens and produces the following exception. 2017-04-04 16:38:41.061 107009 INFO keystone.common.wsgi [req-4b741f7f-129c-4e64-ada0-7d96678f7f7d - - - - -] GET https://osp10.example.com:13357/v2.0/tokens/revoked 2017-04-04 16:38:41.062 107009 WARNING keystone.common.controller [req-4b741f7f-129c-4e64-ada0-7d96678f7f7d - - - - -] RBAC: Bypassing authorization 2017-04-04 16:38:41.077 107009 ERROR keystoneclient.common.cms [req-4b741f7f-129c-4e64-ada0-7d96678f7f7d - - - - -] Signing error: Unable to load certificate - ensure you have configured PKI with "keystone-manage pki_setup" 2017-04-04 16:38:41.078 107009 ERROR keystone.common.wsgi [req-4b741f7f-129c-4e64-ada0-7d96678f7f7d - - - - -] Command 'openssl' returned non-zero exit status 3 2017-04-04 16:38:41.078 107009 ERROR keystone.common.wsgi Traceback (most recent call last): 2017-04-04 16:38:41.078 107009 ERROR keystone.common.wsgi File "/usr/lib/python2.7/site-packages/keystone/common/wsgi.py", line 225, in __call__ 2017-04-04 16:38:41.078 107009 ERROR keystone.common.wsgi result = method(req, **params) 2017-04-04 16:38:41.078 107009 ERROR keystone.common.wsgi File "/usr/lib/python2.7/site-packages/oslo_log/versionutils.py", line 174, in wrapped 2017-04-04 16:38:41.078 107009 ERROR keystone.common.wsgi return func_or_cls(*args, **kwargs) 2017-04-04 16:38:41.078 107009 ERROR keystone.common.wsgi File "/usr/lib/python2.7/site-packages/keystone/common/controller.py", line 164, in inner 2017-04-04 16:38:41.078 107009 ERROR keystone.common.wsgi return f(self, request, *args, **kwargs) 2017-04-04 16:38:41.078 107009 ERROR keystone.common.wsgi File "/usr/lib/python2.7/site-packages/keystone/token/controllers.py", line 471, in revocation_list 2017-04-04 16:38:41.078 107009 ERROR keystone.common.wsgi CONF.signing.keyfile) 2017-04-04 16:38:41.078 107009 ERROR keystone.common.wsgi File "/usr/lib/python2.7/site-packages/keystoneclient/common/cms.py", line 325, in cms_sign_text 2017-04-04 16:38:41.078 107009 ERROR keystone.common.wsgi signing_key_file_name, message_digest=message_digest) 2017-04-04 16:38:41.078 107009 ERROR keystone.common.wsgi File "/usr/lib/python2.7/site-packages/keystoneclient/common/cms.py", line 373, in cms_sign_data 2017-04-04 16:38:41.078 107009 ERROR keystone.common.wsgi raise subprocess.CalledProcessError(retcode, 'openssl') 2017-04-04 16:38:41.078 107009 ERROR keystone.common.wsgi CalledProcessError: Command 'openssl' returned non-zero exit status 3 2017-04-04 16:38:41.078 107009 ERROR keystone.common.wsgi Seems like there are two issues here: - radosgw should not request revoked certs from keystone when using UUID tokens. - keystone should handle this request better when not configured for PKI tokens. Version-Release number of selected component (if applicable): Ceph: python-rados-10.2.3-17.el7cp.x86_64 ceph-radosgw-10.2.3-17.el7cp.x86_64 librados2-10.2.3-17.el7cp.x86_64 python-cephfs-10.2.3-17.el7cp.x86_64 ceph-osd-10.2.3-17.el7cp.x86_64 ceph-selinux-10.2.3-17.el7cp.x86_64 ceph-mon-10.2.3-17.el7cp.x86_64 ceph-base-10.2.3-17.el7cp.x86_64 puppet-ceph-2.2.1-3.el7ost.noarch libcephfs1-10.2.3-17.el7cp.x86_64 ceph-common-10.2.3-17.el7cp.x86_64 Keystone: python-keystoneclient-3.5.0-1.el7ost.noarch python-keystonemiddleware-4.9.0-1.el7ost.noarch python-keystoneauth1-2.12.2-1.el7ost.noarch python-keystone-10.0.0-4.el7ost.noarch openstack-keystone-10.0.0-4.el7ost.noarch puppet-keystone-9.4.0-2.el7ost.noarch How reproducible: 100% for the one time I attempted Steps to Reproduce: 1. Deploy OSP 10 w/ ceph and radosgw 2. 3. Actual results: Expected results: Additional info:
Actually, this bug is causing the issue in Openstack 10 and RGW 1.3.x integration because now we do not have ca.pem and signing_cert.pem in /etc/keystone/ssl path because as Matt said OSP 10 is using the UUID tokens not the PKI tokens so we can not run below steps for importing certs in RGW node. mkdir /var/ceph/nss openssl x509 -in /etc/keystone/ssl/certs/ca.pem -pubkey | \ certutil -d /var/ceph/nss -A -n ca -t "TCu,Cu,Tuw" openssl x509 -in /etc/keystone/ssl/certs/signing_cert.pem -pubkey | \ certutil -A -d /var/ceph/nss -n signing_cert -t "P,P,P" and this is causing this issue. In OSP 8 all is working, I had the similar issue in OSP8 but when I imported certs with above steps all good. Same was recommended in the upstream tracker: http://tracker.ceph.com/issues/17186#note-3.
After setting these values, the errors can no longer be seen: rgw keystone token cache size = 0 or rgw keystone revocation interval = 0
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2017:1497