Description of problem: Hi, I'm attempting to enable the S3 APIs in Swift with authentication via Keystone. According to this page from the Grizzly docs: http://docs.openstack.org/grizzly/openstack-compute/admin/content/configuring-swift-with-s3-emulation-to-use-keystone.html the s3token middleware is needed to enable Keystone authentication. However after adding s3token to the pipeline I get this error in swift-startup.log: Traceback (most recent call last): File "/usr/bin/swift-proxy-server", line 23, in <module> sys.exit(run_wsgi(conf_file, 'proxy-server', default_port=8080, **options)) File "/usr/lib/python2.6/site-packages/swift/common/wsgi.py", line 257, in run_wsgi loadapp(conf_path, global_conf={'log_name': log_name}) File "/usr/lib/python2.6/site-packages/swift/common/wsgi.py", line 107, in wrapper return f(conf_uri, *args, **kwargs) File "/usr/lib/python2.6/site-packages/PasteDeploy-1.5.0-py2.6.egg/paste/deploy/loadwsgi.py", line 247, in loadapp return loadobj(APP, uri, name=name, **kw) File "/usr/lib/python2.6/site-packages/PasteDeploy-1.5.0-py2.6.egg/paste/deploy/loadwsgi.py", line 271, in loadobj global_conf=global_conf) File "/usr/lib/python2.6/site-packages/PasteDeploy-1.5.0-py2.6.egg/paste/deploy/loadwsgi.py", line 296, in loadcontext global_conf=global_conf) File "/usr/lib/python2.6/site-packages/PasteDeploy-1.5.0-py2.6.egg/paste/deploy/loadwsgi.py", line 320, in _loadconfig return loader.get_context(object_type, name, global_conf) File "/usr/lib/python2.6/site-packages/swift/common/wsgi.py", line 55, in get_context object_type, name=name, global_conf=global_conf) File "/usr/lib/python2.6/site-packages/PasteDeploy-1.5.0-py2.6.egg/paste/deploy/loadwsgi.py", line 450, in get_context global_additions=global_additions) File "/usr/lib/python2.6/site-packages/PasteDeploy-1.5.0-py2.6.egg/paste/deploy/loadwsgi.py", line 562, in _pipeline_app_context for name in pipeline[:-1]] File "/usr/lib/python2.6/site-packages/swift/common/wsgi.py", line 55, in get_context object_type, name=name, global_conf=global_conf) File "/usr/lib/python2.6/site-packages/PasteDeploy-1.5.0-py2.6.egg/paste/deploy/loadwsgi.py", line 458, in get_context section) File "/usr/lib/python2.6/site-packages/PasteDeploy-1.5.0-py2.6.egg/paste/deploy/loadwsgi.py", line 517, in _context_from_explicit value = import_string(found_expr) File "/usr/lib/python2.6/site-packages/PasteDeploy-1.5.0-py2.6.egg/paste/deploy/loadwsgi.py", line 22, in import_string return pkg_resources.EntryPoint.parse("x=" + s).load(False) File "/usr/lib/python2.6/site-packages/pkg_resources.py", line 1948, in load entry = __import__(self.module_name, globals(),globals(), ['__name__']) File "/usr/lib/python2.6/site-packages/keystoneclient/middleware/s3_token.py", line 35, in <module> import webob ImportError: No module named webob I found that we have the package python-webob1.2 installed on the Swift proxy servers (exact version: 1.2.3-2.2.el6ost), but obviously it's not being found. After further investigation, I found this bugzilla: https://bugzilla.redhat.com/show_bug.cgi?id=971026 Which seems to indicate that this exact issue was corrected in python-keystoneclient-0.2.3-4.el6ost a year ago. However we still seem to be experiencing this issue despite having a newer version (python-keystoneclient-0.7.1-2.el6ost) installed. I've also found that installing the package python-webob (version: 0.9.6.1-3.el6) gets round this issue and allows Swift to start with the s3token middleware in the pipeline. However as this appears to be a recurrence of a known and previously fixed bug I thought I'd ask for help. So I have 2 questions: 1. Is the s3token middleware still required for S3 API authentication via Keystone please? 2. Is there a way to get the s3token middleware working with python-webob1.2? Actual results: Expected results: Additional info:
Which is NVR of openstack-swift RPM?
To answer Eduard's questions: 1. Yes, s3_token is required. 2. One could make the s3_token working with python-webob1.2 by applying a corrective patch in the field, taken from bug 971026. Of course you have to re-apply this by hand every time there's an update. Most likely, we'll make sure that the same patch is applied again in order to fix this bug (I'll talk it over with Alan). That, however, is still undesirable, I think. The reason is, the middleware creates a webob.Request, then feeds it down into the rest of Swift. But Swift switched to swob, which may not be perfectly compatible. When it operates on a request, Swift core considers it a swob.Request. I need to come up with a solution like this: https://review.openstack.org/32825 Unfortunately, fixing s3_token is trickier, since it uses a full Request.
> 2. One could make the s3_token working with python-webob1.2 by > applying a corrective patch in the field, taken from bug 971026. > Of course you have to re-apply this by hand every time there's > an update. > > Most likely, we'll make sure that the same patch is applied > again in order to fix this bug (I'll talk it over with Alan). Note: that would apply to rhos4 only, in rhos5 we dropped parallel versions and updated webob to python-webob-1.2.3-8.el6ost > That, however, is still undesirable, I think. The reason is, > the middleware creates a webob.Request, then feeds it down > into the rest of Swift. But Swift switched to swob, which > may not be perfectly compatible. When it operates on a request, > Swift core considers it a swob.Request. Seems that swob.Request is compatible enough with python-webob-0.9.6.1-3.el6 for duck-typing to work so we could just add dep on python-webob as a workaround until proper fix is done upstream.
Created attachment 931374 [details] proposed patch for RHOS4
Created attachment 931388 [details] proposed patch for RHOS4 <= Icehouse keystoneclient includes s3_token only (ec2_token was moved, along with s3_token m/w to keystonemiddleware project in Juno)
Created attachment 931389 [details] proposed patch for RHOS5
(In reply to Alan Pevec from comment #7) > Created attachment 931388 [details] > proposed patch for RHOS4 Even better than the add-dist magic. I'm sad about extra dependency, but it can't be helped, it appears. RHOS 4 only.
Can someone tell me what to do after this step in order to configure s3token ? After installing webob 0.9.6 and configuring the s3token middleware like this in /etc/swift/proxy-server.conf: ... [pipeline:main] pipeline = healthcheck cache swift3 authtoken keystone ceilometer proxy-server ... [filter:s3token] paste.filter_factory = keystoneclient.middleware.s3_token:filter_factory auth_port = 35357 auth_protocol = http auth_host = keystone.xxx.ch I'm getting the following error in /var/log/swift-startup.log: /usr/lib/python2.6/site-packages/ceilometer/__init__.py:34: UserWarning: Module webob was already imported from /usr/lib/python2.6/site-packages/webob/__init__.pyc, but /usr/lib/python2.6/site-packages/WebOb-1.2.3-py2.6.egg is being added to sys.path return pkg_resources.require(requirement) And this error from the server when attempting to test the S3 API using the s3curl tool: $ ./s3-curl/s3curl.pl --debug --id=swsos http://swift.xxx.ch:8080/ s3curl: Found the url: host=swift.xxx.ch; port=8080; uri=/; query=; s3curl: ordinary endpoint signing case s3curl: StringToSign='GET\n\n\nTue, 09 Sep 2014 09:56:15 +0000\n/' s3curl: exec curl -H Date: Tue, 09 Sep 2014 09:56:15 +0000 -H Authorization: AWS user:password -L -H content-type: http://swift.xxx.ch:8080/ Traceback (most recent call last): File "/usr/lib/python2.6/site-packages/eventlet/wsgi.py", line 389, in handle_one_response result = self.application(self.environ, start_response) File "/usr/lib/python2.6/site-packages/swift/common/middleware/healthcheck.py", line 57, in __call__ return self.app(env, start_response) File "/usr/lib/python2.6/site-packages/swift/common/middleware/memcache.py", line 67, in __call__ return self.app(env, start_response) File "/usr/lib/python2.6/site-packages/swift3/middleware.py", line 516, in __call__ res = getattr(controller, req.method)(env, start_response) File "/usr/lib/python2.6/site-packages/swift3/middleware.py", line 181, in GET body_iter = self._app_call(env) File "/usr/lib/python2.6/site-packages/swift/common/wsgi.py", line 390, in _app_call resp = self.app(env, self._start_response) File "/usr/lib/python2.6/site-packages/keystoneclient/middleware/s3_token.py", line 233, in __call__ self.logger.debug(msg % (resp.status_code)) AttributeError: 'Response' object has no attribute 'status_code' It seems that enabling the s3token middleware with webob 0.9 basically breaks everything! I need to get the S3 API working, so what's our next step please? Thanks
Created attachment 936085 [details] proposed patch for RHOS4 with Pete's fix from rhbz#971026
Comment on attachment 936085 [details] proposed patch for RHOS4 with Pete's fix from rhbz#971026 Fix verified to work for the original report (using python-webob1.2).
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://rhn.redhat.com/errata/RHSA-2014-1784.html