Hide Forgot
With RDO-Icehouse (m2 testday packages) on RHEL-6.5, negative Tempest identity tests fails as Keystone responds with HTTP 400. For example test tempest.api.identity.admin.v3.test_projects.ProjectsTestJSON.test_project_create_duplicate gives following output: > Request: POST http://192.168.1.16:35357/v3/projects > Request Headers: {'Content-Type': 'application/json', 'Accept': 'application/json', 'X-Auth-Token': '<Token omitted>'} > Request Body: {"project": {"enabled": true, "description": null, "name": "project-dup--1737299968", "domain_id": "default"}} > Response Status: 400 > Response Headers: {'content-length': '143', 'date': 'Tue, 04 Feb 2014 07:47:53 GMT', 'content-type': 'application/json', 'vary': 'X-Auth-Token', 'connection': 'close'} > Response Body: {"error": {"message": "exceptions must be old-style classes or derived from BaseException, not NoneType", "code": 400, "title": "Bad Request"}} In keystone.log the exception can be seen as > keystone.common.wsgi Traceback (most recent call last): > keystone.common.wsgi File "/usr/lib/python2.6/site-packages/keystone/common/wsgi.py", line 214, in __call__ > keystone.common.wsgi result = method(context, **params) > keystone.common.wsgi File "/usr/lib/python2.6/site-packages/keystone/common/controller.py", line 174, in inner > keystone.common.wsgi return f(self, context, *args, **kwargs) > keystone.common.wsgi File "/usr/lib/python2.6/site-packages/keystone/assignment/controllers.py", line 390, in create_project > keystone.common.wsgi ref = self.assignment_api.create_project(ref['id'], ref) > keystone.common.wsgi File "/usr/lib/python2.6/site-packages/keystone/notifications.py", line 53, in wrapper > keystone.common.wsgi result = f(*args, **kwargs) > keystone.common.wsgi File "/usr/lib/python2.6/site-packages/keystone/assignment/core.py", line 72, in create_project > keystone.common.wsgi ret = self.driver.create_project(tenant_id, tenant) > keystone.common.wsgi File "/usr/lib/python2.6/site-packages/keystone/common/sql/core.py", line 165, in wrapper > keystone.common.wsgi return method(*args, **kwargs) > keystone.common.wsgi File "/usr/lib/python2.6/site-packages/keystone/assignment/backends/sql.py", line 411, in create_project > keystone.common.wsgi return tenant_ref.to_dict() > keystone.common.wsgi File "/usr/lib64/python2.6/contextlib.py", line 23, in __exit__ > keystone.common.wsgi self.gen.next() > keystone.common.wsgi File "/usr/lib/python2.6/site-packages/keystone/common/sql/core.py", line 156, in transaction > keystone.common.wsgi yield session > keystone.common.wsgi File "/usr/lib64/python2.6/site-packages/SQLAlchemy-0.7.8-py2.6-linux-x86_64.egg/sqlalchemy/orm/session.py", line 405, in __exit__ > keystone.common.wsgi raise > keystone.common.wsgi TypeError: exceptions must be old-style classes or derived from BaseException, not NoneType > keystone.common.wsgi Deployed with packstack, no related configuration changes. Installed packages: > openstack-keystone.noarch 2014.1-0.3.b2.el6 > python-keystone.noarch 2014.1-0.3.b2.el6 > python-keystoneclient.noarch 1:0.4.1-3.el6 > python-sqlalchemy0.7.x86_64 0.7.8-1.el6 > python-eventlet.noarch 0.9.17-2.el6 > python-libs.x86_64 2.6.6-51.el6 This is NOT reproducible on Fedora20 with versions: > openstack-keystone.noarch 2014.1-0.3.b2.fc21 > python-keystone.noarch 2014.1-0.3.b2.fc21 > python-keystoneclient.noarch 1:0.4.1-3.fc20 > python-sqlalchemy.x86_64 0.8.4-1.fc20 > python-eventlet.noarch 0.12.0-2.fc20 > python-libs.x86_64 2.7.5-9.fc20 Neither it happened on Fedora19/20 or RHEL-6.5 with devstack/tempest master branches. This seems as related to eventlet/tpool etc issues, here with SQLAlchemy. For example like https://bitbucket.org/eventlet/eventlet/issue/118/exceptions-are-cleared-during
Created attachment 859223 [details] answer file used with packstack
Created attachment 859224 [details] keystone.log APIv2 deprecation warnings filtered out with: > cat keystone.log | grep -v 'Deprecated: v2 API is deprecated as of Icehouse in favor of v3 API and may be removed in K.' > keystone_filtered.log
So i'm by no means an expert in this but looking briefly exc info is cleared by preserves_excinfo defined here: https://bitbucket.org/eventlet/eventlet/src/ba405bb1eb5981c703624967d13f04f0352093c6/eventlet/support/greenlets.py?at=default so if you are using greenlet>=0.3.2 which is what is currently defined by the global requirements.txt then exception handling should be ok. Can you confirm that this is the case? I'm not sure what the fallback modes are. I don't think this is something we'll ever be able to handle in keystone.
> so if you are using greenlet>=0.3.2 which is what is currently defined by > the global requirements.txt then exception handling should be ok. RDO el6 uses greenlet from EPEL6 which has python-greenlet-0.3.1-12.el6
> With RDO-Icehouse (m2 testday packages) on RHEL-6.5, Please try m3 and http://copr-be.cloud.fedoraproject.org/results/jruzicka/rdo-icehouse-epel-6/epel-6-x86_64/python-greenlet-0.4.2-1.el6/python-greenlet-0.4.2-1.el6.x86_64.rpm
Upgrading to python-greenlet-0.4.2-1.el6.x86_64.rpm does not solves the issue, however it is recommended to use a version which is matches to the global-requirements. Upgrading sqlalchemy to 0.8.5 (same as in the current fedora) solves the issue, I also tried 0.7.10 which does not solves the issue. SQLAlchemy 0.8.6 also working. I recommend to upgrade to sqlalchemy to >=0.8.5. Looks like the global requirements wrong about the minimum sqlalchemy version: SQLAlchemy>=0.7.8,<=0.9.99
Look like an evenlet issue, evenlet upgrade also solves the issue. The current global requirement is : eventlet>=0.13.0 The currently packaged: python-eventlet-0.9.17-2.el6.noarch the pip installed version was: eventlet-0.14.0.tar.gz
Simple CLI reproducer for an another similar failure: # keystone tenant-create --name foo +-------------+----------------------------------+ | Property | Value | +-------------+----------------------------------+ | description | | | enabled | True | | id | 260b1573ef924d159a2ddb70b2cccb96 | | name | foo | +-------------+----------------------------------+ # # Wrong response # # keystone tenant-create --name foo exceptions must be old-style classes or derived from BaseException, not NoneType (HTTP 400) # # Workaround # # yum install -y http://copr-be.cloud.fedoraproject.org/results/jruzicka/rdo-icehouse-epel-6/epel-6-x86_64/python-greenlet-0.4.2-1.el6/python-greenlet-0.4.2-1.el6.x86_64.rpm # pip install 'eventlet>=0.13.0' # service openstack-keystone restart # # Expected a 409 status code. # # keystone tenant-create --name foo Conflict occurred attempting to store project. (IntegrityError) (1062, "Duplicate entry 'default-foo' for key 'domain_id'") 'INSERT INTO project (id, name, domain_id, description, enabled, extra) VALUES (%s, %s, %s, %s, %s, %s)' ('f381b10377304900907cd48c0d2a51e9', 'foo', 'default', None, 1, '{}') (HTTP 409) It seems too verbose information for an end user, but the 409 is the good status code, and this operation must not cause error trace in the log files. Both python-eventlet and python-greenlet needs to be updated. python-eventlet >=0.12 includes a fix related to eventlet exception handling.
eventlet and greenlet will be updated in epel6 but pushed as an immediate RDO update while the epel update goes through Bodhi.
https://admin.fedoraproject.org/updates/python-greenlet-0.4.2-1.el6,python-eventlet-0.14.0-1.el6
Note the latest python-olso-messaging has a Requires python-eventlet >= 0.13.0 and so will pull in this dependency for any packages depending on that (including keystone).
eventlet and greenlet where updated in RDO Icehouse repo.