Bug 1061329 - Keystone returns HTTP 400 as SQLAlchemy raises None exceptions
Summary: Keystone returns HTTP 400 as SQLAlchemy raises None exceptions
Alias: None
Product: RDO
Classification: Community
Component: distribution
Version: unspecified
Hardware: Unspecified
OS: Unspecified
Target Milestone: GA
: Icehouse
Assignee: Alan Pevec
QA Contact: Ami Jeain
Depends On:
Blocks: 1061137
TreeView+ depends on / blocked
Reported: 2014-02-04 15:58 UTC by Pavel Sedlák
Modified: 2016-04-27 05:45 UTC (History)
10 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Clone Of:
Last Closed: 2014-05-07 15:42:06 UTC

Attachments (Terms of Use)
answer file used with packstack (14.55 KB, text/plain)
2014-02-04 15:58 UTC, Pavel Sedlák
no flags Details
keystone.log (45.48 KB, text/x-log)
2014-02-04 16:02 UTC, Pavel Sedlák
no flags Details

System ID Priority Status Summary Last Updated
Launchpad 1291047 None None None Never

Description Pavel Sedlák 2014-02-04 15:58:06 UTC
With RDO-Icehouse (m2 testday packages) on RHEL-6.5,
negative Tempest identity tests fails as Keystone responds with HTTP 400.

For example test tempest.api.identity.admin.v3.test_projects.ProjectsTestJSON.test_project_create_duplicate gives following output:

> Request: POST
> Request Headers: {'Content-Type': 'application/json', 'Accept': 'application/json', 'X-Auth-Token': '<Token omitted>'}
> Request Body: {"project": {"enabled": true, "description": null, "name": "project-dup--1737299968", "domain_id": "default"}}
> Response Status: 400
> Response Headers: {'content-length': '143', 'date': 'Tue, 04 Feb 2014 07:47:53 GMT', 'content-type': 'application/json', 'vary': 'X-Auth-Token', 'connection': 'close'}
> Response Body: {"error": {"message": "exceptions must be old-style classes or derived from BaseException, not NoneType", "code": 400, "title": "Bad Request"}}

In keystone.log the exception can be seen as
> keystone.common.wsgi Traceback (most recent call last):
> keystone.common.wsgi   File "/usr/lib/python2.6/site-packages/keystone/common/wsgi.py", line 214, in __call__
> keystone.common.wsgi     result = method(context, **params)
> keystone.common.wsgi   File "/usr/lib/python2.6/site-packages/keystone/common/controller.py", line 174, in inner
> keystone.common.wsgi     return f(self, context, *args, **kwargs)
> keystone.common.wsgi   File "/usr/lib/python2.6/site-packages/keystone/assignment/controllers.py", line 390, in create_project
> keystone.common.wsgi     ref = self.assignment_api.create_project(ref['id'], ref)
> keystone.common.wsgi   File "/usr/lib/python2.6/site-packages/keystone/notifications.py", line 53, in wrapper
> keystone.common.wsgi     result = f(*args, **kwargs)
> keystone.common.wsgi   File "/usr/lib/python2.6/site-packages/keystone/assignment/core.py", line 72, in create_project
> keystone.common.wsgi     ret = self.driver.create_project(tenant_id, tenant)
> keystone.common.wsgi   File "/usr/lib/python2.6/site-packages/keystone/common/sql/core.py", line 165, in wrapper
> keystone.common.wsgi     return method(*args, **kwargs)
> keystone.common.wsgi   File "/usr/lib/python2.6/site-packages/keystone/assignment/backends/sql.py", line 411, in create_project
> keystone.common.wsgi     return tenant_ref.to_dict()
> keystone.common.wsgi   File "/usr/lib64/python2.6/contextlib.py", line 23, in __exit__
> keystone.common.wsgi     self.gen.next()
> keystone.common.wsgi   File "/usr/lib/python2.6/site-packages/keystone/common/sql/core.py", line 156, in transaction
> keystone.common.wsgi     yield session
> keystone.common.wsgi   File "/usr/lib64/python2.6/site-packages/SQLAlchemy-0.7.8-py2.6-linux-x86_64.egg/sqlalchemy/orm/session.py", line 405, in __exit__
> keystone.common.wsgi     raise
> keystone.common.wsgi TypeError: exceptions must be old-style classes or derived from BaseException, not NoneType
> keystone.common.wsgi 

Deployed with packstack, no related configuration changes.

Installed packages:
> openstack-keystone.noarch          2014.1-0.3.b2.el6
> python-keystone.noarch             2014.1-0.3.b2.el6
> python-keystoneclient.noarch       1:0.4.1-3.el6
> python-sqlalchemy0.7.x86_64        0.7.8-1.el6
> python-eventlet.noarch             0.9.17-2.el6
> python-libs.x86_64                 2.6.6-51.el6

This is NOT reproducible on Fedora20 with versions:
> openstack-keystone.noarch       2014.1-0.3.b2.fc21
> python-keystone.noarch          2014.1-0.3.b2.fc21
> python-keystoneclient.noarch    1:0.4.1-3.fc20
> python-sqlalchemy.x86_64        0.8.4-1.fc20
> python-eventlet.noarch          0.12.0-2.fc20
> python-libs.x86_64              2.7.5-9.fc20

Neither it happened on Fedora19/20 or RHEL-6.5 with devstack/tempest master branches. 

This seems as related to eventlet/tpool etc issues, here with SQLAlchemy.
For example like https://bitbucket.org/eventlet/eventlet/issue/118/exceptions-are-cleared-during

Comment 1 Pavel Sedlák 2014-02-04 15:58:54 UTC
Created attachment 859223 [details]
answer file used with packstack

Comment 2 Pavel Sedlák 2014-02-04 16:02:04 UTC
Created attachment 859224 [details]

APIv2 deprecation warnings filtered out with:
> cat keystone.log | grep -v 'Deprecated: v2 API is deprecated as of Icehouse in favor of v3 API and may be removed in K.' > keystone_filtered.log

Comment 3 Jamie Lennox 2014-03-11 21:37:34 UTC
So i'm by no means an expert in this but looking briefly exc info is cleared by preserves_excinfo defined here:


so if you are using greenlet>=0.3.2 which is what is currently defined by the global requirements.txt then exception handling should be ok.

Can you confirm that this is the case? I'm not sure what the fallback modes are. I don't think this is something we'll ever be able to handle in keystone.

Comment 4 Alan Pevec 2014-03-28 18:24:03 UTC
> so if you are using greenlet>=0.3.2 which is what is currently defined by
> the global requirements.txt then exception handling should be ok.

RDO el6 uses greenlet from EPEL6 which has python-greenlet-0.3.1-12.el6

Comment 5 Alan Pevec 2014-03-28 22:33:04 UTC
> With RDO-Icehouse (m2 testday packages) on RHEL-6.5,

Please try m3 and http://copr-be.cloud.fedoraproject.org/results/jruzicka/rdo-icehouse-epel-6/epel-6-x86_64/python-greenlet-0.4.2-1.el6/python-greenlet-0.4.2-1.el6.x86_64.rpm

Comment 6 Attila Fazekas 2014-04-19 12:23:46 UTC
Upgrading to python-greenlet-0.4.2-1.el6.x86_64.rpm does not solves the issue, however it is recommended to use a version which is matches to the global-requirements.

Upgrading sqlalchemy to 0.8.5 (same as in the current fedora) solves the issue, I also tried 0.7.10 which does not solves the issue.
SQLAlchemy 0.8.6 also working.

I recommend to upgrade to sqlalchemy to >=0.8.5.

Looks like the global requirements wrong about the minimum sqlalchemy version: 

Comment 7 Attila Fazekas 2014-04-22 07:11:22 UTC
Look like an evenlet issue, evenlet upgrade also solves the issue.
The current global requirement is : eventlet>=0.13.0
The currently packaged: python-eventlet-0.9.17-2.el6.noarch
the pip installed version was: eventlet-0.14.0.tar.gz

Comment 8 Attila Fazekas 2014-04-22 08:24:53 UTC
Simple CLI reproducer for an another similar failure:
# keystone tenant-create --name foo
|   Property  |              Value               |
| description |                                  |
|   enabled   |               True               |
|      id     | 260b1573ef924d159a2ddb70b2cccb96 |
|     name    |               foo                |

# # Wrong response #

# keystone tenant-create --name foo
exceptions must be old-style classes or derived from BaseException, not NoneType (HTTP 400)

# # Workaround #
# yum install -y http://copr-be.cloud.fedoraproject.org/results/jruzicka/rdo-icehouse-epel-6/epel-6-x86_64/python-greenlet-0.4.2-1.el6/python-greenlet-0.4.2-1.el6.x86_64.rpm
# pip install 'eventlet>=0.13.0'
# service openstack-keystone restart

# # Expected a 409 status code. #
# keystone tenant-create --name foo
Conflict occurred attempting to store project. (IntegrityError) (1062, "Duplicate entry 'default-foo' for key 'domain_id'") 'INSERT INTO project (id, name, domain_id, description, enabled, extra) VALUES (%s, %s, %s, %s, %s, %s)' ('f381b10377304900907cd48c0d2a51e9', 'foo', 'default', None, 1, '{}') (HTTP 409)

It seems too verbose information for an end user, but the 409 is the good status code, and this operation must not cause error trace in the log files.

Both python-eventlet and python-greenlet needs to be updated.

python-eventlet >=0.12 includes a fix related to eventlet exception handling.

Comment 9 Alan Pevec 2014-04-22 18:11:07 UTC
eventlet and greenlet will be updated in epel6 but pushed as an immediate RDO update while the epel update goes through Bodhi.

Comment 12 Pádraig Brady 2014-04-24 19:17:28 UTC
Note the latest python-olso-messaging has a Requires python-eventlet >= 0.13.0
and so will pull in this dependency for any packages depending on that (including keystone).

Comment 13 Alan Pevec 2014-05-07 15:42:06 UTC
eventlet and greenlet where updated in RDO Icehouse repo.

Note You need to log in before you can comment on or make changes to this bug.