Bug 1035032

Requesting a v3 token without a catalog when pki is in use works. im seeing the wsgi keystone logs show a 201 and there is no backtrace. Apache however reports "Bad header=xxxxxxxxxxx" in the error log and the string is part of the pki string. It looks like we are getting something truncated and this is causing the 500 error in httpd.

Missing steps to reproduce:

after installing RPMS:

openstack-db --service keystone --init 


cat > /root/keystonerc_admin << _EOF
export ADMIN_TOKEN=EXAMPLEADMINTOKEN
export OS_USERNAME=admin
export OS_PASSWORD=verybadpass
export OS_TENANT_NAME=admin
export OS_AUTH_URL=http://127.0.0.1:5000/v2.0/
export SERVICE_ENDPOINT=http://127.0.0.1:35357/v2.0/
export SERVICE_TOKEN=EXAMPLEADMINTOKEN
_EOF

after sourcing keystonerc_admin

openstack-config --set /etc/keystone/keystone.conf DEFAULT admin_token $ADMIN_TOKEN


chkconfig  openstack-keystone on
service openstack-keystone start

In order to get a token, you also need to run:

keystone-manage pki_setup --keystone-user=keystone --keystone-group=keystone


If you don't you get a 500 error, which sounds suspiciously like the original problem.

Note that when I completed the above steps:

]# curl -H "Content-type: application/json" -d '{"auth": {"identity": {"methods": ["password"], "password": {"user": {"domain": {"name": "Default"}, "name": "tester","password": "tester"}}}, "scope": {"project": {"domain": {"name": "Default"},"name": "tester"}}}}' http://127.0.0.1:5000/v3/auth/tokens

returned the following


{"token": {"methods": ["password"], "roles": [{"id": "26091480c0f14c678c55406ac020ba60", "name": "tester"}], "expires_at": "2013-12-05T22:14:26.024862Z", "project": {"domain": {"id": "default", "name": "Default"}, "id": "ba3867ee1c6b4382aea83799f034c9c2", "name": "tester"}, "catalog": [], "extras": {}, "user": {"domain": {"id": "default", "name": "Default"}, "id": "a2a9cab7966f4da79755b07b1e0a583e", "name": "tester"}, "issued_at": "2013-12-04T22:14:26.024919Z"}}[root@ayoung-rhos4-keystone ~]# 

I suspect that there is something in your setup that is not explcitly listed above.

Problem seems to be that the Header size is limited to 8190 bytes by default, and the Service catalog is making the token header swell to over 9000.  One possible solution is to increase the header size with the configuration option shown in https://httpd.apache.org/docs/2.2/mod/core.html  LimitRequestFieldSize

Looks like there is no configuration for it:


I have run across a response header size limitation that I don’t know how to get around. I am using mod_wsgi with Apache2, SSL, and OpenStack Keystone. The problem I am running into is that the tokens returned by Keystone can be > 8190 bytes in length. When they are greater than 8190 I get the following error:

 

 

[Thu Jan 16 22:27:47 2014] [info] Initial (No.1) HTTPS request received for child 231 (server d00-50-56-8e-75-82.cloudos.org:5000)

[Thu Jan 16 22:27:47 2014] [info] [client 192.168.124.2] mod_wsgi (pid=24676, process='keystone', application='d00-50-56-8e-75-82.cloudos.org:5000|'): Loading WSGI script '/etc/apache2/wsgi/keystone/main'.

[Thu Jan 16 22:27:48 2014] [error] [client 192.168.124.2] malformed header from script. Bad header=mVmOTdhMmUzIn0sIHsidXJsIjogImh: main

[Thu Jan 16 22:27:48 2014] [debug] mod_deflate.c(615): [client 192.168.124.2] Zlib: Compressed 592 to 377 : URL /v3/auth/tokens

[Thu Jan 16 22:27:48 2014] [debug] ssl_engine_kernel.c(1884): OpenSSL: Write: SSL negotiation finished successfully

[Thu Jan 16 22:27:48 2014] [info] [client 192.168.124.2] Connection closed to child 231 with standard shutdown (server d00-50-56-8e-75-82.cloudos.org:5000)

 

Is there some way to increase the response header size limit?

 

This is a hardwired limitation within the Apache function used by mod_wsgi to parse the response headers returned from a WSGI application running in daemon mode.

 

        if (!(l = strchr(w, ':'))) {

            if (!buffer) {

                /* Soak up all the script output - may save an outright kill */

                while ((*getsfunc)(w, MAX_STRING_LEN - 1, getsfunc_data) > 0) {

                    continue;

                }

            }

 

            ap_log_rerror(SCRIPT_LOG_MARK, APLOG_ERR|APLOG_TOCLIENT, 0, r,

                          "malformed header from script '%s': Bad header: %.30s",

                          apr_filepath_name_get(r->filename), w);

            return HTTP_INTERNAL_SERVER_ERROR;

        }

 

Besides copying the function from Apache into mod_wsgi and modifying it, which has been on the cards for a while for other reasons, the only thing I can suggest is to used embedded mode, which in general I wouldn't recommend.

By reducing superfluous elements of the service catalog, we are able to shrink the Token size.  This fix is slated for backport to Icehouse RC2 and will likely be backported to Havana Stable.


Using https://review.openstack.org/#/c/85147/ I was able to create a token in HTTPD on a defaul devstack deployment.  This was failing without the patch due to the token size.

Adam, even though this bug is not ON_QA, I can see that your patch in the latest Icehouse puddle so I assume that it should be fixed. However, I am still running into the issue ...

When keystone is running in Apache, I can only get a token if I use uuid instead of pki, or if I use "?nocatalog" in the request, or if I get a v2.0 token. Any ideas?

Yes, Nathan's comments are correct.

(In reply to Nathan Kinder from comment #12)
> This is only partially fixed, as tokens can still trigger a 500 error if
> they have a large service catalog.  There was work done to shrink the token
> for OSP 5.0, but compressed tokens are really needed here, which is an OSP
> 6.0 item.  Moving this bug.

Does this mean that in order to run Keystone with HTTPD, pkiz has to be used?

(In reply to Arthur Berezin from comment #14)
> (In reply to Nathan Kinder from comment #12)
> > This is only partially fixed, as tokens can still trigger a 500 error if
> > they have a large service catalog.  There was work done to shrink the token
> > for OSP 5.0, but compressed tokens are really needed here, which is an OSP
> > 6.0 item.  Moving this bug.
> 
> Does this mean that in order to run Keystone with HTTPD, pkiz has to be used?

UUID tokens can be used as well (which is what most operators use according to an upstream poll).  UUID is also the default now.  Basically, we should document that the PKI token format should not be used when deploying in httpd.

This issue is not fixable with PKI tokens.  If httpd is being used, a smaller token format must be configured (such as UUID or PKIZ).

I'm closing this bug as WONTFIX.