Bug 888975 - Internal Server Error when downloading EC2 credentials
Summary: Internal Server Error when downloading EC2 credentials
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: Red Hat OpenStack
Classification: Red Hat
Component: python-django-horizon
Version: 2.1
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: snapshot2
: 2.1
Assignee: Julie Pichon
QA Contact: Nir Magnezi
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2012-12-19 22:49 UTC by Russell Bryant
Modified: 2015-02-15 22:02 UTC (History)
3 users (show)

Fixed In Version: python-django-horizon-2012.2.1-3.el6ost
Doc Type: Bug Fix
Doc Text:
Clone Of:
Environment:
Last Closed: 2013-02-14 18:24:09 UTC
Target Upstream Version:
Embargoed:

Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Launchpad 1102411 0 None None None Never
OpenStack gerrit 20133 0 None None None Never
OpenStack gerrit 20314 0 None None None Never
Red Hat Product Errata RHBA-2013:0260 0 normal SHIPPED_LIVE Red Hat OpenStack 2.0 (Folsom) Preview bug fix and enhancement update 2013-02-14 23:21:02 UTC

Description Russell Bryant 2012-12-19 22:49:08 UTC 
Description of problem:

When trying to download EC2 credentials from the user settings area, Horizon blew up and gave me an Internal Server Error page.


Version-Release number of selected component (if applicable):

python-django-horizon-2012.2.1-2.el6ost.noarch


Steps to Reproduce:

1. Click "settings" in the top right after logging in to the dashboard

2. Click "EC2 Credentials" on the left.

3. Click the "Download EC2 Credentials" button.

/var/log/httpd/error_log contains:

[Wed Dec 19 22:41:16 2012] [error] [client 10.3.112.5] mod_wsgi (pid=18172): Exception occurred processing WSGI script '/usr/share/openstack-dashboard/openstack_dashboard/wsgi/django.wsgi'.
[Wed Dec 19 22:41:16 2012] [error] [client 10.3.112.5] Traceback (most recent call last):
[Wed Dec 19 22:41:16 2012] [error] [client 10.3.112.5]   File "/usr/lib/python2.6/site-packages/django/core/handlers/wsgi.py", line 241, in __call__
[Wed Dec 19 22:41:16 2012] [error] [client 10.3.112.5]     response = self.get_response(request)
[Wed Dec 19 22:41:16 2012] [error] [client 10.3.112.5]   File "/usr/lib/python2.6/site-packages/django/core/handlers/base.py", line 179, in get_response
[Wed Dec 19 22:41:16 2012] [error] [client 10.3.112.5]     response = self.handle_uncaught_exception(request, resolver, sys.exc_info())
[Wed Dec 19 22:41:16 2012] [error] [client 10.3.112.5]   File "/usr/lib/python2.6/site-packages/django/core/handlers/base.py", line 228, in handle_uncaught_exception
[Wed Dec 19 22:41:16 2012] [error] [client 10.3.112.5]     return callback(request, **param_dict)
[Wed Dec 19 22:41:16 2012] [error] [client 10.3.112.5]   File "/usr/lib/python2.6/site-packages/django/utils/decorators.py", line 91, in _wrapped_view
[Wed Dec 19 22:41:16 2012] [error] [client 10.3.112.5]     response = view_func(request, *args, **kwargs)
[Wed Dec 19 22:41:16 2012] [error] [client 10.3.112.5]   File "/usr/lib/python2.6/site-packages/django/views/defaults.py", line 33, in server_error
[Wed Dec 19 22:41:16 2012] [error] [client 10.3.112.5]     return http.HttpResponseServerError(t.render(Context({})))
[Wed Dec 19 22:41:16 2012] [error] [client 10.3.112.5]   File "/usr/lib/python2.6/site-packages/django/template/base.py", line 140, in render
[Wed Dec 19 22:41:16 2012] [error] [client 10.3.112.5]     return self._render(context)
[Wed Dec 19 22:41:16 2012] [error] [client 10.3.112.5]   File "/usr/lib/python2.6/site-packages/django/template/base.py", line 134, in _render
[Wed Dec 19 22:41:16 2012] [error] [client 10.3.112.5]     return self.nodelist.render(context)
[Wed Dec 19 22:41:16 2012] [error] [client 10.3.112.5]   File "/usr/lib/python2.6/site-packages/django/template/base.py", line 823, in render
[Wed Dec 19 22:41:16 2012] [error] [client 10.3.112.5]     bit = self.render_node(node, context)
[Wed Dec 19 22:41:16 2012] [error] [client 10.3.112.5]   File "/usr/lib/python2.6/site-packages/django/template/base.py", line 837, in render_node
[Wed Dec 19 22:41:16 2012] [error] [client 10.3.112.5]     return node.render(context)
[Wed Dec 19 22:41:16 2012] [error] [client 10.3.112.5]   File "/usr/lib/python2.6/site-packages/django/template/loader_tags.py", line 123, in render
[Wed Dec 19 22:41:16 2012] [error] [client 10.3.112.5]     return compiled_parent._render(context)
[Wed Dec 19 22:41:16 2012] [error] [client 10.3.112.5]   File "/usr/lib/python2.6/site-packages/django/template/base.py", line 134, in _render
[Wed Dec 19 22:41:16 2012] [error] [client 10.3.112.5]     return self.nodelist.render(context)
[Wed Dec 19 22:41:16 2012] [error] [client 10.3.112.5]   File "/usr/lib/python2.6/site-packages/django/template/base.py", line 823, in render
[Wed Dec 19 22:41:16 2012] [error] [client 10.3.112.5]     bit = self.render_node(node, context)
[Wed Dec 19 22:41:16 2012] [error] [client 10.3.112.5]   File "/usr/lib/python2.6/site-packages/django/template/base.py", line 837, in render_node
[Wed Dec 19 22:41:16 2012] [error] [client 10.3.112.5]     return node.render(context)
[Wed Dec 19 22:41:16 2012] [error] [client 10.3.112.5]   File "/usr/lib/python2.6/site-packages/django/template/loader_tags.py", line 155, in render
[Wed Dec 19 22:41:16 2012] [error] [client 10.3.112.5]     return self.render_template(self.template, context)
[Wed Dec 19 22:41:16 2012] [error] [client 10.3.112.5]   File "/usr/lib/python2.6/site-packages/django/template/loader_tags.py", line 137, in render_template
[Wed Dec 19 22:41:16 2012] [error] [client 10.3.112.5]     output = template.render(context)
[Wed Dec 19 22:41:16 2012] [error] [client 10.3.112.5]   File "/usr/lib/python2.6/site-packages/django/template/base.py", line 140, in render
[Wed Dec 19 22:41:16 2012] [error] [client 10.3.112.5]     return self._render(context)
[Wed Dec 19 22:41:16 2012] [error] [client 10.3.112.5]   File "/usr/lib/python2.6/site-packages/django/template/base.py", line 134, in _render
[Wed Dec 19 22:41:16 2012] [error] [client 10.3.112.5]     return self.nodelist.render(context)
[Wed Dec 19 22:41:16 2012] [error] [client 10.3.112.5]   File "/usr/lib/python2.6/site-packages/django/template/base.py", line 823, in render
[Wed Dec 19 22:41:16 2012] [error] [client 10.3.112.5]     bit = self.render_node(node, context)
[Wed Dec 19 22:41:16 2012] [error] [client 10.3.112.5]   File "/usr/lib/python2.6/site-packages/django/template/base.py", line 837, in render_node
[Wed Dec 19 22:41:16 2012] [error] [client 10.3.112.5]     return node.render(context)
[Wed Dec 19 22:41:16 2012] [error] [client 10.3.112.5]   File "/usr/lib/python2.6/site-packages/compressor/templatetags/compress.py", line 147, in render
[Wed Dec 19 22:41:16 2012] [error] [client 10.3.112.5]     return self.render_compressed(context, self.kind, self.mode, forced=forced)
[Wed Dec 19 22:41:16 2012] [error] [client 10.3.112.5]   File "/usr/lib/python2.6/site-packages/compressor/templatetags/compress.py", line 88, in render_compressed
[Wed Dec 19 22:41:16 2012] [error] [client 10.3.112.5]     cached_offline = self.render_offline(context, forced=forced)
[Wed Dec 19 22:41:16 2012] [error] [client 10.3.112.5]   File "/usr/lib/python2.6/site-packages/compressor/templatetags/compress.py", line 72, in render_offline
[Wed Dec 19 22:41:16 2012] [error] [client 10.3.112.5]     'You may need to run "python manage.py compress".' % key)
[Wed Dec 19 22:41:16 2012] [error] [client 10.3.112.5] OfflineGenerationError: You have offline compression enabled but key "1056718f92f8d4204721bac759b3871a" is missing from offline manifest. You may need to run "python manage.py compress".
Comment 2 Matthias Runge 2012-12-20 09:32:02 UTC 
I can't reproduce this. Please provide more info.

did you install with keystone sample data?
As login user you used? Has this user the admin role?
Comment 3 Matthias Runge 2012-12-20 09:33:11 UTC 
We had this bug reported as bug https://bugzilla.redhat.com/show_bug.cgi?id=801690

That turned out to be an error in keystone sample data, fixed about a month ago.
Comment 4 Julie Pichon 2012-12-21 15:29:26 UTC 
I was able to reproduce the problem locally, this might be configuration related.
Comment 5 Julie Pichon 2012-12-28 17:31:38 UTC 
From the Horizon logs:

[Fri Dec 28 17:17:51 2012] [error] \x1b[31;1mRecoverable error: The server has either erred or is incapable of performing the requested operation. (HTTP 500) (Request-ID: req-71d37c9e-4e9a-4650-82d4-43312c381d76)\x1b[0m
[Fri Dec 28 17:17:51 2012] [error] Traceback (most recent call last):
[Fri Dec 28 17:17:51 2012] [error]   File "/usr/lib/python2.6/site-packages/horizon/dashboards/settings/ec2/forms.py", line 70, in handle
[Fri Dec 28 17:17:51 2012] [error]     credentials = api.nova.get_x509_credentials(request)
[Fri Dec 28 17:17:51 2012] [error]   File "/usr/lib/python2.6/site-packages/horizon/api/nova.py", line 581, in get_x509_credentials
[Fri Dec 28 17:17:51 2012] [error]     return novaclient(request).certs.create()
[Fri Dec 28 17:17:51 2012] [error]   File "/usr/lib/python2.6/site-packages/novaclient/v1_1/certs.py", line 42, in create
[Fri Dec 28 17:17:51 2012] [error]     return self._create('/os-certificates', {}, 'certificate')
[Fri Dec 28 17:17:51 2012] [error]   File "/usr/lib/python2.6/site-packages/novaclient/base.py", line 148, in _create
[Fri Dec 28 17:17:51 2012] [error]     _resp, body = self.api.client.post(url, body=body)
[Fri Dec 28 17:17:51 2012] [error]   File "/usr/lib/python2.6/site-packages/novaclient/client.py", line 244, in post
[Fri Dec 28 17:17:51 2012] [error]     return self._cs_request(url, 'POST', **kwargs)
[Fri Dec 28 17:17:51 2012] [error]   File "/usr/lib/python2.6/site-packages/novaclient/client.py", line 228, in _cs_request
[Fri Dec 28 17:17:51 2012] [error]     **kwargs)
[Fri Dec 28 17:17:51 2012] [error]   File "/usr/lib/python2.6/site-packages/novaclient/client.py", line 210, in _time_request
[Fri Dec 28 17:17:51 2012] [error]     resp, body = self.request(url, method, **kwargs)
[Fri Dec 28 17:17:51 2012] [error]   File "/usr/lib/python2.6/site-packages/novaclient/client.py", line 204, in request
[Fri Dec 28 17:17:51 2012] [error]     raise exceptions.from_response(resp, body)
[Fri Dec 28 17:17:51 2012] [error] ClientException: The server has either erred or is incapable of performing the requested operation. (HTTP 500) (Request-ID: req-71d37c9e-4e9a-4650-82d4-43312c381d76)


From the Nova logs:

2012-12-28 17:19:15 ERROR nova.api.openstack [req-a56e514b-cf83-4dc6-89ff-e855bd1265c9 987af50f7bf649d9918a7cf636f7df11 aedfe7e8a40d41c6a559855af0e3ed8d] Caught error: Unexpected error while running command.
Command: openssl ca -batch -out /tmp/tmpSx6IT3/outbound.csr -config ./openssl.cnf -infiles /tmp/tmpSx6IT3/inbound.csr
Exit code: 1
Stdout: ''
Stderr: 'Using configuration from ./openssl.cnf\nunable to load number from ./serial\nerror while loading serial number\n140081502267208:error:0D066096:asn1 encoding routines:a2i_ASN1_INTEGER:short line:f_int.c:215:\n'
Traceback (most recent call last):

  File "/usr/lib/python2.6/site-packages/nova/openstack/common/rpc/amqp.py", line 276, in _process_data
    rval = self.proxy.dispatch(ctxt, version, method, **args)

  File "/usr/lib/python2.6/site-packages/nova/openstack/common/rpc/dispatcher.py", line 145, in dispatch
    return getattr(proxyobj, method)(ctxt, **kwargs)

  File "/usr/lib/python2.6/site-packages/nova/cert/manager.py", line 58, in generate_x509_cert
    return crypto.generate_x509_cert(user_id, project_id)

  File "/usr/lib/python2.6/site-packages/nova/crypto.py", line 242, in generate_x509_cert
    (serial, signed_csr) = sign_csr(csr, project_id)

  File "/usr/lib/python2.6/site-packages/nova/crypto.py", line 286, in sign_csr
    return _sign_csr(csr_text, ca_folder())

  File "/usr/lib/python2.6/site-packages/nova/crypto.py", line 307, in _sign_csr
    './openssl.cnf', '-infiles', inbound)

  File "/usr/lib/python2.6/site-packages/nova/utils.py", line 206, in execute
    cmd=' '.join(cmd))

ProcessExecutionError: Unexpected error while running command.
Command: openssl ca -batch -out /tmp/tmpSx6IT3/outbound.csr -config ./openssl.cnf -infiles /tmp/tmpSx6IT3/inbound.csr
Exit code: 1
Stdout: ''
Stderr: 'Using configuration from ./openssl.cnf\nunable to load number from ./serial\nerror while loading serial number\n140081502267208:error:0D066096:asn1 encoding routines:a2i_ASN1_INTEGER:short line:f_int.c:215:\n'

2012-12-28 17:19:15 2780 TRACE nova.api.openstack Traceback (most recent call last):
2012-12-28 17:19:15 2780 TRACE nova.api.openstack   File "/usr/lib/python2.6/site-packages/nova/api/openstack/__init__.py", line 78, in __call__
2012-12-28 17:19:15 2780 TRACE nova.api.openstack     return req.get_response(self.application)
2012-12-28 17:19:15 2780 TRACE nova.api.openstack   File "/usr/lib/python2.6/site-packages/WebOb-1.0.8-py2.6.egg/webob/request.py", line 1053, in get_response
2012-12-28 17:19:15 2780 TRACE nova.api.openstack     application, catch_exc_info=False)
2012-12-28 17:19:15 2780 TRACE nova.api.openstack   File "/usr/lib/python2.6/site-packages/WebOb-1.0.8-py2.6.egg/webob/request.py", line 1022, in call_application
2012-12-28 17:19:15 2780 TRACE nova.api.openstack     app_iter = application(self.environ, start_response)
2012-12-28 17:19:15 2780 TRACE nova.api.openstack   File "/usr/lib/python2.6/site-packages/WebOb-1.0.8-py2.6.egg/webob/dec.py", line 159, in __call__
2012-12-28 17:19:15 2780 TRACE nova.api.openstack     return resp(environ, start_response)
2012-12-28 17:19:15 2780 TRACE nova.api.openstack   File "/usr/lib/python2.6/site-packages/keystone/middleware/auth_token.py", line 278, in __call__
2012-12-28 17:19:15 2780 TRACE nova.api.openstack     return self.app(env, start_response)
2012-12-28 17:19:15 2780 TRACE nova.api.openstack   File "/usr/lib/python2.6/site-packages/WebOb-1.0.8-py2.6.egg/webob/dec.py", line 159, in __call__
2012-12-28 17:19:15 2780 TRACE nova.api.openstack     return resp(environ, start_response)
2012-12-28 17:19:15 2780 TRACE nova.api.openstack   File "/usr/lib/python2.6/site-packages/WebOb-1.0.8-py2.6.egg/webob/dec.py", line 159, in __call__
2012-12-28 17:19:15 2780 TRACE nova.api.openstack     return resp(environ, start_response)
2012-12-28 17:19:15 2780 TRACE nova.api.openstack   File "/usr/lib/python2.6/site-packages/WebOb-1.0.8-py2.6.egg/webob/dec.py", line 159, in __call__
2012-12-28 17:19:15 2780 TRACE nova.api.openstack     return resp(environ, start_response)
2012-12-28 17:19:15 2780 TRACE nova.api.openstack   File "/usr/lib/python2.6/site-packages/Routes-1.12.3-py2.6.egg/routes/middleware.py", line 131, in __call__
2012-12-28 17:19:15 2780 TRACE nova.api.openstack     response = self.app(environ, start_response)
2012-12-28 17:19:15 2780 TRACE nova.api.openstack   File "/usr/lib/python2.6/site-packages/WebOb-1.0.8-py2.6.egg/webob/dec.py", line 159, in __call__
2012-12-28 17:19:15 2780 TRACE nova.api.openstack     return resp(environ, start_response)
2012-12-28 17:19:15 2780 TRACE nova.api.openstack   File "/usr/lib/python2.6/site-packages/WebOb-1.0.8-py2.6.egg/webob/dec.py", line 147, in __call__
2012-12-28 17:19:15 2780 TRACE nova.api.openstack     resp = self.call_func(req, *args, **self.kwargs)
2012-12-28 17:19:15 2780 TRACE nova.api.openstack   File "/usr/lib/python2.6/site-packages/WebOb-1.0.8-py2.6.egg/webob/dec.py", line 208, in call_func
2012-12-28 17:19:15 2780 TRACE nova.api.openstack     return self.func(req, *args, **kwargs)
2012-12-28 17:19:15 2780 TRACE nova.api.openstack   File "/usr/lib/python2.6/site-packages/nova/api/openstack/wsgi.py", line 894, in __call__
2012-12-28 17:19:15 2780 TRACE nova.api.openstack     content_type, body, accept)
2012-12-28 17:19:15 2780 TRACE nova.api.openstack   File "/usr/lib/python2.6/site-packages/nova/api/openstack/wsgi.py", line 942, in _process_stack
2012-12-28 17:19:15 2780 TRACE nova.api.openstack     action_result = self.dispatch(meth, request, action_args)
2012-12-28 17:19:15 2780 TRACE nova.api.openstack   File "/usr/lib/python2.6/site-packages/nova/api/openstack/wsgi.py", line 1030, in dispatch
2012-12-28 17:19:15 2780 TRACE nova.api.openstack     return method(req=request, **action_args)
2012-12-28 17:19:15 2780 TRACE nova.api.openstack   File "/usr/lib/python2.6/site-packages/nova/api/openstack/compute/contrib/certificates.py", line 88, in create
2012-12-28 17:19:15 2780 TRACE nova.api.openstack     user_id=context.user_id, project_id=context.project_id)
2012-12-28 17:19:15 2780 TRACE nova.api.openstack   File "/usr/lib/python2.6/site-packages/nova/cert/rpcapi.py", line 67, in generate_x509_cert
2012-12-28 17:19:15 2780 TRACE nova.api.openstack     project_id=project_id))
2012-12-28 17:19:15 2780 TRACE nova.api.openstack   File "/usr/lib/python2.6/site-packages/nova/openstack/common/rpc/proxy.py", line 80, in call
2012-12-28 17:19:15 2780 TRACE nova.api.openstack     return rpc.call(context, self._get_topic(topic), msg, timeout)
2012-12-28 17:19:15 2780 TRACE nova.api.openstack   File "/usr/lib/python2.6/site-packages/nova/openstack/common/rpc/__init__.py", line 108, in call
2012-12-28 17:19:15 2780 TRACE nova.api.openstack     return _get_impl().call(cfg.CONF, context, topic, msg, timeout)
2012-12-28 17:19:15 2780 TRACE nova.api.openstack   File "/usr/lib/python2.6/site-packages/nova/openstack/common/rpc/impl_qpid.py", line 543, in call
2012-12-28 17:19:15 2780 TRACE nova.api.openstack     rpc_amqp.get_connection_pool(conf, Connection))
2012-12-28 17:19:15 2780 TRACE nova.api.openstack   File "/usr/lib/python2.6/site-packages/nova/openstack/common/rpc/amqp.py", line 369, in call
2012-12-28 17:19:15 2780 TRACE nova.api.openstack     rv = list(rv)
2012-12-28 17:19:15 2780 TRACE nova.api.openstack   File "/usr/lib/python2.6/site-packages/nova/openstack/common/rpc/amqp.py", line 337, in __iter__
2012-12-28 17:19:15 2780 TRACE nova.api.openstack     raise result
2012-12-28 17:19:15 2780 TRACE nova.api.openstack ProcessExecutionError_Remote: Unexpected error while running command.
2012-12-28 17:19:15 2780 TRACE nova.api.openstack Command: openssl ca -batch -out /tmp/tmpSx6IT3/outbound.csr -config ./openssl.cnf -infiles /tmp/tmpSx6IT3/inbound.csr
2012-12-28 17:19:15 2780 TRACE nova.api.openstack Exit code: 1
2012-12-28 17:19:15 2780 TRACE nova.api.openstack Stdout: ''
2012-12-28 17:19:15 2780 TRACE nova.api.openstack Stderr: 'Using configuration from ./openssl.cnf\nunable to load number from ./serial\nerror while loading serial number\n140081502267208:error:0D066096:asn1 encoding routines:a2i_ASN1_INTEGER:short line:f_int.c:215:\n'
Comment 6 Matthias Runge 2013-01-17 07:57:09 UTC 
So, from the logs, it's a nova problem?
Comment 7 Julie Pichon 2013-01-21 14:20:49 UTC 
I was mislead by the OpenSSL error.

The real problem is that there is no EC2 service and endpoint defined (which happens when following the Getting Started docs) - "Invalid service catalog service: ec2".

I think the EC2 Credential link on the left panel shouldn't be visible when no EC2 endpoint is defined.
Comment 12 Nir Magnezi 2013-01-31 13:10:20 UTC 
Verified With: python-django-horizon-2012.2.1-3.el6ost.noarch
Followed the following steps and managed to download EC2 Credentials

1. Click "Settings" in the top right after logging in to the dashboard.
2. Click "EC2 Credentials" on the left.
3. Click the "Download EC2 Credentials" button.
Comment 13 Nir Magnezi 2013-01-31 13:30:13 UTC 
Adding some more steps to complete the verification:

4. Removed the EC2 endpoint from keystone.
5. Logged out from Horizon.
6. Logged in to Horizon.
7. Clicked Settings
8. Verified that EC2 Credentials URL is no longer present.
Comment 15 errata-xmlrpc 2013-02-14 18:24:09 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

http://rhn.redhat.com/errata/RHBA-2013-0260.html
Note You need to log in before you can comment on or make changes to this bug.