Bug 1287621 - Horizon - Launch an instance on admin project return Internal Server Error
Horizon - Launch an instance on admin project return Internal Server Error
Product: Red Hat OpenStack
Classification: Red Hat
Component: openstack-cinder (Show other bugs)
7.0 (Kilo)
x86_64 Linux
high Severity urgent
: z3
: 7.0 (Kilo)
Assigned To: Eric Harney
: Regression, ZStream
: 1287664 1288015 (view as bug list)
Depends On:
Blocks: 1288652 1288654
  Show dependency treegraph
Reported: 2015-12-02 08:09 EST by Ido Ovadia
Modified: 2016-04-26 10:24 EDT (History)
10 users (show)

See Also:
Fixed In Version: openstack-cinder-2015.1.2-5.el7ost
Doc Type: Bug Fix
Doc Text:
Story Points: ---
Clone Of:
: 1288652 (view as bug list)
Last Closed: 2015-12-21 12:04:01 EST
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---

Attachments (Terms of Use)

  None (edit)
Description Ido Ovadia 2015-12-02 08:09:16 EST
Description of problem:
Launch an instance (on horizon) on admin project return Internal Server Error

2015-12-02 12:20:52,972 3930 ERROR django.request Internal Server Error: /dashboard/project/instances/launch
Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/django/core/handlers/base.py", line 132, in get_response
    response = wrapped_callback(request, *callback_args, **callback_kwargs)
  File "/usr/lib/python2.7/site-packages/horizon/decorators.py", line 36, in dec
    return view_func(request, *args, **kwargs)
  File "/usr/lib/python2.7/site-packages/horizon/decorators.py", line 52, in dec
    return view_func(request, *args, **kwargs)
  File "/usr/lib/python2.7/site-packages/horizon/decorators.py", line 36, in dec
    return view_func(request, *args, **kwargs)
  File "/usr/lib/python2.7/site-packages/horizon/decorators.py", line 84, in dec
    return view_func(request, *args, **kwargs)
  File "/usr/lib/python2.7/site-packages/django/views/generic/base.py", line 71, in view
    return self.dispatch(request, *args, **kwargs)
  File "/usr/lib/python2.7/site-packages/django/views/generic/base.py", line 89, in dispatch
    return handler(request, *args, **kwargs)
  File "/usr/lib/python2.7/site-packages/horizon/workflows/views.py", line 167, in post
    context = self.get_context_data(**kwargs)
  File "/usr/lib/python2.7/site-packages/horizon/workflows/views.py", line 91, in get_context_data
    workflow = self.get_workflow()
  File "/usr/lib/python2.7/site-packages/horizon/workflows/views.py", line 81, in get_workflow
  File "/usr/lib/python2.7/site-packages/horizon/workflows/base.py", line 648, in __init__
    valid = step.action.is_valid()
  File "/usr/lib/python2.7/site-packages/django/forms/forms.py", line 184, in is_valid
    return self.is_bound and not self.errors
  File "/usr/lib/python2.7/site-packages/django/forms/forms.py", line 176, in errors
  File "/usr/lib/python2.7/site-packages/django/forms/forms.py", line 393, in full_clean
  File "/usr/lib/python2.7/site-packages/django/forms/forms.py", line 417, in _clean_form
    cleaned_data = self.clean()
  File "/usr/share/openstack-dashboard/openstack_dashboard/wsgi/../../openstack_dashboard/dashboards/project/instances/workflows/create_instance.py", line 357, in clean
  File "/usr/share/openstack-dashboard/openstack_dashboard/wsgi/../../openstack_dashboard/dashboards/project/instances/workflows/create_instance.py", line 209, in _check_quotas
    usages = quotas.tenant_quota_usages(self.request)
  File "/usr/lib/python2.7/site-packages/horizon/utils/memoized.py", line 90, in wrapped
    value = cache[key] = func(*args, **kwargs)
  File "/usr/share/openstack-dashboard/openstack_dashboard/wsgi/../../openstack_dashboard/usage/quotas.py", line 361, in tenant_quota_usages
    _get_tenant_volume_usages(request, usages, disabled_quotas, tenant_id)
  File "/usr/share/openstack-dashboard/openstack_dashboard/wsgi/../../openstack_dashboard/usage/quotas.py", line 332, in _get_tenant_volume_usages
    snapshots = cinder.volume_snapshot_list(request, opts)
  File "/usr/share/openstack-dashboard/openstack_dashboard/wsgi/../../openstack_dashboard/api/cinder.py", line 302, in volume_snapshot_list
  File "/usr/lib/python2.7/site-packages/cinderclient/v2/volume_snapshots.py", line 131, in list
  File "/usr/lib/python2.7/site-packages/cinderclient/base.py", line 65, in _list
    resp, body = self.api.client.get(url)
  File "/usr/lib/python2.7/site-packages/cinderclient/client.py", line 326, in get
    return self._cs_request(url, 'GET', **kwargs)
  File "/usr/lib/python2.7/site-packages/cinderclient/client.py", line 289, in _cs_request
  File "/usr/lib/python2.7/site-packages/cinderclient/client.py", line 272, in request
    raise exceptions.from_response(resp, body)
ClientException: The server has either erred or is incapable of performing the requested operation. (HTTP 500) (Request-ID: req-cb7c6533-65f0-4905-8c72-2f8aef8e5b46)

Version-Release number of selected component:

How reproducible:

Steps to Reproduce:
1. Deploy RHOS 7.2 (z3)
2. Connect to Horizon as admin 
3. Create an instance

Actual results:
An error appears 

Danger: There was an error submitting the form. Please try again

Expected results:
instance created successfully
Comment 1 Matthias Runge 2015-12-02 10:13:21 EST
Cinder logs are showing:
2015-12-01 10:49:40.238 12951 ERROR cinder.api.middleware.fault [req-48f9277d-0ea8-4af9-b80c-a3e9eacdca6c
 00074bde5412442499a7390d98c824d9 0788735cc9a445ac95611db9dffafebb - - -] Caught error: Entity '<class 'c
inder.db.sqlalchemy.models.Snapshot'>' has no property 'tenant_id'
2015-12-01 10:49:40.239 12951 INFO cinder.api.middleware.fault [req-48f9277d-0ea8-4af9-b80c-a3e9eacdca6c 00074bde5412442499a7390d98c824d9 0788735cc9a445ac95611db9dffafebb - - -] http://cougar11.scl.lab.tlv.redhat.com:8776/v2/0788735cc9a445ac95611db9dffafebb/snapshots/detail?alltenants=1&tenant_id=0788735cc9a445ac95611db9dffafebb returned with HTTP 500
2015-12-01 10:49:40.240 12951 INFO eventlet.wsgi.server [req-48f9277d-0ea8-4af9-b80c-a3e9eacdca6c 00074bde5412442499a7390d98c824d9 0788735cc9a445ac95611db9dffafebb - - -] - - [01/Dec/2015 10:49:40] "GET /v2/0788735cc9a445ac95611db9dffafebb/snapshots/detail?alltenants=1&tenant_id=0788735cc9a445ac95611db9dffafebb HTTP/1.1" 500 425 0.007403

Which makes me think, this is a cinder issue.
Comment 2 Matthias Runge 2015-12-02 10:14:51 EST
*** Bug 1287664 has been marked as a duplicate of this bug. ***
Comment 3 Marius Cornea 2015-12-03 06:20:39 EST
*** Bug 1288015 has been marked as a duplicate of this bug. ***
Comment 4 Eric Harney 2015-12-03 09:34:22 EST
There is a Cinder bug here, but I think it is only triggered by Horizon, due to the search_opts passed in to snapshot detail list.  We'll fix up the Cinder side and then see if any corresponding changes are also needed in Horizon.
Comment 5 Eric Harney 2015-12-03 16:58:04 EST
Reproducer for cinder server problem:

Edit cinderclient's v2/shell.py: change search_opts in snapshot_list to send "alltenants" instead of "all_tenants" as the name of the field.

$ cinder snapshot-list --all-tenants 1
Comment 6 Eric Harney 2015-12-03 17:07:47 EST
Reverting the patch 

e080aba7 Filter snapshots data on the DB side

from Cinder causes Cinder to no longer throw HTTP 500 errors.

(Added in bug 1279547.)
Comment 7 Gorka Eguileor 2015-12-04 04:06:06 EST
Root of the problem is an horizon bug (even though Cinder should not return 500): https://bugs.launchpad.net/horizon/+bug/1469589

And it gets fixed in here: https://github.com/openstack/horizon/commit/faa44bfe1d6c4f342b8572bc4939771cda2039c7

As for Cinder, it should not return 500, so we need to we need to add filter validation (https://github.com/openstack/cinder/commit/8ba66fc5d2672bff69c21e36918e5e1882167649).  Some of the validations introduced in this patch are already in place, though some are missing on 7.0.

We need Horizon fix as well as the Cinder fix, because with only the Cinder fix Horizon will receive an empty list due to the malformed request.
Comment 8 Matthias Runge 2015-12-04 06:02:33 EST
*** Bug 1287770 has been marked as a duplicate of this bug. ***
Comment 9 Matthias Runge 2015-12-04 08:35:02 EST
Horizon backport is here https://review.openstack.org/253496
Comment 10 Eric Harney 2015-12-04 16:31:26 EST
Tested python-cinder-2014.2.3-11.el7ost.noarch for OSP6:

The Cinder server does not throw an HTTP 500.  It reports:

2015-12-04 16:31:49.473 19854 INFO cinder.api.openstack.wsgi [req-6e320001-798b-47ad-b7ea-b0368e7c023d 5bfa55d0e4844a96b744cb311c8ac4ce 252783c016b54c21999d73d3ececa650 - - -] GET
2015-12-04 16:31:49.477 19854 DEBUG cinder.db.sqlalchemy.api [req-6e320001-798b-47ad-b7ea-b0368e7c023d 5bfa55d0e4844a96b744cb311c8ac4ce 252783c016b54c21999d73d3ececa650 - - -] 'alltenants' filter key is not valid. snapshot_get_all /usr/lib/python2.7/site-packages/cinder/db/sqlalchemy/api.py:1742

It returns HTTP 200 with an empty list.  This should probably be an HTTP 400, but seems much less harmful than the OSP7 behavior.
Comment 11 Sergey Gotliv 2015-12-05 19:22:52 EST
Reverting this patch [1] doesn't make the situation any better. Yes, it addresses the immediate concern described in that case, but the idea is to move forward. I agree with comment#7 with one small correction, I think HTTP 400 is the better way to deal with the wrong filters than returning an empty list. 

1. Horizon should stop sending faulty/non-existing filters, exactly as comment#7 says.

2. If Horizon fixes the problem then Cinder should start validating filters and respond with HTTP 400 if filter is wrong. If Horizon doesn't fix the problem then
Cinder still has to validate filters but do that in the backward compatible way - means ignore faulty filters.

[1] https://code.engineering.redhat.com/gerrit/#/c/61203/
Comment 12 Sergey Gotliv 2015-12-05 19:55:57 EST
I want to modify my previous comment:

> If Horizon doesn't fix the problem then
> Cinder still has to validate filters but do that in the backward compatible way > - means ignore faulty filters.

If Horizon doesn't fix the problem then Cinder still has to validate filters and return an empty list if there are faulty filters as Cinder does in OSP 6.0, see  comment #10.
Comment 13 Matthias Runge 2015-12-07 08:55:58 EST
Just to make sure, I understand the issue on RHOS-6: I can not confirm an issue with and without having modified filters on horizon. 

I just tested openstack-cinder-2014.2.3-6.el7ost.noarch
Is the patch appearing to make this issue worse being backported to rhos-6 too?
Comment 15 Ido Ovadia 2015-12-10 08:43:04 EST
Comment 18 errata-xmlrpc 2015-12-21 12:04:01 EST
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.


Note You need to log in before you can comment on or make changes to this bug.