Description of problem: ======================= Launch an instance (on horizon) on admin project return Internal Server Error 2015-12-02 12:20:52,972 3930 ERROR django.request Internal Server Error: /dashboard/project/instances/launch Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/django/core/handlers/base.py", line 132, in get_response response = wrapped_callback(request, *callback_args, **callback_kwargs) File "/usr/lib/python2.7/site-packages/horizon/decorators.py", line 36, in dec return view_func(request, *args, **kwargs) File "/usr/lib/python2.7/site-packages/horizon/decorators.py", line 52, in dec return view_func(request, *args, **kwargs) File "/usr/lib/python2.7/site-packages/horizon/decorators.py", line 36, in dec return view_func(request, *args, **kwargs) File "/usr/lib/python2.7/site-packages/horizon/decorators.py", line 84, in dec return view_func(request, *args, **kwargs) File "/usr/lib/python2.7/site-packages/django/views/generic/base.py", line 71, in view return self.dispatch(request, *args, **kwargs) File "/usr/lib/python2.7/site-packages/django/views/generic/base.py", line 89, in dispatch return handler(request, *args, **kwargs) File "/usr/lib/python2.7/site-packages/horizon/workflows/views.py", line 167, in post context = self.get_context_data(**kwargs) File "/usr/lib/python2.7/site-packages/horizon/workflows/views.py", line 91, in get_context_data workflow = self.get_workflow() File "/usr/lib/python2.7/site-packages/horizon/workflows/views.py", line 81, in get_workflow entry_point=entry_point) File "/usr/lib/python2.7/site-packages/horizon/workflows/base.py", line 648, in __init__ valid = step.action.is_valid() File "/usr/lib/python2.7/site-packages/django/forms/forms.py", line 184, in is_valid return self.is_bound and not self.errors File "/usr/lib/python2.7/site-packages/django/forms/forms.py", line 176, in errors self.full_clean() File "/usr/lib/python2.7/site-packages/django/forms/forms.py", line 393, in full_clean self._clean_form() File "/usr/lib/python2.7/site-packages/django/forms/forms.py", line 417, in _clean_form cleaned_data = self.clean() File "/usr/share/openstack-dashboard/openstack_dashboard/wsgi/../../openstack_dashboard/dashboards/project/instances/workflows/create_instance.py", line 357, in clean self._check_quotas(cleaned_data) File "/usr/share/openstack-dashboard/openstack_dashboard/wsgi/../../openstack_dashboard/dashboards/project/instances/workflows/create_instance.py", line 209, in _check_quotas usages = quotas.tenant_quota_usages(self.request) File "/usr/lib/python2.7/site-packages/horizon/utils/memoized.py", line 90, in wrapped value = cache[key] = func(*args, **kwargs) File "/usr/share/openstack-dashboard/openstack_dashboard/wsgi/../../openstack_dashboard/usage/quotas.py", line 361, in tenant_quota_usages _get_tenant_volume_usages(request, usages, disabled_quotas, tenant_id) File "/usr/share/openstack-dashboard/openstack_dashboard/wsgi/../../openstack_dashboard/usage/quotas.py", line 332, in _get_tenant_volume_usages snapshots = cinder.volume_snapshot_list(request, opts) File "/usr/share/openstack-dashboard/openstack_dashboard/wsgi/../../openstack_dashboard/api/cinder.py", line 302, in volume_snapshot_list search_opts=search_opts)] File "/usr/lib/python2.7/site-packages/cinderclient/v2/volume_snapshots.py", line 131, in list "snapshots") File "/usr/lib/python2.7/site-packages/cinderclient/base.py", line 65, in _list resp, body = self.api.client.get(url) File "/usr/lib/python2.7/site-packages/cinderclient/client.py", line 326, in get return self._cs_request(url, 'GET', **kwargs) File "/usr/lib/python2.7/site-packages/cinderclient/client.py", line 289, in _cs_request **kwargs) File "/usr/lib/python2.7/site-packages/cinderclient/client.py", line 272, in request raise exceptions.from_response(resp, body) ClientException: The server has either erred or is incapable of performing the requested operation. (HTTP 500) (Request-ID: req-cb7c6533-65f0-4905-8c72-2f8aef8e5b46) Version-Release number of selected component: ============================================ python-django-horizon-2015.1.2-3.el7ost.noarch How reproducible: ================ 100% Steps to Reproduce: =================== 1. Deploy RHOS 7.2 (z3) 2. Connect to Horizon as admin 3. Create an instance Actual results: =============== An error appears Danger: There was an error submitting the form. Please try again Expected results: ================= instance created successfully
Cinder logs are showing: 2015-12-01 10:49:40.238 12951 ERROR cinder.api.middleware.fault [req-48f9277d-0ea8-4af9-b80c-a3e9eacdca6c 00074bde5412442499a7390d98c824d9 0788735cc9a445ac95611db9dffafebb - - -] Caught error: Entity '<class 'c inder.db.sqlalchemy.models.Snapshot'>' has no property 'tenant_id' 2015-12-01 10:49:40.239 12951 INFO cinder.api.middleware.fault [req-48f9277d-0ea8-4af9-b80c-a3e9eacdca6c 00074bde5412442499a7390d98c824d9 0788735cc9a445ac95611db9dffafebb - - -] http://cougar11.scl.lab.tlv.redhat.com:8776/v2/0788735cc9a445ac95611db9dffafebb/snapshots/detail?alltenants=1&tenant_id=0788735cc9a445ac95611db9dffafebb returned with HTTP 500 2015-12-01 10:49:40.240 12951 INFO eventlet.wsgi.server [req-48f9277d-0ea8-4af9-b80c-a3e9eacdca6c 00074bde5412442499a7390d98c824d9 0788735cc9a445ac95611db9dffafebb - - -] 10.35.160.111 - - [01/Dec/2015 10:49:40] "GET /v2/0788735cc9a445ac95611db9dffafebb/snapshots/detail?alltenants=1&tenant_id=0788735cc9a445ac95611db9dffafebb HTTP/1.1" 500 425 0.007403 Which makes me think, this is a cinder issue.
*** Bug 1287664 has been marked as a duplicate of this bug. ***
*** Bug 1288015 has been marked as a duplicate of this bug. ***
There is a Cinder bug here, but I think it is only triggered by Horizon, due to the search_opts passed in to snapshot detail list. We'll fix up the Cinder side and then see if any corresponding changes are also needed in Horizon.
Reproducer for cinder server problem: Edit cinderclient's v2/shell.py: change search_opts in snapshot_list to send "alltenants" instead of "all_tenants" as the name of the field. $ cinder snapshot-list --all-tenants 1
Reverting the patch e080aba7 Filter snapshots data on the DB side from Cinder causes Cinder to no longer throw HTTP 500 errors. (Added in bug 1279547.)
Root of the problem is an horizon bug (even though Cinder should not return 500): https://bugs.launchpad.net/horizon/+bug/1469589 And it gets fixed in here: https://github.com/openstack/horizon/commit/faa44bfe1d6c4f342b8572bc4939771cda2039c7 As for Cinder, it should not return 500, so we need to we need to add filter validation (https://github.com/openstack/cinder/commit/8ba66fc5d2672bff69c21e36918e5e1882167649). Some of the validations introduced in this patch are already in place, though some are missing on 7.0. We need Horizon fix as well as the Cinder fix, because with only the Cinder fix Horizon will receive an empty list due to the malformed request.
*** Bug 1287770 has been marked as a duplicate of this bug. ***
Horizon backport is here https://review.openstack.org/253496
Tested python-cinder-2014.2.3-11.el7ost.noarch for OSP6: The Cinder server does not throw an HTTP 500. It reports: 2015-12-04 16:31:49.473 19854 INFO cinder.api.openstack.wsgi [req-6e320001-798b-47ad-b7ea-b0368e7c023d 5bfa55d0e4844a96b744cb311c8ac4ce 252783c016b54c21999d73d3ececa650 - - -] GET http://10.16.151.248:8776/v2/252783c016b54c21999d73d3ececa650/snapshots/detail?all_tenants=1&alltenants=1 ... 2015-12-04 16:31:49.477 19854 DEBUG cinder.db.sqlalchemy.api [req-6e320001-798b-47ad-b7ea-b0368e7c023d 5bfa55d0e4844a96b744cb311c8ac4ce 252783c016b54c21999d73d3ececa650 - - -] 'alltenants' filter key is not valid. snapshot_get_all /usr/lib/python2.7/site-packages/cinder/db/sqlalchemy/api.py:1742 It returns HTTP 200 with an empty list. This should probably be an HTTP 400, but seems much less harmful than the OSP7 behavior.
Reverting this patch [1] doesn't make the situation any better. Yes, it addresses the immediate concern described in that case, but the idea is to move forward. I agree with comment#7 with one small correction, I think HTTP 400 is the better way to deal with the wrong filters than returning an empty list. 1. Horizon should stop sending faulty/non-existing filters, exactly as comment#7 says. 2. If Horizon fixes the problem then Cinder should start validating filters and respond with HTTP 400 if filter is wrong. If Horizon doesn't fix the problem then Cinder still has to validate filters but do that in the backward compatible way - means ignore faulty filters. [1] https://code.engineering.redhat.com/gerrit/#/c/61203/
I want to modify my previous comment: > If Horizon doesn't fix the problem then > Cinder still has to validate filters but do that in the backward compatible way > - means ignore faulty filters. If Horizon doesn't fix the problem then Cinder still has to validate filters and return an empty list if there are faulty filters as Cinder does in OSP 6.0, see comment #10.
Just to make sure, I understand the issue on RHOS-6: I can not confirm an issue with and without having modified filters on horizon. I just tested openstack-cinder-2014.2.3-6.el7ost.noarch Is the patch appearing to make this issue worse being backported to rhos-6 too?
Verified ======== openstack-cinder-2015.1.2-5.el7ost.noarch
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2015:2679