1709930 – [OSP15] Exceptions In Neutron Policy Enforcement

Bug 1709930 - [OSP15] Exceptions In Neutron Policy Enforcement

Summary: [OSP15] Exceptions In Neutron Policy Enforcement

Keywords:
Status:	CLOSED ERRATA
Alias:	None
Product:	Red Hat OpenStack
Classification:	Red Hat
Component:	openstack-neutron
Sub Component:
Version:	15.0 (Stein)
Hardware:	Unspecified
OS:	Unspecified
Priority:	high
Severity:	high
Target Milestone:	beta
Target Release:	15.0 (Stein)
Assignee:	Nate Johnston
QA Contact:	Candido Campos
Docs Contact:
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+	depends on / blocked

Reported:	2019-05-14 15:01 UTC by Vadim Khitrin
Modified:	2019-10-29 17:16 UTC (History)
CC List:	10 users (show)
Fixed In Version:	openstack-neutron-14.0.2-0.20190604181640.a9d291b.el8
Doc Type:	No Doc Update
Doc Text:
Clone Of:
Environment:
Last Closed:	2019-09-21 11:21:58 UTC
Target Upstream Version:

Attachments	(Terms of Use)
Add an attachment (proposed patch, testcase, etc.)

Links
System	ID	Priority	Status	Summary	Last Updated
Launchpad	1829304	None	None	None	2019-05-15 23:07:17 UTC
OpenStack gerrit	659396	None	None	None	2019-05-15 23:07:17 UTC
OpenStack gerrit	659397	None	None	None	2019-05-15 23:25:15 UTC
OpenStack gerrit	662461	None	None	None	2019-06-05 12:07:26 UTC
Red Hat Product Errata	RHEA-2019:2811	None	None	None	2019-09-21 11:22:17 UTC

Description Vadim Khitrin 2019-05-14 15:01:54 UTC

Description of problem:

When deploying with a modified list of Neutron API policies, post deployment, policies which worked on previous versions will result in Neutron API Server returning 'HttpException: 500' when using the API with non admin users.

Additional API policies which were passed during deployment: http://paste.openstack.org/show/751347/

Example:

1. Source credentials with non admin user
[stack@undercloud-0 ~]$ source /home/stack/overcloudrc_user_tenant
2. Query port list with as non admin user
(overcloud) [stack@undercloud-0 ~]$ openstack port list

At this point, neutron will return:
HttpException: 500: Server Error for url: http://10.35.141.150:9696/v2.0/ports, Internal Server Error

And the following exception will be generated inside server.log on controller nodes:
server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors [req-98bfd77f-1fc1-4d13-a0fd-02e82f6caa53 2236f6cc04c04964a0b435599ffb7acb ef4de28cbec04ea785b855010e7f46a1 - default default] An error occurred during processing the request: POST /v2.0/ports HTTP/1.0
server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors Traceback (most recent call last):
server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors   File "/usr/lib/python3.6/site-packages/oslo_middleware/catch_errors.py", line 40, in __call__
server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors     response = req.get_response(self.application)
server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors   File "/usr/lib/python3.6/site-packages/webob/request.py", line 1314, in send
server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors     application, catch_exc_info=False)
server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors   File "/usr/lib/python3.6/site-packages/webob/request.py", line 1278, in call_application
server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors     app_iter = application(self.environ, start_response)
server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors   File "/usr/lib/python3.6/site-packages/webob/dec.py", line 129, in __call__
server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors     resp = self.call_func(req, *args, **kw)
server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors   File "/usr/lib/python3.6/site-packages/webob/dec.py", line 193, in call_func
server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors     return self.func(req, *args, **kwargs)
server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors   File "/usr/lib/python3.6/site-packages/osprofiler/web.py", line 112, in __call__
server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors     return request.get_response(self.application)
server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors   File "/usr/lib/python3.6/site-packages/webob/request.py", line 1314, in send
server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors     application, catch_exc_info=False)
server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors   File "/usr/lib/python3.6/site-packages/webob/request.py", line 1278, in call_application
server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors     app_iter = application(self.environ, start_response)
server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors   File "/usr/lib/python3.6/site-packages/webob/dec.py", line 129, in __call__
server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors     resp = self.call_func(req, *args, **kw)
server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors   File "/usr/lib/python3.6/site-packages/webob/dec.py", line 193, in call_func
server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors     return self.func(req, *args, **kwargs)
server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors   File "/usr/lib/python3.6/site-packages/keystonemiddleware/auth_token/__init__.py", line 333, in __call__
server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors     response = req.get_response(self._app)
server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors   File "/usr/lib/python3.6/site-packages/webob/request.py", line 1314, in send
server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors     application, catch_exc_info=False)
server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors   File "/usr/lib/python3.6/site-packages/webob/request.py", line 1278, in call_application
server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors     app_iter = application(self.environ, start_response)
server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors   File "/usr/lib/python3.6/site-packages/webob/dec.py", line 143, in __call__
server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors     return resp(environ, start_response)
server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors   File "/usr/lib/python3.6/site-packages/webob/dec.py", line 143, in __call__
server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors     return resp(environ, start_response)
server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors   File "/usr/lib/python3.6/site-packages/routes/middleware.py", line 141, in __call__
server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors     response = self.app(environ, start_response)
server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors   File "/usr/lib/python3.6/site-packages/webob/dec.py", line 143, in __call__
server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors     return resp(environ, start_response)
server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors   File "/usr/lib/python3.6/site-packages/pecan/middleware/recursive.py", line 56, in __call__
server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors     return self.application(environ, start_response)
server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors   File "/usr/lib/python3.6/site-packages/pecan/core.py", line 840, in __call__
server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors     return super(Pecan, self).__call__(environ, start_response)
server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors   File "/usr/lib/python3.6/site-packages/pecan/core.py", line 736, in __call__
server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors     state
server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors   File "/usr/lib/python3.6/site-packages/pecan/core.py", line 865, in handle_hooks
server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors     return super(Pecan, self).handle_hooks(hooks, *args, **kw)
server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors   File "/usr/lib/python3.6/site-packages/pecan/core.py", line 342, in handle_hooks
server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors     result = getattr(hook, hook_type)(*args)
server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors   File "/usr/lib/python3.6/site-packages/neutron/pecan_wsgi/hooks/policy_enforcement.py", line 185, in after
server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors     for item in to_process
server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors   File "/usr/lib/python3.6/site-packages/neutron/pecan_wsgi/hooks/policy_enforcement.py", line 189, in <listcomp>
server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors     pluralized=collection))]
server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors   File "/usr/lib/python3.6/site-packages/neutron/pecan_wsgi/hooks/policy_enforcement.py", line 207, in _get_filtered_item
server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors     neutron_context, controller, resource, collection, data)
server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors   File "/usr/lib/python3.6/site-packages/neutron/pecan_wsgi/hooks/policy_enforcement.py", line 226, in _exclude_attributes_by_policy
server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors     for attr_name in data.keys():
server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors RuntimeError: dictionary changed size during iteration
server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors 
}

Version-Release number of selected component (if applicable):
Compose: RHOS_TRUNK-15.0-RHEL-8-20190509.n.1
rpm -qa | grep neutron
puppet-neutron-14.4.1-0.20190420042323.400fd54.el8ost.noarch
python3-neutronclient-6.12.0-0.20190312100012.680b417.el8ost.noarch


How reproducible:
Always


Steps to Reproduce:
1. Deploy Overcloud with modified Neutron APIs
2. Create non admin user/tenant
3. Attempt to list ports

Actual results:
Fail to retrieve ports and receive python exceptions

Expected results:
List of ports is returned

Additional info:

Comment 1 Vadim Khitrin 2019-05-14 15:02:13 UTC

'RuntimeError: dictionary changed size during iteration' is raised because of different behaviour between Python2 and Python3, current workaround that I've tried is:

1) Log in to each controller node
2) Locate the overlay file system of 'neutron_api' container
podman mount neutron_api
/var/lib/containers/storage/overlay/9f1b0ad9960c377d7a6f65401835a4d78d7b314674b54620607ea1f7c1dea4f6/merged
3) Navigate to the path returned above
3) Open file 'usr/lib/python3.6/site-packages/neutron/pecan_wsgi/hooks/policy_enforcement.py' for editing
sudoedit usr/lib/python3.6/site-packages/neutron/pecan_wsgi/hooks/policy_enforcement.py
4) Replace line containing 'data.keys()' to 'list(data)' (as per this post, it's line number 226)
5) Restart 'neutron_api' container
podman restart neutron_api

After applying this, ports query works:
openstack port list
+--------------------------------------+------------------------------+-------------------+------------------------------------------------------------------------------+--------+
| ID                                   | Name                         | MAC Address       | Fixed IP Addresses                                                           | Status |
+--------------------------------------+------------------------------+-------------------+------------------------------------------------------------------------------+--------+
| 0f52f5eb-2430-44c2-866d-f127e6bba24b |                              | fa:16:3e:79:f7:1c | ip_address='40.0.0.101', subnet_id='3a64f067-bbe0-4715-b091-2965f9510726'    | DOWN   |
| 1382b3fe-0136-4f6e-84cf-2b2e745b2328 |                              | fa:16:3e:32:28:df | ip_address='40.0.0.100', subnet_id='3a64f067-bbe0-4715-b091-2965f9510726'    | DOWN   |
| 293d2303-be41-4fd1-8744-07950660c6e1 |                              | fa:16:3e:a2:ae:a0 | ip_address='40.0.0.102', subnet_id='3a64f067-bbe0-4715-b091-2965f9510726'    | DOWN   |
| 4540355c-5c22-4849-b09b-42f9be429377 |                              | fa:16:3e:ed:a0:36 | ip_address='10.10.110.102', subnet_id='405a2399-49d0-4bf1-8984-b4575d31ff94' | ACTIVE |
| a5119264-8080-44fc-a7fe-6ffacdfc291f | tempest-port-smoke-422258903 | fa:16:3e:85:44:eb | ip_address='10.10.110.109', subnet_id='405a2399-49d0-4bf1-8984-b4575d31ff94' | DOWN   |
| b9013482-aa63-4f87-a18d-cde70730eda0 |                              | fa:16:3e:37:8c:de | ip_address='50.0.0.100', subnet_id='f554b7cf-6b14-4425-9682-bfccd10d9bde'    | DOWN   |
| c5419c1a-b586-4953-9e6a-12294fe06b85 |                              | fa:16:3e:04:8d:f3 | ip_address='10.10.110.100', subnet_id='405a2399-49d0-4bf1-8984-b4575d31ff94' | ACTIVE |
| c7a80ba0-a525-4a15-adc3-6e9f53396ce8 |                              | fa:16:3e:01:97:72 | ip_address='50.0.0.102', subnet_id='f554b7cf-6b14-4425-9682-bfccd10d9bde'    | DOWN   |
| def7eee7-f4c2-401e-8b04-b292ce4cc735 |                              | fa:16:3e:73:86:00 | ip_address='50.0.0.101', subnet_id='f554b7cf-6b14-4425-9682-bfccd10d9bde'    | DOWN   |
| f3f5642e-bea1-480a-9f61-4197ebe8d96e |                              | fa:16:3e:5b:7f:20 | ip_address='10.10.110.101', subnet_id='405a2399-49d0-4bf1-8984-b4575d31ff94' | ACTIVE |
+--------------------------------------+------------------------------+-------------------+------------------------------------------------------------------------------+--------+

Comment 2 Nate Johnston 2019-05-15 23:07:17 UTC

Upstreamed the recommended fix; it should get approved quickly.

Comment 3 Nate Johnston 2019-05-15 23:25:15 UTC

Pushed change https://review.opendev.org/659397 to master, first change (stable/rocky) was a mistake.

Comment 4 Bernard Cafarelli 2019-05-17 10:00:52 UTC

Master change merged

Comment 10 Bernard Cafarelli 2019-06-05 12:07:26 UTC

Adding upstream review for additional fix

Comment 17 errata-xmlrpc 2019-09-21 11:21:58 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2019:2811

Note You need to log in before you can comment on or make changes to this bug.