Bug 1709930

Summary: [OSP15] Exceptions In Neutron Policy Enforcement
Product: Red Hat OpenStack Reporter: Vadim Khitrin <vkhitrin>
Component: openstack-neutronAssignee: Nate Johnston <njohnston>
Status: CLOSED ERRATA QA Contact: Candido Campos <ccamposr>
Severity: high Docs Contact:
Priority: high    
Version: 15.0 (Stein)CC: amuller, bcafarel, cfontain, chrisw, fbaudin, njohnston, scohen, skaplons, supadhya, twilson
Target Milestone: betaKeywords: Triaged
Target Release: 15.0 (Stein)   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: openstack-neutron-14.0.2-0.20190604181640.a9d291b.el8 Doc Type: No Doc Update
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2019-09-21 11:21:58 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Vadim Khitrin 2019-05-14 15:01:54 UTC 
Description of problem:

When deploying with a modified list of Neutron API policies, post deployment, policies which worked on previous versions will result in Neutron API Server returning 'HttpException: 500' when using the API with non admin users.

Additional API policies which were passed during deployment: http://paste.openstack.org/show/751347/

Example:

1. Source credentials with non admin user
[stack@undercloud-0 ~]$ source /home/stack/overcloudrc_user_tenant
2. Query port list with as non admin user
(overcloud) [stack@undercloud-0 ~]$ openstack port list

At this point, neutron will return:
HttpException: 500: Server Error for url: http://10.35.141.150:9696/v2.0/ports, Internal Server Error

And the following exception will be generated inside server.log on controller nodes:
server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors [req-98bfd77f-1fc1-4d13-a0fd-02e82f6caa53 2236f6cc04c04964a0b435599ffb7acb ef4de28cbec04ea785b855010e7f46a1 - default default] An error occurred during processing the request: POST /v2.0/ports HTTP/1.0
server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors Traceback (most recent call last):
server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors   File "/usr/lib/python3.6/site-packages/oslo_middleware/catch_errors.py", line 40, in __call__
server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors     response = req.get_response(self.application)
server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors   File "/usr/lib/python3.6/site-packages/webob/request.py", line 1314, in send
server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors     application, catch_exc_info=False)
server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors   File "/usr/lib/python3.6/site-packages/webob/request.py", line 1278, in call_application
server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors     app_iter = application(self.environ, start_response)
server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors   File "/usr/lib/python3.6/site-packages/webob/dec.py", line 129, in __call__
server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors     resp = self.call_func(req, *args, **kw)
server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors   File "/usr/lib/python3.6/site-packages/webob/dec.py", line 193, in call_func
server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors     return self.func(req, *args, **kwargs)
server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors   File "/usr/lib/python3.6/site-packages/osprofiler/web.py", line 112, in __call__
server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors     return request.get_response(self.application)
server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors   File "/usr/lib/python3.6/site-packages/webob/request.py", line 1314, in send
server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors     application, catch_exc_info=False)
server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors   File "/usr/lib/python3.6/site-packages/webob/request.py", line 1278, in call_application
server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors     app_iter = application(self.environ, start_response)
server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors   File "/usr/lib/python3.6/site-packages/webob/dec.py", line 129, in __call__
server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors     resp = self.call_func(req, *args, **kw)
server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors   File "/usr/lib/python3.6/site-packages/webob/dec.py", line 193, in call_func
server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors     return self.func(req, *args, **kwargs)
server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors   File "/usr/lib/python3.6/site-packages/keystonemiddleware/auth_token/__init__.py", line 333, in __call__
server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors     response = req.get_response(self._app)
server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors   File "/usr/lib/python3.6/site-packages/webob/request.py", line 1314, in send
server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors     application, catch_exc_info=False)
server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors   File "/usr/lib/python3.6/site-packages/webob/request.py", line 1278, in call_application
server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors     app_iter = application(self.environ, start_response)
server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors   File "/usr/lib/python3.6/site-packages/webob/dec.py", line 143, in __call__
server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors     return resp(environ, start_response)
server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors   File "/usr/lib/python3.6/site-packages/webob/dec.py", line 143, in __call__
server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors     return resp(environ, start_response)
server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors   File "/usr/lib/python3.6/site-packages/routes/middleware.py", line 141, in __call__
server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors     response = self.app(environ, start_response)
server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors   File "/usr/lib/python3.6/site-packages/webob/dec.py", line 143, in __call__
server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors     return resp(environ, start_response)
server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors   File "/usr/lib/python3.6/site-packages/pecan/middleware/recursive.py", line 56, in __call__
server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors     return self.application(environ, start_response)
server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors   File "/usr/lib/python3.6/site-packages/pecan/core.py", line 840, in __call__
server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors     return super(Pecan, self).__call__(environ, start_response)
server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors   File "/usr/lib/python3.6/site-packages/pecan/core.py", line 736, in __call__
server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors     state
server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors   File "/usr/lib/python3.6/site-packages/pecan/core.py", line 865, in handle_hooks
server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors     return super(Pecan, self).handle_hooks(hooks, *args, **kw)
server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors   File "/usr/lib/python3.6/site-packages/pecan/core.py", line 342, in handle_hooks
server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors     result = getattr(hook, hook_type)(*args)
server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors   File "/usr/lib/python3.6/site-packages/neutron/pecan_wsgi/hooks/policy_enforcement.py", line 185, in after
server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors     for item in to_process
server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors   File "/usr/lib/python3.6/site-packages/neutron/pecan_wsgi/hooks/policy_enforcement.py", line 189, in <listcomp>
server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors     pluralized=collection))]
server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors   File "/usr/lib/python3.6/site-packages/neutron/pecan_wsgi/hooks/policy_enforcement.py", line 207, in _get_filtered_item
server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors     neutron_context, controller, resource, collection, data)
server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors   File "/usr/lib/python3.6/site-packages/neutron/pecan_wsgi/hooks/policy_enforcement.py", line 226, in _exclude_attributes_by_policy
server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors     for attr_name in data.keys():
server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors RuntimeError: dictionary changed size during iteration
server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors 
}

Version-Release number of selected component (if applicable):
Compose: RHOS_TRUNK-15.0-RHEL-8-20190509.n.1
rpm -qa | grep neutron
puppet-neutron-14.4.1-0.20190420042323.400fd54.el8ost.noarch
python3-neutronclient-6.12.0-0.20190312100012.680b417.el8ost.noarch


How reproducible:
Always


Steps to Reproduce:
1. Deploy Overcloud with modified Neutron APIs
2. Create non admin user/tenant
3. Attempt to list ports

Actual results:
Fail to retrieve ports and receive python exceptions

Expected results:
List of ports is returned

Additional info:
Comment 1 Vadim Khitrin 2019-05-14 15:02:13 UTC 
'RuntimeError: dictionary changed size during iteration' is raised because of different behaviour between Python2 and Python3, current workaround that I've tried is:

1) Log in to each controller node
2) Locate the overlay file system of 'neutron_api' container
podman mount neutron_api
/var/lib/containers/storage/overlay/9f1b0ad9960c377d7a6f65401835a4d78d7b314674b54620607ea1f7c1dea4f6/merged
3) Navigate to the path returned above
3) Open file 'usr/lib/python3.6/site-packages/neutron/pecan_wsgi/hooks/policy_enforcement.py' for editing
sudoedit usr/lib/python3.6/site-packages/neutron/pecan_wsgi/hooks/policy_enforcement.py
4) Replace line containing 'data.keys()' to 'list(data)' (as per this post, it's line number 226)
5) Restart 'neutron_api' container
podman restart neutron_api

After applying this, ports query works:
openstack port list
+--------------------------------------+------------------------------+-------------------+------------------------------------------------------------------------------+--------+
| ID                                   | Name                         | MAC Address       | Fixed IP Addresses                                                           | Status |
+--------------------------------------+------------------------------+-------------------+------------------------------------------------------------------------------+--------+
| 0f52f5eb-2430-44c2-866d-f127e6bba24b |                              | fa:16:3e:79:f7:1c | ip_address='40.0.0.101', subnet_id='3a64f067-bbe0-4715-b091-2965f9510726'    | DOWN   |
| 1382b3fe-0136-4f6e-84cf-2b2e745b2328 |                              | fa:16:3e:32:28:df | ip_address='40.0.0.100', subnet_id='3a64f067-bbe0-4715-b091-2965f9510726'    | DOWN   |
| 293d2303-be41-4fd1-8744-07950660c6e1 |                              | fa:16:3e:a2:ae:a0 | ip_address='40.0.0.102', subnet_id='3a64f067-bbe0-4715-b091-2965f9510726'    | DOWN   |
| 4540355c-5c22-4849-b09b-42f9be429377 |                              | fa:16:3e:ed:a0:36 | ip_address='10.10.110.102', subnet_id='405a2399-49d0-4bf1-8984-b4575d31ff94' | ACTIVE |
| a5119264-8080-44fc-a7fe-6ffacdfc291f | tempest-port-smoke-422258903 | fa:16:3e:85:44:eb | ip_address='10.10.110.109', subnet_id='405a2399-49d0-4bf1-8984-b4575d31ff94' | DOWN   |
| b9013482-aa63-4f87-a18d-cde70730eda0 |                              | fa:16:3e:37:8c:de | ip_address='50.0.0.100', subnet_id='f554b7cf-6b14-4425-9682-bfccd10d9bde'    | DOWN   |
| c5419c1a-b586-4953-9e6a-12294fe06b85 |                              | fa:16:3e:04:8d:f3 | ip_address='10.10.110.100', subnet_id='405a2399-49d0-4bf1-8984-b4575d31ff94' | ACTIVE |
| c7a80ba0-a525-4a15-adc3-6e9f53396ce8 |                              | fa:16:3e:01:97:72 | ip_address='50.0.0.102', subnet_id='f554b7cf-6b14-4425-9682-bfccd10d9bde'    | DOWN   |
| def7eee7-f4c2-401e-8b04-b292ce4cc735 |                              | fa:16:3e:73:86:00 | ip_address='50.0.0.101', subnet_id='f554b7cf-6b14-4425-9682-bfccd10d9bde'    | DOWN   |
| f3f5642e-bea1-480a-9f61-4197ebe8d96e |                              | fa:16:3e:5b:7f:20 | ip_address='10.10.110.101', subnet_id='405a2399-49d0-4bf1-8984-b4575d31ff94' | ACTIVE |
+--------------------------------------+------------------------------+-------------------+------------------------------------------------------------------------------+--------+
Comment 2 Nate Johnston 2019-05-15 23:07:17 UTC 
Upstreamed the recommended fix; it should get approved quickly.
Comment 3 Nate Johnston 2019-05-15 23:25:15 UTC 
Pushed change https://review.opendev.org/659397 to master, first change (stable/rocky) was a mistake.
Comment 4 Bernard Cafarelli 2019-05-17 10:00:52 UTC 
Master change merged
Comment 10 Bernard Cafarelli 2019-06-05 12:07:26 UTC 
Adding upstream review for additional fix
Comment 17 errata-xmlrpc 2019-09-21 11:21:58 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHEA-2019:2811