Description of problem: When deploying with a modified list of Neutron API policies, post deployment, policies which worked on previous versions will result in Neutron API Server returning 'HttpException: 500' when using the API with non admin users. Additional API policies which were passed during deployment: http://paste.openstack.org/show/751347/ Example: 1. Source credentials with non admin user [stack@undercloud-0 ~]$ source /home/stack/overcloudrc_user_tenant 2. Query port list with as non admin user (overcloud) [stack@undercloud-0 ~]$ openstack port list At this point, neutron will return: HttpException: 500: Server Error for url: http://10.35.141.150:9696/v2.0/ports, Internal Server Error And the following exception will be generated inside server.log on controller nodes: server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors [req-98bfd77f-1fc1-4d13-a0fd-02e82f6caa53 2236f6cc04c04964a0b435599ffb7acb ef4de28cbec04ea785b855010e7f46a1 - default default] An error occurred during processing the request: POST /v2.0/ports HTTP/1.0 server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors Traceback (most recent call last): server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors File "/usr/lib/python3.6/site-packages/oslo_middleware/catch_errors.py", line 40, in __call__ server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors response = req.get_response(self.application) server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors File "/usr/lib/python3.6/site-packages/webob/request.py", line 1314, in send server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors application, catch_exc_info=False) server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors File "/usr/lib/python3.6/site-packages/webob/request.py", line 1278, in call_application server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors app_iter = application(self.environ, start_response) server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors File "/usr/lib/python3.6/site-packages/webob/dec.py", line 129, in __call__ server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors resp = self.call_func(req, *args, **kw) server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors File "/usr/lib/python3.6/site-packages/webob/dec.py", line 193, in call_func server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors return self.func(req, *args, **kwargs) server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors File "/usr/lib/python3.6/site-packages/osprofiler/web.py", line 112, in __call__ server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors return request.get_response(self.application) server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors File "/usr/lib/python3.6/site-packages/webob/request.py", line 1314, in send server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors application, catch_exc_info=False) server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors File "/usr/lib/python3.6/site-packages/webob/request.py", line 1278, in call_application server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors app_iter = application(self.environ, start_response) server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors File "/usr/lib/python3.6/site-packages/webob/dec.py", line 129, in __call__ server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors resp = self.call_func(req, *args, **kw) server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors File "/usr/lib/python3.6/site-packages/webob/dec.py", line 193, in call_func server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors return self.func(req, *args, **kwargs) server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors File "/usr/lib/python3.6/site-packages/keystonemiddleware/auth_token/__init__.py", line 333, in __call__ server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors response = req.get_response(self._app) server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors File "/usr/lib/python3.6/site-packages/webob/request.py", line 1314, in send server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors application, catch_exc_info=False) server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors File "/usr/lib/python3.6/site-packages/webob/request.py", line 1278, in call_application server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors app_iter = application(self.environ, start_response) server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors File "/usr/lib/python3.6/site-packages/webob/dec.py", line 143, in __call__ server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors return resp(environ, start_response) server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors File "/usr/lib/python3.6/site-packages/webob/dec.py", line 143, in __call__ server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors return resp(environ, start_response) server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors File "/usr/lib/python3.6/site-packages/routes/middleware.py", line 141, in __call__ server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors response = self.app(environ, start_response) server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors File "/usr/lib/python3.6/site-packages/webob/dec.py", line 143, in __call__ server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors return resp(environ, start_response) server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors File "/usr/lib/python3.6/site-packages/pecan/middleware/recursive.py", line 56, in __call__ server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors return self.application(environ, start_response) server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors File "/usr/lib/python3.6/site-packages/pecan/core.py", line 840, in __call__ server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors return super(Pecan, self).__call__(environ, start_response) server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors File "/usr/lib/python3.6/site-packages/pecan/core.py", line 736, in __call__ server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors state server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors File "/usr/lib/python3.6/site-packages/pecan/core.py", line 865, in handle_hooks server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors return super(Pecan, self).handle_hooks(hooks, *args, **kw) server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors File "/usr/lib/python3.6/site-packages/pecan/core.py", line 342, in handle_hooks server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors result = getattr(hook, hook_type)(*args) server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors File "/usr/lib/python3.6/site-packages/neutron/pecan_wsgi/hooks/policy_enforcement.py", line 185, in after server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors for item in to_process server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors File "/usr/lib/python3.6/site-packages/neutron/pecan_wsgi/hooks/policy_enforcement.py", line 189, in <listcomp> server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors pluralized=collection))] server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors File "/usr/lib/python3.6/site-packages/neutron/pecan_wsgi/hooks/policy_enforcement.py", line 207, in _get_filtered_item server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors neutron_context, controller, resource, collection, data) server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors File "/usr/lib/python3.6/site-packages/neutron/pecan_wsgi/hooks/policy_enforcement.py", line 226, in _exclude_attributes_by_policy server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors for attr_name in data.keys(): server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors RuntimeError: dictionary changed size during iteration server.log:2019-05-08 07:33:43.076 22 ERROR oslo_middleware.catch_errors } Version-Release number of selected component (if applicable): Compose: RHOS_TRUNK-15.0-RHEL-8-20190509.n.1 rpm -qa | grep neutron puppet-neutron-14.4.1-0.20190420042323.400fd54.el8ost.noarch python3-neutronclient-6.12.0-0.20190312100012.680b417.el8ost.noarch How reproducible: Always Steps to Reproduce: 1. Deploy Overcloud with modified Neutron APIs 2. Create non admin user/tenant 3. Attempt to list ports Actual results: Fail to retrieve ports and receive python exceptions Expected results: List of ports is returned Additional info:
'RuntimeError: dictionary changed size during iteration' is raised because of different behaviour between Python2 and Python3, current workaround that I've tried is: 1) Log in to each controller node 2) Locate the overlay file system of 'neutron_api' container podman mount neutron_api /var/lib/containers/storage/overlay/9f1b0ad9960c377d7a6f65401835a4d78d7b314674b54620607ea1f7c1dea4f6/merged 3) Navigate to the path returned above 3) Open file 'usr/lib/python3.6/site-packages/neutron/pecan_wsgi/hooks/policy_enforcement.py' for editing sudoedit usr/lib/python3.6/site-packages/neutron/pecan_wsgi/hooks/policy_enforcement.py 4) Replace line containing 'data.keys()' to 'list(data)' (as per this post, it's line number 226) 5) Restart 'neutron_api' container podman restart neutron_api After applying this, ports query works: openstack port list +--------------------------------------+------------------------------+-------------------+------------------------------------------------------------------------------+--------+ | ID | Name | MAC Address | Fixed IP Addresses | Status | +--------------------------------------+------------------------------+-------------------+------------------------------------------------------------------------------+--------+ | 0f52f5eb-2430-44c2-866d-f127e6bba24b | | fa:16:3e:79:f7:1c | ip_address='40.0.0.101', subnet_id='3a64f067-bbe0-4715-b091-2965f9510726' | DOWN | | 1382b3fe-0136-4f6e-84cf-2b2e745b2328 | | fa:16:3e:32:28:df | ip_address='40.0.0.100', subnet_id='3a64f067-bbe0-4715-b091-2965f9510726' | DOWN | | 293d2303-be41-4fd1-8744-07950660c6e1 | | fa:16:3e:a2:ae:a0 | ip_address='40.0.0.102', subnet_id='3a64f067-bbe0-4715-b091-2965f9510726' | DOWN | | 4540355c-5c22-4849-b09b-42f9be429377 | | fa:16:3e:ed:a0:36 | ip_address='10.10.110.102', subnet_id='405a2399-49d0-4bf1-8984-b4575d31ff94' | ACTIVE | | a5119264-8080-44fc-a7fe-6ffacdfc291f | tempest-port-smoke-422258903 | fa:16:3e:85:44:eb | ip_address='10.10.110.109', subnet_id='405a2399-49d0-4bf1-8984-b4575d31ff94' | DOWN | | b9013482-aa63-4f87-a18d-cde70730eda0 | | fa:16:3e:37:8c:de | ip_address='50.0.0.100', subnet_id='f554b7cf-6b14-4425-9682-bfccd10d9bde' | DOWN | | c5419c1a-b586-4953-9e6a-12294fe06b85 | | fa:16:3e:04:8d:f3 | ip_address='10.10.110.100', subnet_id='405a2399-49d0-4bf1-8984-b4575d31ff94' | ACTIVE | | c7a80ba0-a525-4a15-adc3-6e9f53396ce8 | | fa:16:3e:01:97:72 | ip_address='50.0.0.102', subnet_id='f554b7cf-6b14-4425-9682-bfccd10d9bde' | DOWN | | def7eee7-f4c2-401e-8b04-b292ce4cc735 | | fa:16:3e:73:86:00 | ip_address='50.0.0.101', subnet_id='f554b7cf-6b14-4425-9682-bfccd10d9bde' | DOWN | | f3f5642e-bea1-480a-9f61-4197ebe8d96e | | fa:16:3e:5b:7f:20 | ip_address='10.10.110.101', subnet_id='405a2399-49d0-4bf1-8984-b4575d31ff94' | ACTIVE | +--------------------------------------+------------------------------+-------------------+------------------------------------------------------------------------------+--------+
Upstreamed the recommended fix; it should get approved quickly.
Pushed change https://review.opendev.org/659397 to master, first change (stable/rocky) was a mistake.
Master change merged
Adding upstream review for additional fix
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHEA-2019:2811