Bug 2141470

Summary: neutron fails to check tenant_id when using custom policy.
Product: Red Hat OpenStack Reporter: Keigo Noha <knoha>
Component: python-neutron-libAssignee: Rodolfo Alonso <ralonsoh>
Status: CLOSED ERRATA QA Contact: nlevinki <nlevinki>
Severity: medium Docs Contact:
Priority: medium    
Version: 16.1 (Train)CC: apevec, chrisw, dhill, egarciar, jjoyce, jschluet, lhh, ralonsoh, scohen
Target Milestone: z5Keywords: Triaged
Target Release: 16.2 (Train on RHEL 8.4)   
Hardware: x86_64   
OS: Linux   
Whiteboard:
Fixed In Version: python-neutron-lib-1.29.1-2.20221201145504.4ef4b71.el8ost Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of:
: 2149905 (view as bug list) Environment:
Last Closed: 2023-04-26 12:17:18 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 2149905    

Description Keigo Noha 2022-11-10 01:39:35 UTC 
Description of problem:
We don't support custom policy itself. But this seems to be a bug in neutron.

When a user configures the following custom policy for neutron.
~~~
"create_policy":"rule:regular_user",
"delete_policy":"rule:admin_or_owner",
"update_policy":"rule:admin_or_owner",
"create_policy_bandwidth_limit_rule":"rule:regular_user",
"delete_policy_bandwidth_limit_rule":"rule:admin_or_owner",
"update_policy_bandwidth_limit_rule":"rule:admin_or_owner",
~~~

Then, they create a qos rule with a normal user and update the rule then gets an error.

~~~
~~~
(overcloud_test_user) [stack@undercloud-0 ~]$ openstack network qos rule set --max-kbps 30000 b22f27af-1141-4517-85cd-c0707c14ded6 ff0154d4-266e-4d0d-a484-cf981ced1ee1
Failed to set Network QoS rule ID "ff0154d4-266e-4d0d-a484-cf981ced1ee1": HttpException: 500: Server Error for url: http://10.0.0.130:9696/v2.0/qos/policies/b22f27af-1141-4517-85cd-c0707c14ded6/bandwidth_limit_rules/ff0154d4-266e-4d0d-a484-cf981ced1ee1, Failed to check policy tenant_id:%(tenant_id)s because Unable to verify match:%(tenant_id)s as the parent resource: tenant was not found.
~~~

Neutron server.log shows following logs.
~~~
2022-11-08 15:55:54.304 40 DEBUG neutron.api.v2.base [req-0deba5d6-a07d-40f0-9f7e-fb3e074af583 d00950961aac403db4c5988ecd9c43c8 a6812dcb67324fc6857a7bdad1c6ba
53 - default default] Request body: {'bandwidth_limit_rule': {'max_kbps': 30000}} prepare_request_body /usr/lib/python3.6/site-packages/neutron/api/v2/base.py
:719
2022-11-08 15:55:54.341 40 DEBUG neutron.policy [req-0deba5d6-a07d-40f0-9f7e-fb3e074af583 d00950961aac403db4c5988ecd9c43c8 a6812dcb67324fc6857a7bdad1c6ba53 - 
default default] Unable to find ':' as separator in tenant_id. __call__ /usr/lib/python3.6/site-packages/neutron/policy.py:303
2022-11-08 15:55:54.342 40 ERROR neutron.policy [req-0deba5d6-a07d-40f0-9f7e-fb3e074af583 d00950961aac403db4c5988ecd9c43c8 a6812dcb67324fc6857a7bdad1c6ba53 - 
default default] Unable to verify match:%(tenant_id)s as the parent resource: tenant was not found
2022-11-08 15:55:54.342 40 ERROR neutron.api.v2.resource [req-0deba5d6-a07d-40f0-9f7e-fb3e074af583 d00950961aac403db4c5988ecd9c43c8 a6812dcb67324fc6857a7bdad1
c6ba53 - default default] update failed: No details.: neutron_lib.exceptions.PolicyCheckError: Failed to check policy tenant_id:%(tenant_id)s because Unable t
o verify match:%(tenant_id)s as the parent resource: tenant was not found.
2022-11-08 15:55:54.342 40 ERROR neutron.api.v2.resource Traceback (most recent call last):
2022-11-08 15:55:54.342 40 ERROR neutron.api.v2.resource   File "/usr/lib/python3.6/site-packages/neutron/api/v2/resource.py", line 98, in resource
2022-11-08 15:55:54.342 40 ERROR neutron.api.v2.resource     result = method(request=request, **args)
2022-11-08 15:55:54.342 40 ERROR neutron.api.v2.resource   File "/usr/lib/python3.6/site-packages/neutron/api/v2/base.py", line 625, in update
2022-11-08 15:55:54.342 40 ERROR neutron.api.v2.resource     return self._update(request, id, body, **kwargs)
2022-11-08 15:55:54.342 40 ERROR neutron.api.v2.resource   File "/usr/lib/python3.6/site-packages/neutron_lib/db/api.py", line 139, in wrapped
2022-11-08 15:55:54.342 40 ERROR neutron.api.v2.resource     setattr(e, '_RETRY_EXCEEDED', True)
2022-11-08 15:55:54.342 40 ERROR neutron.api.v2.resource   File "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2022-11-08 15:55:54.342 40 ERROR neutron.api.v2.resource     self.force_reraise()
2022-11-08 15:55:54.342 40 ERROR neutron.api.v2.resource   File "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2022-11-08 15:55:54.342 40 ERROR neutron.api.v2.resource     six.reraise(self.type_, self.value, self.tb)
2022-11-08 15:55:54.342 40 ERROR neutron.api.v2.resource   File "/usr/lib/python3.6/site-packages/six.py", line 675, in reraise
2022-11-08 15:55:54.342 40 ERROR neutron.api.v2.resource     raise value
2022-11-08 15:55:54.342 40 ERROR neutron.api.v2.resource   File "/usr/lib/python3.6/site-packages/neutron_lib/db/api.py", line 135, in wrapped
2022-11-08 15:55:54.342 40 ERROR neutron.api.v2.resource     return f(*args, **kwargs)
2022-11-08 15:55:54.342 40 ERROR neutron.api.v2.resource   File "/usr/lib/python3.6/site-packages/oslo_db/api.py", line 154, in wrapper
2022-11-08 15:55:54.342 40 ERROR neutron.api.v2.resource     ectxt.value = e.inner_exc
2022-11-08 15:55:54.342 40 ERROR neutron.api.v2.resource   File "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2022-11-08 15:55:54.342 40 ERROR neutron.api.v2.resource     self.force_reraise()
2022-11-08 15:55:54.342 40 ERROR neutron.api.v2.resource   File "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2022-11-08 15:55:54.342 40 ERROR neutron.api.v2.resource     six.reraise(self.type_, self.value, self.tb)
2022-11-08 15:55:54.342 40 ERROR neutron.api.v2.resource   File "/usr/lib/python3.6/site-packages/six.py", line 675, in reraise
2022-11-08 15:55:54.342 40 ERROR neutron.api.v2.resource     raise value
2022-11-08 15:55:54.342 40 ERROR neutron.api.v2.resource   File "/usr/lib/python3.6/site-packages/oslo_db/api.py", line 142, in wrapper
2022-11-08 15:55:54.342 40 ERROR neutron.api.v2.resource     return f(*args, **kwargs)
2022-11-08 15:55:54.342 40 ERROR neutron.api.v2.resource   File "/usr/lib/python3.6/site-packages/neutron_lib/db/api.py", line 183, in wrapped
2022-11-08 15:55:54.342 40 ERROR neutron.api.v2.resource     LOG.debug("Retry wrapper got retriable exception: %s", e)
2022-11-08 15:55:54.342 40 ERROR neutron.api.v2.resource   File "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 220, in __exit__
2022-11-08 15:55:54.342 40 ERROR neutron.api.v2.resource     self.force_reraise()
2022-11-08 15:55:54.342 40 ERROR neutron.api.v2.resource   File "/usr/lib/python3.6/site-packages/oslo_utils/excutils.py", line 196, in force_reraise
2022-11-08 15:55:54.342 40 ERROR neutron.api.v2.resource     six.reraise(self.type_, self.value, self.tb)
2022-11-08 15:55:54.342 40 ERROR neutron.api.v2.resource   File "/usr/lib/python3.6/site-packages/six.py", line 675, in reraise
2022-11-08 15:55:54.342 40 ERROR neutron.api.v2.resource     raise value
2022-11-08 15:55:54.342 40 ERROR neutron.api.v2.resource   File "/usr/lib/python3.6/site-packages/neutron_lib/db/api.py", line 179, in wrapped
2022-11-08 15:55:54.342 40 ERROR neutron.api.v2.resource     return f(*dup_args, **dup_kwargs)
2022-11-08 15:55:54.342 40 ERROR neutron.api.v2.resource   File "/usr/lib/python3.6/site-packages/neutron/api/v2/base.py", line 664, in _update
2022-11-08 15:55:54.342 40 ERROR neutron.api.v2.resource     pluralized=self._collection)
2022-11-08 15:55:54.342 40 ERROR neutron.api.v2.resource   File "/usr/lib/python3.6/site-packages/neutron/policy.py", line 477, in enforce
2022-11-08 15:55:54.342 40 ERROR neutron.api.v2.resource     do_raise=True)
2022-11-08 15:55:54.342 40 ERROR neutron.api.v2.resource   File "/usr/lib/python3.6/site-packages/oslo_policy/policy.py", line 952, in enforce
2022-11-08 15:55:54.342 40 ERROR neutron.api.v2.resource     current_rule=None,
2022-11-08 15:55:54.342 40 ERROR neutron.api.v2.resource   File "/usr/lib/python3.6/site-packages/oslo_policy/_checks.py", line 86, in _check
2022-11-08 15:55:54.342 40 ERROR neutron.api.v2.resource     return rule(*rule_args)
2022-11-08 15:55:54.342 40 ERROR neutron.api.v2.resource   File "/usr/lib/python3.6/site-packages/oslo_policy/_checks.py", line 267, in __call__
2022-11-08 15:55:54.342 40 ERROR neutron.api.v2.resource     current_rule=current_rule,
2022-11-08 15:55:54.342 40 ERROR neutron.api.v2.resource   File "/usr/lib/python3.6/site-packages/oslo_policy/_checks.py", line 86, in _check
2022-11-08 15:55:54.342 40 ERROR neutron.api.v2.resource     return rule(*rule_args)
2022-11-08 15:55:54.342 40 ERROR neutron.api.v2.resource   File "/usr/lib/python3.6/site-packages/oslo_policy/_checks.py", line 267, in __call__
2022-11-08 15:55:54.342 40 ERROR neutron.api.v2.resource     current_rule=current_rule,
2022-11-08 15:55:54.342 40 ERROR neutron.api.v2.resource   File "/usr/lib/python3.6/site-packages/oslo_policy/_checks.py", line 86, in _check
2022-11-08 15:55:54.342 40 ERROR neutron.api.v2.resource     return rule(*rule_args)
2022-11-08 15:55:54.342 40 ERROR neutron.api.v2.resource   File "/usr/lib/python3.6/site-packages/oslo_policy/_checks.py", line 218, in __call__
2022-11-08 15:55:54.342 40 ERROR neutron.api.v2.resource     if _check(rule, target, cred, enforcer, current_rule):
2022-11-08 15:55:54.342 40 ERROR neutron.api.v2.resource   File "/usr/lib/python3.6/site-packages/oslo_policy/_checks.py", line 86, in _check
2022-11-08 15:55:54.342 40 ERROR neutron.api.v2.resource     return rule(*rule_args)
2022-11-08 15:55:54.342 40 ERROR neutron.api.v2.resource   File "/usr/lib/python3.6/site-packages/oslo_policy/_checks.py", line 267, in __call__
2022-11-08 15:55:54.342 40 ERROR neutron.api.v2.resource     current_rule=current_rule,
2022-11-08 15:55:54.342 40 ERROR neutron.api.v2.resource   File "/usr/lib/python3.6/site-packages/oslo_policy/_checks.py", line 86, in _check
2022-11-08 15:55:54.342 40 ERROR neutron.api.v2.resource     return rule(*rule_args)
2022-11-08 15:55:54.342 40 ERROR neutron.api.v2.resource   File "/usr/lib/python3.6/site-packages/neutron/policy.py", line 328, in __call__
2022-11-08 15:55:54.342 40 ERROR neutron.api.v2.resource     reason=err_reason)
2022-11-08 15:55:54.342 40 ERROR neutron.api.v2.resource neutron_lib.exceptions.PolicyCheckError: Failed to check policy tenant_id:%(tenant_id)s because Unable to verify match:%(tenant_id)s as the parent resource: tenant was not found.
2022-11-08 15:55:54.342 40 ERROR neutron.api.v2.resource 
2022-11-08 15:55:54.385 40 INFO neutron.wsgi [req-0deba5d6-a07d-40f0-9f7e-fb3e074af583 d00950961aac403db4c5988ecd9c43c8 a6812dcb67324fc6857a7bdad1c6ba53 - default default] 172.17.1.81 "PUT /v2.0/qos/policies/b22f27af-1141-4517-85cd-c0707c14ded6/bandwidth_limit_rules/ff0154d4-266e-4d0d-a484-cf981ced1ee1 HTTP/1.1" status: 500  len: 406 time: 0.0972245
~~~

It looks that the policy check failed to retrieve the tenant information for the operation.


Version-Release number of selected component (if applicable):
OSP16.1.6, 
openstack-neutron-15.2.1-1.20210409073445.40d217c.el8ost.noarch

How reproducible:
Everytime to update or delete a rule.

Steps to Reproduce:
1. Deploy Overcloud with the custom policy.
~~~
"create_policy":"rule:regular_user",
"delete_policy":"rule:admin_or_owner",
"update_policy":"rule:admin_or_owner",
"create_policy_bandwidth_limit_rule":"rule:regular_user",
"delete_policy_bandwidth_limit_rule":"rule:admin_or_owner",
"update_policy_bandwidth_limit_rule":"rule:admin_or_owner",
~~~
2. Create a project and admin user and normal user in the project. 
3. Use a normal user for create a policy and qos rule
4. Update qos rule then get the error.

Actual results:
The update operation failed.

Expected results:
The update operation succeeded.

Additional info:
Comment 21 errata-xmlrpc 2023-04-26 12:17:18 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Red Hat OpenStack Platform 16.2.5 (Train) bug fix and enhancement advisory), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2023:1763