Bug 2101846

Summary: TripleO policy override file isn't rendering the correct definitions
Product: Red Hat OpenStack Reporter: Omer Schwartz <oschwart>
Component: openstack-octaviaAssignee: Gregory Thiemonge <gthiemon>
Status: CLOSED CURRENTRELEASE QA Contact: Bruna Bonguardo <bbonguar>
Severity: high Docs Contact:
Priority: high    
Version: 17.0 (Wallaby)CC: beagles, dmendiza, dwilde, gregraka, gthiemon, ihrachys, jjoyce, jschluet, lpeer, majopela, michjohn, njohnston, scohen, slinaber, tvignaud
Target Milestone: AlphaKeywords: AutomationBlocker, Triaged
Target Release: 17.1   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: Bug Fix
Doc Text:
Before this update, if secure RBAC was enabled, missing roles in the RHOSP deployment could cause Load-balancing service (octavia) API failures. In RHOSP 17.1 GA, this issue has been resolved.
Story Points: ---
Clone Of: Environment:
Last Closed: 2023-02-02 15:33:10 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1901686, 2124617    

Description Omer Schwartz 2022-06-28 14:29:17 UTC
Description of problem:
The TripleO policy override file https://github.com/openstack/tripleo-heat-templates/blob/master/environments/enable-secure-rbac.yaml isn't rendering the correct definitions for (at least) Octavia and probably other Openstack services as well.

For example:
In the following override section for Octavia policy
https://github.com/openstack/tripleo-heat-templates/blob/master/environments/enable-secure-rbac.yaml#L3701

we see role rules, such as
value: "role:admin or rule:project-reader"
value: "role:admin or rule:project-member"

that we do not see on our rendered policy.yaml file in our Jenkins jobs (here is a link from our latest build):
http://rhos-ci-logs.lab.eng.tlv2.redhat.com/logs/rcj/DFG-network-octavia-17.0_director-rhel-virthost-3cont_3comp-ipv4-geneve-actstby/66/controller-0/var/lib/config-data/octavia/etc/octavia/policy.yaml.gz

The policy.yaml file isn't rendering a definition for "rule:project-reader" nor "rule:project-member".

In order to verify the Secure RBAC feature, each OSP component needs to have those role rules defined, e.g. Octavia define them this way:
https://github.com/openstack/octavia/blob/master/etc/policy/keystone_default_roles-policy.yaml#L13


The fact that those role rules are not rendered causes many Secure RBAC tests to fail and blocks the Secure RBAC feature.


Version-Release number of selected component (if applicable):
core_puddle: RHOS-17.0-RHEL-9-20220623.n.1

How reproducible:
100%

Steps to Reproduce:
1. Run any Octavia OSP17 job on CI and add "enable-secure-rbac" on OVERCLOUD_TEMPLATES, and add 
--config-options load_balancer.enforce_new_defaults='True'
On TEMPEST_RUN_OVERRIDE_OPTIONS.
A link for the Active standby job (could be any):
https://rhos-ci-jenkins.lab.eng.tlv2.redhat.com/view/DFG/view/network/view/octavia/job/DFG-network-octavia-17.0_director-rhel-virthost-3cont_3comp-ipv4-geneve-actstby/


Actual results:
Important role rules are not being rendered on /controller-0/var/lib/config-data/octavia/etc/octavia/policy.yaml.gz
And many Secure RBAC tests fail.

Expected results:
All Secure RBAC should pass

Comment 2 Michael Johnson 2022-06-29 15:19:21 UTC
Root issue is the roles "project_reader" and "project_member" are not defined in the policy override rendered via tripleo and the enable-secure-rbac.yaml environment file.

Comment 3 Douglas Mendizábal 2022-07-08 19:51:23 UTC
I'm not sure I understand what the issue is here?  There seems to be a misunderstanding on how the policy.yaml override files work.

IIUC, the effective policy used by a service is a combination of both the default rules defined in code in the project as well the override rules specified in the policy.yaml file.  For Octavia, the "project-member" and "project-reader" rules have default values defined here: https://opendev.org/openstack/octavia/src/branch/stable/wallaby/octavia/policies/base.py#L52-L64

Because we are not changing the value of the "project-member" and "project-reader" rules, then we do not need to specify them in the policy.yaml override file.  In fact, the oslo.policy library provides a mechanism to help identify redundant rules that are present in both the default rules and the policy.yaml, noting that "These are rules that can be removed from the policy file with no change in effective policy": https://docs.openstack.org/oslo.policy/latest/user/usage.html#list-of-redundant-configuration

Are you seeing different behavior in Octavia?  Does the effective policy not contain the default rules?

Comment 4 Michael Johnson 2022-07-11 17:12:51 UTC
Yeah, you are right, those are globally defined in the defaults, so... hmmm, what is failing here?

Omer, can you point us to a current run that is failing? The linked job is passing and not using the policy override.

Comment 5 Michael Johnson 2022-07-12 00:08:20 UTC
I have looked into this some more. I found the job Omer was running (http://rhos-ci-logs.lab.eng.tlv2.redhat.com/logs/rcj/DFG-network-octavia-17.0_director-rhel-virthost-3cont_3comp-ipv4-geneve-actstby/66/controller-0/var/lib/config-data/octavia/etc/octavia/policy.yaml.gz)

I checked the tempest.conf and the RBAC test type is incorrect:
[load_balancer]
RBAC_test_type = owner_or_admin

http://rhos-ci-logs.lab.eng.tlv2.redhat.com/logs/rcj/DFG-network-octavia-17.0_director-rhel-virthost-3cont_3comp-ipv4-geneve-actstby/66/undercloud-0/home/stack/tempest-dir/etc/tempest.conf.gz

Which is likely causing this:
2022-06-27 13:48:59.490 19 DEBUG octavia.common.policy [req-55138779-6e68-4983-bf53-0beecfa8f691 - d2859124b48d4b68b28905c53203b92b - default default] Policy check for os_load-balancer_api:provider:get_all failed with credentials {'is_admin': False, 'user_id': None, 'user_domain_id': 'default', 'system_scope': None, 'domain_id': None, 'project_id': 'd2859124b48d4b68b28905c53203b92b', 'project_domain_id': 'default', 'roles': ['swiftoperator'], 'is_admin_project': True, 'service_user_id': None, 'service_user_domain_id': None, 'service_project_id': None, 'service_project_domain_id': None, 'service_roles': []} authorize /usr/lib/python3.9/site-packages/octavia/common/policy.py:126

Where the user being used doesn't have the reader or member role (It only has "swiftoperator" for whatever reason).

Please make sure you configure tempest to test the proper RBAC configuration for Octavia.

[load_balancer]
RBAC_test_type = keystone_default_roles