Bug 1690848

Summary: The cluster user cannot use the resources provided by the operators
Product: OpenShift Container Platform Reporter: Jian Zhang <jiazha>
Component: OLMAssignee: Evan Cordell <ecordell>
Status: CLOSED ERRATA QA Contact: Jian Zhang <jiazha>
Severity: urgent Docs Contact:
Priority: high    
Version: 4.1.0CC: chezhang, dyan, jfan, jforrest, zitang
Target Milestone: ---   
Target Release: 4.1.0   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2019-06-04 10:46:13 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Jian Zhang 2019-03-20 10:50:34 UTC
Description of problem:
The cluster-admin user install the etcd-operator in a project, and, the project owner(common user) can see these service provided by the etcd-operator on the Web console. But, got errors when creating them. Got below errors:
etcdclusters.etcd.database.coreos.com is forbidden: User "pm1" cannot create resource "etcdclusters" in API group "etcd.database.coreos.com" in the namespace "test"

Version-Release number of selected component (if applicable):
Cluster version is 4.0.0-0.nightly-2019-03-18-223058

How reproducible:
always

Steps to Reproduce:
1. Install the OCP 4.0.
2. Set the htpasswd auth method, here is a script for it: https://mojo.redhat.com/docs/DOC-1191799-htpasswd-idp-on-40-cluster

3. Log in the cluster as a common user, and create a project called "test".

4. The cluster-admin user install the etcd-operator in project "test".

5. The common user click "Catalog" -> "Developer Catalog" -> "etcd Cluster" -> "Create", remove the below two lines, and click "Create".
  annotations:
    etcd.database.coreos.com/scope: clusterwide


Actual results:
Got the errors:
etcdclusters.etcd.database.coreos.com is forbidden: User "pm1" cannot create resource "etcdclusters" in API group "etcd.database.coreos.com" in the namespace "test"


Expected results:
The common user can create the "etcd Cluster" successfully. Or remove the "etcd Cluster" resource from the "Developer Catalog" section.

Additional info:
[jzhang@dhcp-140-18 ocp119]$ oc get role
NAME                        AGE
etcdoperator.v0.9.2-4dntm   127m
[jzhang@dhcp-140-18 ocp119]$ oc get rolebinding
NAME                                            AGE
admin                                           136m
etcdoperator.v0.9.2-4dntm-etcd-operator-d45zx   127m
system:deployers                                136m
system:image-builders                           136m
system:image-pullers                            136m

The clusterrole admin doesn't have the "etcd.database.coreos.com" permission.
[jzhang@dhcp-140-18 ocp119]$ oc get clusterrole admin -o yaml |grep etcd
[jzhang@dhcp-140-18 ocp119]$ 

[jzhang@dhcp-140-18 ocp119]$ oc get role etcdoperator.v0.9.2-4dntm -o yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  creationTimestamp: 2019-03-20T08:39:26Z
  labels:
    olm.owner: etcdoperator.v0.9.2
    olm.owner.kind: ClusterServiceVersion
    olm.owner.namespace: test
  name: etcdoperator.v0.9.2-4dntm
  namespace: test
  ownerReferences:
  - apiVersion: operators.coreos.com/v1alpha1
    blockOwnerDeletion: false
    controller: false
    kind: ClusterServiceVersion
    name: etcdoperator.v0.9.2
    uid: ab287577-4aeb-11e9-834e-021537aca5dc
  resourceVersion: "998993"
  selfLink: /apis/rbac.authorization.k8s.io/v1/namespaces/test/roles/etcdoperator.v0.9.2-4dntm
  uid: aba3181c-4aeb-11e9-834e-021537aca5dc
rules:
- apiGroups:
  - etcd.database.coreos.com
  resources:
  - etcdclusters
  - etcdbackups
  - etcdrestores
  verbs:
  - '*'
- apiGroups:
  - ""
  resources:
  - pods
  - services
  - endpoints
  - persistentvolumeclaims
  - events
  verbs:
  - '*'
- apiGroups:
  - apps
  resources:
  - deployments
  verbs:
  - '*'
- apiGroups:
  - ""
  resources:
  - secrets
  verbs:
  - get

[jzhang@dhcp-140-18 ocp119]$ oc get rolebinding etcdoperator.v0.9.2-4dntm-etcd-operator-d45zx -o yaml
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  creationTimestamp: 2019-03-20T08:39:27Z
  labels:
    olm.owner: etcdoperator.v0.9.2
    olm.owner.kind: ClusterServiceVersion
    olm.owner.namespace: test
  name: etcdoperator.v0.9.2-4dntm-etcd-operator-d45zx
  namespace: test
  ownerReferences:
  - apiVersion: operators.coreos.com/v1alpha1
    blockOwnerDeletion: false
    controller: false
    kind: ClusterServiceVersion
    name: etcdoperator.v0.9.2
    uid: ab287577-4aeb-11e9-834e-021537aca5dc
  resourceVersion: "1043433"
  selfLink: /apis/rbac.authorization.k8s.io/v1/namespaces/test/rolebindings/etcdoperator.v0.9.2-4dntm-etcd-operator-d45zx
  uid: abfe9f3b-4aeb-11e9-834e-021537aca5dc
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: etcdoperator.v0.9.2-4dntm
subjects:
- kind: ServiceAccount
  name: etcd-operator
  namespace: test

Comment 1 ge liu 2019-03-20 11:00:12 UTC
The issue result from missed user in rolebinding, add:
"
  apiGroup: rbac.authorization.k8s.io
  kind: User
  name: pm1
"

and this problem also appears with prometheus, couchbase operator also,

couchbaseclusters.couchbase.com is forbidden: User "pm5" cannot list resource "couchbaseclusters" in API group "couchbase.com" in the namespace "lgpm"

prometheuses.monitoring.coreos.com is forbidden: User "pm5" cannot list resource "prometheuses" in API group "monitoring.coreos.com" in the namespace "lgpm"

Comment 2 Jessica Forrester 2019-03-21 13:50:42 UTC
I'm reassigning this to the OLM team, its not specific to the etcd operator.

It sounds like we are not automatically aggregating permissions to the project admin and project editor roles when we make the operator available within a namespace.

Comment 3 Jian Zhang 2019-03-22 06:44:01 UTC
Jessica,

> It sounds like we are not automatically aggregating permissions to the project admin and project editor roles when we make the operator available within a namespace.

Yes, based on my understanding, the root cause is here:
Now, the etcd-operator(prometheus, couchbase, etc.) CSV used the permission[1] field, not the clusterpermissions.
So, the OLM couldn't aggregate these permissions to the ClusterRole "admin" automatically.
Hence, the project owner cannot manage these resources since the admin rolebinding referenced the ClusterRole "admin".

[1]: https://github.com/operator-framework/community-operators/blob/master/community-operators/etcd/etcdoperator.v0.9.2.clusterserviceversion.yaml#L146

Comment 7 Jian Zhang 2019-04-22 07:51:37 UTC
LGTM, verify it.

1, Login the cluster as the common user and create a project called "jian".
2, Login the cluster as the cluster-admin user and install the etcd-operator in project "jian".
3, The common user create the etcdCluster resource. Looks good to me, as below:
mac:~ jianzhang$ oc whoami
pm1
mac:~ jianzhang$ oc get etcdcluster
NAME      AGE
example   40s
mac:~ jianzhang$ oc get pods
NAME                            READY     STATUS     RESTARTS   AGE
etcd-operator-ffc975954-s8n2j   3/3       Running    0          8m51s
example-jkh7h74prq              0/1       Init:0/1   0          8s
example-lwmk5jm2dr              1/1       Running    0          32s
mac:~ jianzhang$ oc delete etcdcluster example
etcdcluster.etcd.database.coreos.com "example" deleted
mac:~ jianzhang$ oc get etcdcluster
No resources found.
mac:~ jianzhang$ oc get pods
NAME                 READY     STATUS        RESTARTS   AGE
example-lwmk5jm2dr   0/1       Terminating   0          65m

OLM version: io.openshift.build.commit.id=c718ec855bb26a111d66ba2ba193d30e54f7feb1
Cluster version is 4.1.0-0.nightly-2019-04-22-005054

Comment 9 errata-xmlrpc 2019-06-04 10:46:13 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2019:0758