Bug 1961317 - storage ClusterOperator does not declare ClusterRoleBindings in relatedObjects
Summary: storage ClusterOperator does not declare ClusterRoleBindings in relatedObjects
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Storage
Version: 4.9
Hardware: Unspecified
OS: Unspecified
unspecified
medium
Target Milestone: ---
: 4.10.0
Assignee: Jan Safranek
QA Contact: Wei Duan
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2021-05-17 17:07 UTC by W. Trevor King
Modified: 2022-03-12 04:35 UTC (History)
3 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2022-03-12 04:35:20 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Github openshift cluster-storage-operator pull 175 0 None Merged Bug 1961317: Adding static assets to related objects 2022-01-08 00:29:09 UTC
Github openshift cluster-storage-operator pull 251 0 None open Bug 1961317: Add operator's ClusterRoleBinding to RelatedObjects 2022-01-08 00:29:08 UTC
Github openshift library-go pull 1099 0 None closed Bug 1961317: Adding static assets to related objects 2021-06-16 15:48:16 UTC
Red Hat Product Errata RHSA-2022:0056 0 None None None 2022-03-12 04:35:40 UTC

Description W. Trevor King 2021-05-17 17:07:57 UTC
The storage operator declares a cluster-storage-operator-role ClusterRoleBinding [1].  However, it does not mention that resource in its ClusterOperator relatedObjects:

$ curl -s https://gcsweb-ci.apps.ci.l2s4.p1.openshiftapps.com/gcs/origin-ci-test/pr-logs/pull/openshift_cluster-version-operator/562/pull-ci-openshift-cluster-version-operator-master-e2e-agnostic/1393324758669463552/artifacts/e2e-agnostic/gather-extra/artifacts/clusteroperators.json | jq -c '.items[] | select(.metadata.name == "storage").status.relatedObjects[]'
{"group":"","name":"openshift-cluster-storage-operator","resource":"namespaces"}
{"group":"","name":"openshift-cluster-csi-drivers","resource":"namespaces"}
{"group":"operator.openshift.io","name":"cluster","resource":"storages"}

Which means that it is not collected by must-gather:

$ curl -s https://gcsweb-ci.apps.ci.l2s4.p1.openshiftapps.com/gcs/origin-ci-test/pr-logs/pull/openshift_cluster-version-operator/562/pull-ci-openshift-cluster-version-operator-master-e2e-agnostic/1393324758669463552/artifacts/e2e-agnostic/gather-must-gather/artifacts/must-gather.tar | tar tvz | grep /clusterrolebindings/
-rw------- 1001750000/root     608 2021-05-14 16:11 registry-build01-ci-openshift-org-ci-op-t363vyf0-stable-sha256-b26af26229ab63635a788809f1997f11bbe81d2a5a815203f7002c67711f68fb/cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/metrics-daemon-sa-rolebinding.yaml
-rw------- 1001750000/root     619 2021-05-14 16:11 registry-build01-ci-openshift-org-ci-op-t363vyf0-stable-sha256-b26af26229ab63635a788809f1997f11bbe81d2a5a815203f7002c67711f68fb/cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/multus-admission-controller-webhook.yaml
-rw------- 1001750000/root     582 2021-05-14 16:11 registry-build01-ci-openshift-org-ci-op-t363vyf0-stable-sha256-b26af26229ab63635a788809f1997f11bbe81d2a5a815203f7002c67711f68fb/cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/multus-whereabouts.yaml
-rw------- 1001750000/root     561 2021-05-14 16:11 registry-build01-ci-openshift-org-ci-op-t363vyf0-stable-sha256-b26af26229ab63635a788809f1997f11bbe81d2a5a815203f7002c67711f68fb/cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/multus.yaml
-rw------- 1001750000/root     613 2021-05-14 16:11 registry-build01-ci-openshift-org-ci-op-t363vyf0-stable-sha256-b26af26229ab63635a788809f1997f11bbe81d2a5a815203f7002c67711f68fb/cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/network-diagnostics.yaml
-rw------- 1001750000/root     552 2021-05-14 16:11 registry-build01-ci-openshift-org-ci-op-t363vyf0-stable-sha256-b26af26229ab63635a788809f1997f11bbe81d2a5a815203f7002c67711f68fb/cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/openshift-image-registry-pruner.yaml
-rw------- 1001750000/root     602 2021-05-14 16:11 registry-build01-ci-openshift-org-ci-op-t363vyf0-stable-sha256-b26af26229ab63635a788809f1997f11bbe81d2a5a815203f7002c67711f68fb/cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/openshift-sdn-controller.yaml
-rw------- 1001750000/root     569 2021-05-14 16:11 registry-build01-ci-openshift-org-ci-op-t363vyf0-stable-sha256-b26af26229ab63635a788809f1997f11bbe81d2a5a815203f7002c67711f68fb/cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/openshift-sdn.yaml
-rw------- 1001750000/root     542 2021-05-14 16:11 registry-build01-ci-openshift-org-ci-op-t363vyf0-stable-sha256-b26af26229ab63635a788809f1997f11bbe81d2a5a815203f7002c67711f68fb/cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/registry-registry-role.yaml

For comparison, here are the image-registry releatedObjects (which lead to the must-gather picking up the registry ClusterRoleBindings):

$ curl -s https://gcsweb-ci.apps.ci.l2s4.p1.openshiftapps.com/gcs/origin-ci-test/pr-logs/pull/openshift_cluster-version-operator/562/pull-ci-openshift-cluster-version-operator-master-e2e-agnostic/1393324758669463552/artifacts/e2e-agnostic/gather-extra/artifacts/clusteroperators.json | jq -c '.items[] | select(.metadata.name == "image-registry").status.relatedObjects[]'
{"group":"imageregistry.operator.openshift.io","name":"cluster","resource":"configs"}
{"group":"imageregistry.operator.openshift.io","name":"cluster","resource":"imagepruners"}
{"group":"rbac.authorization.k8s.io","name":"system:registry","resource":"clusterroles"}
{"group":"rbac.authorization.k8s.io","name":"registry-registry-role","resource":"clusterrolebindings"}
{"group":"rbac.authorization.k8s.io","name":"openshift-image-registry-pruner","resource":"clusterrolebindings"}
{"group":"","name":"openshift-image-registry","resource":"namespaces"}

While we're touching this, we should probably also drop the namespace declaration from the ClusterRoleBinding manifest [2], because ClusterRoleBindings are not namespaced (they are cluster-scoped) [3,4].

And the storage operator should probably grow relatedObjects references in the ClusterOperator manifest [5].  Hard-coding relatedObjects in the ClusterOperator manifest is slightly more work, but means that must-gather will collect those related objects to help debug why your operator failed to come up, if it sticks before getting to the point of populating the ClusterOperator relatedObjects itself.

[1]: https://github.com/openshift/cluster-storage-operator/blob/195603230796c2a7189f6daf45cb58b1a4fb72a3/manifests/08_operator_rbac.yaml#L4
[2]: https://github.com/openshift/cluster-storage-operator/blob/195603230796c2a7189f6daf45cb58b1a4fb72a3/manifests/08_operator_rbac.yaml#L5
[3]: https://kubernetes.io/docs/reference/access-authn-authz/rbac/#clusterrolebinding-example
[4]: https://github.com/kubernetes/api/blob/9b64426eca51a74faa7cc9bd732a533d339c69c2/rbac/v1/types.go#L195-L200
[5]: https://github.com/openshift/cluster-storage-operator/blob/195603230796c2a7189f6daf45cb58b1a4fb72a3/manifests/11_cluster_operator.yaml#L14

Comment 1 W. Trevor King 2021-05-18 17:52:23 UTC
> While we're touching this, we should probably also drop the namespace declaration from the ClusterRoleBinding manifest [2], because ClusterRoleBindings are not namespaced (they are cluster-scoped) [3,4].

This has been addressed through bug 1961538.

Comment 2 Jan Safranek 2021-05-20 11:52:03 UTC
ClusterOperator.Status.RelatedObjects in CSO is composed partly statically at operator startup [1] and partly dynamically when we know what CSI driver we're going to start [2]
1: https://github.com/openshift/cluster-storage-operator/blob/c757199372ff217ef96e379f2add3c5d380d601e/pkg/operator/starter.go#L55
2: https://github.com/openshift/cluster-storage-operator/blob/c757199372ff217ef96e379f2add3c5d380d601e/pkg/operator/csidriveroperator/driverstarter.go#L146

At [2] we know what RBAC (and other) objects we're going to create in ctrl.operatorConfig.StaticAssets, e.g. for AWS EBS: https://github.com/openshift/cluster-storage-operator/blob/c757199372ff217ef96e379f2add3c5d380d601e/pkg/operator/csidriveroperator/csioperatorclient/aws.go#L26

We could add code to parse these YAML files and add them to relatedObjects at [2].

Comment 5 Wei Duan 2021-06-24 03:11:36 UTC
Verified with 4.9.0-0.nightly-2021-06-23-160041

CSI Driver related resource are added:  
$ oc get co storage -ojson | jq -r '.status.relatedObjects[] | select(.resource == "clusterrolebindings")'
{
  "group": "rbac.authorization.k8s.io",
  "name": "aws-ebs-csi-driver-operator-clusterrolebinding",
  "resource": "clusterrolebindings"
}


But there is no clusterrolebindings/cluster-storage-operator-role:  
$ oc get clusterrolebindings cluster-storage-operator-role
NAME                            ROLE                        AGE
cluster-storage-operator-role   ClusterRole/cluster-admin   84m

Assigned back, please help confirm if this is expected result.

Comment 6 melbeher 2021-06-24 07:32:52 UTC
so the `static assets` for the "drivers" which are managed by "cluster-storage-operator" does not contain "cluster/-admin" 

For example, "aws-ebs-csi-driver-operator" is managed by "cluster-storage-operator", contains the following static assets

```
relatedObjects:
  - group: operator.openshift.io
    name: ebs.csi.aws.com
    resource: clustercsidrivers
  - group: ""
    name: aws-ebs-csi-driver-operator
    namespace: openshift-cluster-csi-drivers
    resource: serviceaccounts
  - group: rbac.authorization.k8s.io
    name: aws-ebs-csi-driver-operator-role
    namespace: openshift-cluster-csi-drivers
    resource: roles
  - group: rbac.authorization.k8s.io
    name: aws-ebs-csi-driver-operator-rolebinding
    namespace: openshift-cluster-csi-drivers
    resource: rolebindings
  - group: rbac.authorization.k8s.io
    name: aws-ebs-csi-driver-operator-clusterrole
    resource: clusterroles
  - group: rbac.authorization.k8s.io
    name: aws-ebs-csi-driver-operator-clusterrolebinding
    resource: clusterrolebindings
  - group: rbac.authorization.k8s.io
    name: aws-ebs-csi-driver-operator-aws-config-role
    namespace: openshift-config-managed
    resource: roles
  - group: rbac.authorization.k8s.io
    name: aws-ebs-csi-driver-operator-aws-config-clusterrolebinding
    namespace: openshift-config-managed
    resource: rolebindings
  - group: ""
    name: openshift-cluster-storage-operator
    resource: namespaces
  - group: ""
    name: openshift-cluster-csi-drivers
    resource: namespaces
  - group: operator.openshift.io
    name: cluster
    resource: storages
```

Moreover, any resources which are cluster-wide, or within the `all` category for a resource, will be collected by `oc adm inspect`

Comment 7 Jan Safranek 2022-01-07 16:05:20 UTC
cluster-storage-operator-role is created by CVO and not by cluster-storage-operator. It needs to be listed statically.

Comment 8 melbeher 2022-01-09 15:28:12 UTC
@jan I am still assigned to it, however I do not know what I should do more here.

Comment 9 Jan Safranek 2022-01-10 09:56:40 UTC
It's assigned to me, now we miss just one ClusterRoleBinding that I'm adding manually in the linked PR.

Comment 10 melbeher 2022-01-10 11:12:08 UTC
thanks so much

Comment 13 Wei Duan 2022-01-17 02:31:27 UTC
Verified pass with 4.10.0-0.nightly-2022-01-16-191814

$ oc get co storage -ojson | jq -r '.status.relatedObjects[] | select(.resource == "clusterrolebindings")'
{
  "group": "rbac.authorization.k8s.io",
  "name": "gcp-pd-csi-driver-operator-clusterrolebinding",
  "resource": "clusterrolebindings"
}
{
  "group": "rbac.authorization.k8s.io",
  "name": "cluster-storage-operator-role",
  "resource": "clusterrolebindings"
}

Comment 16 errata-xmlrpc 2022-03-12 04:35:20 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: OpenShift Container Platform 4.10.3 security update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2022:0056


Note You need to log in before you can comment on or make changes to this bug.