Hide Forgot
The storage operator declares a cluster-storage-operator-role ClusterRoleBinding [1]. However, it does not mention that resource in its ClusterOperator relatedObjects: $ curl -s https://gcsweb-ci.apps.ci.l2s4.p1.openshiftapps.com/gcs/origin-ci-test/pr-logs/pull/openshift_cluster-version-operator/562/pull-ci-openshift-cluster-version-operator-master-e2e-agnostic/1393324758669463552/artifacts/e2e-agnostic/gather-extra/artifacts/clusteroperators.json | jq -c '.items[] | select(.metadata.name == "storage").status.relatedObjects[]' {"group":"","name":"openshift-cluster-storage-operator","resource":"namespaces"} {"group":"","name":"openshift-cluster-csi-drivers","resource":"namespaces"} {"group":"operator.openshift.io","name":"cluster","resource":"storages"} Which means that it is not collected by must-gather: $ curl -s https://gcsweb-ci.apps.ci.l2s4.p1.openshiftapps.com/gcs/origin-ci-test/pr-logs/pull/openshift_cluster-version-operator/562/pull-ci-openshift-cluster-version-operator-master-e2e-agnostic/1393324758669463552/artifacts/e2e-agnostic/gather-must-gather/artifacts/must-gather.tar | tar tvz | grep /clusterrolebindings/ -rw------- 1001750000/root 608 2021-05-14 16:11 registry-build01-ci-openshift-org-ci-op-t363vyf0-stable-sha256-b26af26229ab63635a788809f1997f11bbe81d2a5a815203f7002c67711f68fb/cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/metrics-daemon-sa-rolebinding.yaml -rw------- 1001750000/root 619 2021-05-14 16:11 registry-build01-ci-openshift-org-ci-op-t363vyf0-stable-sha256-b26af26229ab63635a788809f1997f11bbe81d2a5a815203f7002c67711f68fb/cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/multus-admission-controller-webhook.yaml -rw------- 1001750000/root 582 2021-05-14 16:11 registry-build01-ci-openshift-org-ci-op-t363vyf0-stable-sha256-b26af26229ab63635a788809f1997f11bbe81d2a5a815203f7002c67711f68fb/cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/multus-whereabouts.yaml -rw------- 1001750000/root 561 2021-05-14 16:11 registry-build01-ci-openshift-org-ci-op-t363vyf0-stable-sha256-b26af26229ab63635a788809f1997f11bbe81d2a5a815203f7002c67711f68fb/cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/multus.yaml -rw------- 1001750000/root 613 2021-05-14 16:11 registry-build01-ci-openshift-org-ci-op-t363vyf0-stable-sha256-b26af26229ab63635a788809f1997f11bbe81d2a5a815203f7002c67711f68fb/cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/network-diagnostics.yaml -rw------- 1001750000/root 552 2021-05-14 16:11 registry-build01-ci-openshift-org-ci-op-t363vyf0-stable-sha256-b26af26229ab63635a788809f1997f11bbe81d2a5a815203f7002c67711f68fb/cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/openshift-image-registry-pruner.yaml -rw------- 1001750000/root 602 2021-05-14 16:11 registry-build01-ci-openshift-org-ci-op-t363vyf0-stable-sha256-b26af26229ab63635a788809f1997f11bbe81d2a5a815203f7002c67711f68fb/cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/openshift-sdn-controller.yaml -rw------- 1001750000/root 569 2021-05-14 16:11 registry-build01-ci-openshift-org-ci-op-t363vyf0-stable-sha256-b26af26229ab63635a788809f1997f11bbe81d2a5a815203f7002c67711f68fb/cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/openshift-sdn.yaml -rw------- 1001750000/root 542 2021-05-14 16:11 registry-build01-ci-openshift-org-ci-op-t363vyf0-stable-sha256-b26af26229ab63635a788809f1997f11bbe81d2a5a815203f7002c67711f68fb/cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/registry-registry-role.yaml For comparison, here are the image-registry releatedObjects (which lead to the must-gather picking up the registry ClusterRoleBindings): $ curl -s https://gcsweb-ci.apps.ci.l2s4.p1.openshiftapps.com/gcs/origin-ci-test/pr-logs/pull/openshift_cluster-version-operator/562/pull-ci-openshift-cluster-version-operator-master-e2e-agnostic/1393324758669463552/artifacts/e2e-agnostic/gather-extra/artifacts/clusteroperators.json | jq -c '.items[] | select(.metadata.name == "image-registry").status.relatedObjects[]' {"group":"imageregistry.operator.openshift.io","name":"cluster","resource":"configs"} {"group":"imageregistry.operator.openshift.io","name":"cluster","resource":"imagepruners"} {"group":"rbac.authorization.k8s.io","name":"system:registry","resource":"clusterroles"} {"group":"rbac.authorization.k8s.io","name":"registry-registry-role","resource":"clusterrolebindings"} {"group":"rbac.authorization.k8s.io","name":"openshift-image-registry-pruner","resource":"clusterrolebindings"} {"group":"","name":"openshift-image-registry","resource":"namespaces"} While we're touching this, we should probably also drop the namespace declaration from the ClusterRoleBinding manifest [2], because ClusterRoleBindings are not namespaced (they are cluster-scoped) [3,4]. And the storage operator should probably grow relatedObjects references in the ClusterOperator manifest [5]. Hard-coding relatedObjects in the ClusterOperator manifest is slightly more work, but means that must-gather will collect those related objects to help debug why your operator failed to come up, if it sticks before getting to the point of populating the ClusterOperator relatedObjects itself. [1]: https://github.com/openshift/cluster-storage-operator/blob/195603230796c2a7189f6daf45cb58b1a4fb72a3/manifests/08_operator_rbac.yaml#L4 [2]: https://github.com/openshift/cluster-storage-operator/blob/195603230796c2a7189f6daf45cb58b1a4fb72a3/manifests/08_operator_rbac.yaml#L5 [3]: https://kubernetes.io/docs/reference/access-authn-authz/rbac/#clusterrolebinding-example [4]: https://github.com/kubernetes/api/blob/9b64426eca51a74faa7cc9bd732a533d339c69c2/rbac/v1/types.go#L195-L200 [5]: https://github.com/openshift/cluster-storage-operator/blob/195603230796c2a7189f6daf45cb58b1a4fb72a3/manifests/11_cluster_operator.yaml#L14
> While we're touching this, we should probably also drop the namespace declaration from the ClusterRoleBinding manifest [2], because ClusterRoleBindings are not namespaced (they are cluster-scoped) [3,4]. This has been addressed through bug 1961538.
ClusterOperator.Status.RelatedObjects in CSO is composed partly statically at operator startup [1] and partly dynamically when we know what CSI driver we're going to start [2] 1: https://github.com/openshift/cluster-storage-operator/blob/c757199372ff217ef96e379f2add3c5d380d601e/pkg/operator/starter.go#L55 2: https://github.com/openshift/cluster-storage-operator/blob/c757199372ff217ef96e379f2add3c5d380d601e/pkg/operator/csidriveroperator/driverstarter.go#L146 At [2] we know what RBAC (and other) objects we're going to create in ctrl.operatorConfig.StaticAssets, e.g. for AWS EBS: https://github.com/openshift/cluster-storage-operator/blob/c757199372ff217ef96e379f2add3c5d380d601e/pkg/operator/csidriveroperator/csioperatorclient/aws.go#L26 We could add code to parse these YAML files and add them to relatedObjects at [2].
Verified with 4.9.0-0.nightly-2021-06-23-160041 CSI Driver related resource are added: $ oc get co storage -ojson | jq -r '.status.relatedObjects[] | select(.resource == "clusterrolebindings")' { "group": "rbac.authorization.k8s.io", "name": "aws-ebs-csi-driver-operator-clusterrolebinding", "resource": "clusterrolebindings" } But there is no clusterrolebindings/cluster-storage-operator-role: $ oc get clusterrolebindings cluster-storage-operator-role NAME ROLE AGE cluster-storage-operator-role ClusterRole/cluster-admin 84m Assigned back, please help confirm if this is expected result.
so the `static assets` for the "drivers" which are managed by "cluster-storage-operator" does not contain "cluster/-admin" For example, "aws-ebs-csi-driver-operator" is managed by "cluster-storage-operator", contains the following static assets ``` relatedObjects: - group: operator.openshift.io name: ebs.csi.aws.com resource: clustercsidrivers - group: "" name: aws-ebs-csi-driver-operator namespace: openshift-cluster-csi-drivers resource: serviceaccounts - group: rbac.authorization.k8s.io name: aws-ebs-csi-driver-operator-role namespace: openshift-cluster-csi-drivers resource: roles - group: rbac.authorization.k8s.io name: aws-ebs-csi-driver-operator-rolebinding namespace: openshift-cluster-csi-drivers resource: rolebindings - group: rbac.authorization.k8s.io name: aws-ebs-csi-driver-operator-clusterrole resource: clusterroles - group: rbac.authorization.k8s.io name: aws-ebs-csi-driver-operator-clusterrolebinding resource: clusterrolebindings - group: rbac.authorization.k8s.io name: aws-ebs-csi-driver-operator-aws-config-role namespace: openshift-config-managed resource: roles - group: rbac.authorization.k8s.io name: aws-ebs-csi-driver-operator-aws-config-clusterrolebinding namespace: openshift-config-managed resource: rolebindings - group: "" name: openshift-cluster-storage-operator resource: namespaces - group: "" name: openshift-cluster-csi-drivers resource: namespaces - group: operator.openshift.io name: cluster resource: storages ``` Moreover, any resources which are cluster-wide, or within the `all` category for a resource, will be collected by `oc adm inspect`
cluster-storage-operator-role is created by CVO and not by cluster-storage-operator. It needs to be listed statically.
@jan I am still assigned to it, however I do not know what I should do more here.
It's assigned to me, now we miss just one ClusterRoleBinding that I'm adding manually in the linked PR.
thanks so much
Verified pass with 4.10.0-0.nightly-2022-01-16-191814 $ oc get co storage -ojson | jq -r '.status.relatedObjects[] | select(.resource == "clusterrolebindings")' { "group": "rbac.authorization.k8s.io", "name": "gcp-pd-csi-driver-operator-clusterrolebinding", "resource": "clusterrolebindings" } { "group": "rbac.authorization.k8s.io", "name": "cluster-storage-operator-role", "resource": "clusterrolebindings" }
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Moderate: OpenShift Container Platform 4.10.3 security update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2022:0056