These objects are updated every 3 minutes: $ for b in $(kubectl get clusterrolebindings -o name | sed 's,.*/,,'); do kubectl get clusterrolebinding $b -o json | jq 'select(.metadata.managedFields[0].time | startswith("2020-09-22T15:1")) | .metadata.name'; done "alertmanager-main" "grafana" "kube-state-metrics" "node-exporter" "openshift-state-metrics" "prometheus-adapter" "prometheus-adapter-view" "prometheus-k8s" "prometheus-operator" "resource-metrics:system:auth-delegator" "telemeter-client" "telemeter-client-view" "thanos-querier"
Stefan suggests possibly waiting until API-server support for server-side apply [1] goes GA and rerolling the CVO's apply logic to use that instead of client-side merging, which might help here. And bug 1879184 might end up with a [Late] CI guard based on the audit logs. But whatever is going on here is unlikely to be new in 4.6, so punting to 4.7. [1]: https://kubernetes.io/blog/2020/04/01/kubernetes-1.18-feature-server-side-apply-beta-2/
This one is important. It leads to RBAC errors and therefore contribute to failed CI.
I think this is a duplicate of https://bugzilla.redhat.com/show_bug.cgi?id=1863011
It's end of sprint, and this is not going to get fixed in the next few hours. Hopefully we will at least get the Late audit guard from bug 1879184 in next sprint, and then we'll see which team should fix this issue.
https://bugzilla.redhat.com/show_bug.cgi?id=1879184 is still open. Pushing this bug to next sprint as we have reached end of sprint.
Looks like the CVO manages no ClusterRoleBindings in 4.8: $ oc adm release extract --to 4.8 quay.io/openshift-release-dev/ocp-release:4.8.0-fc.2-x86_64 $ oc adm release extract --to 4.7 quay.io/openshift-release-dev/ocp-release:4.7.9-x86_64 $ oc adm release extract --to 4.6 quay.io/openshift-release-dev/ocp-release:4.6.27-x86_64 $ grep -ir clusterrolebindings 4.* 4.6/0000_50_cluster-node-tuning-operator_03-rbac.yaml: resources: ["clusterroles","clusterrolebindings"] 4.6/0000_50_cluster-monitoring-operator_02-role.yaml: - clusterrolebindings 4.6/0000_50_cluster-ingress-operator_00-cluster-role.yaml: - clusterrolebindings 4.6/0000_50_cluster-image-registry-operator_02-rbac.yaml: - clusterrolebindings 4.6/0000_70_dns-operator_00-cluster-role.yaml: - clusterrolebindings 4.6/0000_50_cloud-credential-operator_01-cluster-role.yaml: - clusterrolebindings 4.7/0000_50_cluster-node-tuning-operator_40-rbac.yaml: resources: ["clusterroles","clusterrolebindings"] 4.7/0000_50_cluster-monitoring-operator_02-role.yaml: - clusterrolebindings 4.7/0000_50_cluster-ingress-operator_00-cluster-role.yaml: - clusterrolebindings 4.7/0000_50_cluster-image-registry-operator_02-rbac.yaml: - clusterrolebindings 4.7/0000_70_dns-operator_00-cluster-role.yaml: - clusterrolebindings 4.7/0000_50_cloud-credential-operator_01-cluster-role.yaml: - clusterrolebindings 4.8/0000_50_cluster-node-tuning-operator_40-rbac.yaml: resources: ["clusterroles","clusterrolebindings"] 4.8/0000_50_cluster-monitoring-operator_02-role.yaml: - clusterrolebindings 4.8/0000_50_cluster-ingress-operator_00-cluster-role.yaml: - clusterrolebindings 4.8/0000_50_cluster-image-registry-operator_02-rbac.yaml: - clusterrolebindings 4.8/0000_70_dns-operator_00-cluster-role.yaml: - clusterrolebindings 4.8/0000_50_cloud-credential-operator_01-cluster-role.yaml: - clusterrolebindings Checking 4.6.26 -> 4.6.27 CI [1]: $ curl -s https://gcsweb-ci.apps.ci.l2s4.p1.openshiftapps.com/gcs/origin-ci-test/logs/release-openshift-origin-installer-launch-gcp/1387416268704845824/artifacts/launch/must-gather.tar | tar tvz | grep clusterrolebinding drwxr-xr-x 1026890000/root 0 2021-04-28 10:16 ./quay-io-openshift-release-dev-ocp-v4-0-art-dev-sha256-fed3177b3a93ba35725028dc190506f45c886b58fe28fe91d18ce516f94a6c91/cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/ -rwxr-xr-x 1026890000/root 1300 2021-04-28 10:15 ./quay-io-openshift-release-dev-ocp-v4-0-art-dev-sha256-fed3177b3a93ba35725028dc190506f45c886b58fe28fe91d18ce516f94a6c91/cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/metrics-daemon-sa-rolebinding.yaml -rwxr-xr-x 1026890000/root 1317 2021-04-28 10:15 ./quay-io-openshift-release-dev-ocp-v4-0-art-dev-sha256-fed3177b3a93ba35725028dc190506f45c886b58fe28fe91d18ce516f94a6c91/cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/multus-admission-controller-webhook.yaml -rwxr-xr-x 1026890000/root 1263 2021-04-28 10:15 ./quay-io-openshift-release-dev-ocp-v4-0-art-dev-sha256-fed3177b3a93ba35725028dc190506f45c886b58fe28fe91d18ce516f94a6c91/cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/multus-whereabouts.yaml -rwxr-xr-x 1026890000/root 1230 2021-04-28 10:15 ./quay-io-openshift-release-dev-ocp-v4-0-art-dev-sha256-fed3177b3a93ba35725028dc190506f45c886b58fe28fe91d18ce516f94a6c91/cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/multus.yaml -rwxr-xr-x 1026890000/root 1069 2021-04-28 10:14 ./quay-io-openshift-release-dev-ocp-v4-0-art-dev-sha256-fed3177b3a93ba35725028dc190506f45c886b58fe28fe91d18ce516f94a6c91/cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/openshift-image-registry-pruner.yaml -rwxr-xr-x 1026890000/root 1289 2021-04-28 10:15 ./quay-io-openshift-release-dev-ocp-v4-0-art-dev-sha256-fed3177b3a93ba35725028dc190506f45c886b58fe28fe91d18ce516f94a6c91/cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/openshift-sdn-controller.yaml -rwxr-xr-x 1026890000/root 1245 2021-04-28 10:15 ./quay-io-openshift-release-dev-ocp-v4-0-art-dev-sha256-fed3177b3a93ba35725028dc190506f45c886b58fe28fe91d18ce516f94a6c91/cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/openshift-sdn.yaml -rwxr-xr-x 1026890000/root 1050 2021-04-28 10:14 ./quay-io-openshift-release-dev-ocp-v4-0-art-dev-sha256-fed3177b3a93ba35725028dc190506f45c886b58fe28fe91d18ce516f94a6c91/cluster-scoped-resources/rbac.authorization.k8s.io/clusterrolebindings/registry-registry-role.yaml So... I'm going to launch a 4.6.27 cluster to see if I can reproduce comment 0, because I don't see how the CVO's involved here. [1]: https://prow.ci.openshift.org/view/gs/origin-ci-test/logs/release-openshift-origin-installer-launch-gcp/1387416268704845824 I would have expected a 'kind: ClusterRoleBinding'
$ oc get clusterversion -o jsonpath='{.status.desired.version}{"\n"}' version 4.6.27 $ oc get -o json clusterrolebindings | jq -r '.items[].metadata | .name as $n | .managedFields[] | .time + " " + .operation + " " + .manager + " " + $n' | sort 2021-05-04T00:41:22Z Update kube-apiserver cluster-admin 2021-05-04T00:41:22Z Update kube-apiserver system:basic-user 2021-05-04T00:41:22Z Update kube-apiserver system:controller:attachdetach-controller ... 2021-05-04T00:53:22Z Update openshift-apiserver system:webhooks 2021-05-04T00:56:48Z Update cluster-version-operator console 2021-05-04T00:56:48Z Update cluster-version-operator console-extensions-reader 2021-05-04T00:56:48Z Update cluster-version-operator console-operator 2021-05-04T00:56:48Z Update cluster-version-operator console-operator-auth-delegator 2021-05-04T00:56:49Z Update cluster-version-operator helm-chartrepos-view 2021-05-04T00:57:19Z Update cluster-image-registry-operator registry-registry-role 2021-05-04T00:58:02Z Update operator alertmanager-main 2021-05-04T00:58:04Z Update operator prometheus-k8s 2021-05-04T00:58:05Z Update operator thanos-querier Those update times don't seem to be increasing. As I run this, it is: $ date --iso=m --utc 2021-05-04T03:22+00:00 Checking one of the examples with a hopefully grep'able name: $ oc get -o yaml clusterrolebindings console-operator-auth-delegator apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRoleBinding metadata: creationTimestamp: "2021-05-04T00:56:48Z" managedFields: - apiVersion: rbac.authorization.k8s.io/v1 fieldsType: FieldsV1 fieldsV1: f:roleRef: f:apiGroup: {} f:kind: {} f:name: {} f:subjects: {} manager: cluster-version-operator operation: Update time: "2021-05-04T00:56:48Z" name: console-operator-auth-delegator resourceVersion: "15765" selfLink: /apis/rbac.authorization.k8s.io/v1/clusterrolebindings/console-operator-auth-delegator uid: d5b682be-4cc3-43c1-a6ca-488ff276effb roleRef: apiGroup: rbac.authorization.k8s.io kind: ClusterRole name: system:auth-delegator subjects: - kind: ServiceAccount name: console-operator namespace: openshift-console-operator $ grep -r console-operator-auth-delegator 4.* 4.6/0000_50_console-operator_04-rbac-rolebinding-cluster.yaml: name: console-operator-auth-delegator 4.7/0000_50_console-operator_04-rbac-rolebinding-cluster.yaml: name: console-operator-auth-delegator 4.8/0000_50_console-operator_04-rbac-rolebinding-cluster.yaml: name: console-operator-auth-delegator Ah, Looks like I flubbed comment 10 by checking for the plural. Fixing: $ grep -ir 'kind: clusterrolebinding' 4.* 4.6/0000_50_service-ca-operator_00_roles.yaml:kind: ClusterRoleBinding 4.6/0000_50_operator-marketplace_06_role_binding.yaml:kind: ClusterRoleBinding 4.6/0000_50_olm_01-olm-operator.serviceaccount.yaml:kind: ClusterRoleBinding ... Bingo. Poking at the 4.*/0000_50_console-operator_04-rbac-rolebinding-cluster.yaml, the only significant changes appear to be cluster-profile annotations coming in in 4.7. Here's the manifest: $ yaml2json <4.6/0000_50_console-operator_04-rbac-rolebinding-cluster.yaml | jq '.[] | select(.metadata.name == "consle-operator-auth-delegator")' { "apiVersion": "rbac.authorization.k8s.io/v1", "kind": "ClusterRoleBinding", "metadata": { "name": "console-operator-auth-delegator" }, "roleRef": { "apiGroup": "rbac.authorization.k8s.io", "kind": "ClusterRole", "name": "system:auth-delegator" }, "subjects": [ { "kind": "ServiceAccount", "name": "console-operator", "namespace": "openshift-console-operator" } ] } I don't see anything there that the CVO would think is a mismatch. The resourceVersion seems surprisingly high, but checking again now after a number of minutes, it doesn't seems to have changed. And still, no further bumps to any managedFields[].time: $ date --iso=m --utc 2021-05-04T03:34+00:00 $ oc get -o json clusterrolebindings | jq -r '.items[].metadata | .name as $n | .managedFields[] | .time + " " + .operator + " " + .manager + " " + $n' | sort | tail -n1 2021-05-04T00:58:05Z operator thanos-querier So I'm going to mark this INSUFFICIENT_DATA. But I missed something pretty obvious in comment 10 (the plural grep), so feel free to re-open if I'm missing something pretty obvious in this comment too.
CVO hotloops on ClusterRoleBindings when comparing RoleRef: some roles don't have `APIGroup` set in the manifests, so CVO tries to apply them over and over again
Using the PR's presubmit [1]: $ curl -s https://gcsweb-ci.apps.ci.l2s4.p1.openshiftapps.com/gcs/origin-ci-test/pr-logs/pull/openshift_cluster-version-operator/562/pull-ci-openshift-cluster-version-operator-master-e2e-agnostic/1393324758669463552/artifacts/e2e-agnostic/gather-audit-logs/artifacts/audit-logs.tar | tar xvz --strip-components=1 $ zgrep -h '"verb":"update".*"resource":".*rolebindings"' audit_logs/*/*.gz 2>/dev/null | jq -r '.user.username + " " + (.objectRef | .resource + " " + .namespace + " " + .name) + " " + .stageTimestamp + " " + (.responseStatus | tostring)' | grep cluster-version | sort system:serviceaccount:openshift-cluster-version:default clusterrolebindings openshift-cluster-storage-operator cluster-storage-operator-role 2021-05-14T22:29:28.650191Z {"metadata":{},"code":200} system:serviceaccount:openshift-cluster-version:default clusterrolebindings openshift-cluster-storage-operator cluster-storage-operator-role 2021-05-14T22:32:40.552381Z {"metadata":{},"code":200} system:serviceaccount:openshift-cluster-version:default clusterrolebindings openshift-cluster-storage-operator cluster-storage-operator-role 2021-05-14T22:36:23.541930Z {"metadata":{},"code":200} system:serviceaccount:openshift-cluster-version:default clusterrolebindings openshift-cluster-storage-operator cluster-storage-operator-role 2021-05-14T22:40:38.269087Z {"metadata":{},"code":200} system:serviceaccount:openshift-cluster-version:default clusterrolebindings openshift-cluster-storage-operator cluster-storage-operator-role 2021-05-14T22:43:33.047373Z {"metadata":{},"code":200} system:serviceaccount:openshift-cluster-version:default clusterrolebindings openshift-cluster-storage-operator cluster-storage-operator-role 2021-05-14T22:46:51.679138Z {"metadata":{},"code":200} system:serviceaccount:openshift-cluster-version:default clusterrolebindings openshift-cluster-storage-operator cluster-storage-operator-role 2021-05-14T22:50:10.298785Z {"metadata":{},"code":200} system:serviceaccount:openshift-cluster-version:default clusterrolebindings openshift-cluster-storage-operator cluster-storage-operator-role 2021-05-14T22:53:28.960713Z {"metadata":{},"code":200} system:serviceaccount:openshift-cluster-version:default clusterrolebindings openshift-cluster-storage-operator cluster-storage-operator-role 2021-05-14T22:56:47.408016Z {"metadata":{},"code":200} system:serviceaccount:openshift-cluster-version:default clusterrolebindings openshift-cluster-storage-operator cluster-storage-operator-role 2021-05-14T23:00:06.027115Z {"metadata":{},"code":200} system:serviceaccount:openshift-cluster-version:default clusterrolebindings openshift-cluster-storage-operator cluster-storage-operator-role 2021-05-14T23:03:24.513286Z {"metadata":{},"code":200} system:serviceaccount:openshift-cluster-version:default clusterrolebindings openshift-cluster-storage-operator cluster-storage-operator-role 2021-05-14T23:06:43.041391Z {"metadata":{},"code":200} system:serviceaccount:openshift-cluster-version:default clusterrolebindings openshift-cluster-storage-operator cluster-storage-operator-role 2021-05-14T23:10:01.559921Z {"metadata":{},"code":200} system:serviceaccount:openshift-cluster-version:default clusterrolebindings openshift-cluster-storage-operator cluster-storage-operator-role 2021-05-14T23:13:20.061635Z {"metadata":{},"code":200} system:serviceaccount:openshift-cluster-version:default clusterrolebindings openshift-cluster-storage-operator csi-snapshot-controller-operator-role 2021-05-14T22:27:30.283347Z {"metadata":{},"code":200} system:serviceaccount:openshift-cluster-version:default clusterrolebindings openshift-cluster-storage-operator csi-snapshot-controller-operator-role 2021-05-14T22:29:24.700377Z {"metadata":{},"code":200} system:serviceaccount:openshift-cluster-version:default clusterrolebindings openshift-cluster-storage-operator csi-snapshot-controller-operator-role 2021-05-14T22:32:36.677422Z {"metadata":{},"code":200} system:serviceaccount:openshift-cluster-version:default clusterrolebindings openshift-cluster-storage-operator csi-snapshot-controller-operator-role 2021-05-14T22:36:18.641265Z {"metadata":{},"code":200} system:serviceaccount:openshift-cluster-version:default clusterrolebindings openshift-cluster-storage-operator csi-snapshot-controller-operator-role 2021-05-14T22:40:33.818630Z {"metadata":{},"code":200} system:serviceaccount:openshift-cluster-version:default clusterrolebindings openshift-cluster-storage-operator csi-snapshot-controller-operator-role 2021-05-14T22:43:46.829640Z {"metadata":{},"code":200} system:serviceaccount:openshift-cluster-version:default clusterrolebindings openshift-cluster-storage-operator csi-snapshot-controller-operator-role 2021-05-14T22:47:05.337502Z {"metadata":{},"code":200} system:serviceaccount:openshift-cluster-version:default clusterrolebindings openshift-cluster-storage-operator csi-snapshot-controller-operator-role 2021-05-14T22:50:24.068091Z {"metadata":{},"code":200} system:serviceaccount:openshift-cluster-version:default clusterrolebindings openshift-cluster-storage-operator csi-snapshot-controller-operator-role 2021-05-14T22:53:42.632322Z {"metadata":{},"code":200} system:serviceaccount:openshift-cluster-version:default clusterrolebindings openshift-cluster-storage-operator csi-snapshot-controller-operator-role 2021-05-14T22:57:01.060066Z {"metadata":{},"code":200} system:serviceaccount:openshift-cluster-version:default clusterrolebindings openshift-cluster-storage-operator csi-snapshot-controller-operator-role 2021-05-14T23:00:19.723207Z {"metadata":{},"code":200} system:serviceaccount:openshift-cluster-version:default clusterrolebindings openshift-cluster-storage-operator csi-snapshot-controller-operator-role 2021-05-14T23:03:38.160554Z {"metadata":{},"code":200} system:serviceaccount:openshift-cluster-version:default clusterrolebindings openshift-cluster-storage-operator csi-snapshot-controller-operator-role 2021-05-14T23:06:56.697264Z {"metadata":{},"code":200} system:serviceaccount:openshift-cluster-version:default clusterrolebindings openshift-cluster-storage-operator csi-snapshot-controller-operator-role 2021-05-14T23:10:15.112039Z {"metadata":{},"code":200} system:serviceaccount:openshift-cluster-version:default rolebindings openshift-machine-api cluster-autoscaler-operator 2021-05-14T22:27:25.282139Z {"metadata":{},"code":200} system:serviceaccount:openshift-cluster-version:default rolebindings openshift-machine-api cluster-autoscaler-operator 2021-05-14T22:29:20.712805Z {"metadata":{},"code":200} system:serviceaccount:openshift-cluster-version:default rolebindings openshift-machine-api cluster-autoscaler-operator 2021-05-14T22:32:31.883281Z {"metadata":{},"code":200} system:serviceaccount:openshift-cluster-version:default rolebindings openshift-machine-api cluster-autoscaler-operator 2021-05-14T22:36:17.391019Z {"metadata":{},"code":200} system:serviceaccount:openshift-cluster-version:default rolebindings openshift-machine-api cluster-autoscaler-operator 2021-05-14T22:40:29.245923Z {"metadata":{},"code":200} system:serviceaccount:openshift-cluster-version:default rolebindings openshift-machine-api cluster-autoscaler-operator 2021-05-14T22:43:33.337697Z {"metadata":{},"code":200} system:serviceaccount:openshift-cluster-version:default rolebindings openshift-machine-api cluster-autoscaler-operator 2021-05-14T22:46:51.961709Z {"metadata":{},"code":200} system:serviceaccount:openshift-cluster-version:default rolebindings openshift-machine-api cluster-autoscaler-operator 2021-05-14T22:50:10.685357Z {"metadata":{},"code":200} system:serviceaccount:openshift-cluster-version:default rolebindings openshift-machine-api cluster-autoscaler-operator 2021-05-14T22:53:29.250745Z {"metadata":{},"code":200} system:serviceaccount:openshift-cluster-version:default rolebindings openshift-machine-api cluster-autoscaler-operator 2021-05-14T22:56:47.672577Z {"metadata":{},"code":200} system:serviceaccount:openshift-cluster-version:default rolebindings openshift-machine-api cluster-autoscaler-operator 2021-05-14T23:00:06.233520Z {"metadata":{},"code":200} system:serviceaccount:openshift-cluster-version:default rolebindings openshift-machine-api cluster-autoscaler-operator 2021-05-14T23:03:24.774968Z {"metadata":{},"code":200} system:serviceaccount:openshift-cluster-version:default rolebindings openshift-machine-api cluster-autoscaler-operator 2021-05-14T23:06:43.315196Z {"metadata":{},"code":200} system:serviceaccount:openshift-cluster-version:default rolebindings openshift-machine-api cluster-autoscaler-operator 2021-05-14T23:10:01.825318Z {"metadata":{},"code":200} system:serviceaccount:openshift-cluster-version:default rolebindings openshift-machine-api cluster-autoscaler-operator 2021-05-14T23:13:20.318977Z {"metadata":{},"code":200} So we still have some work to do here. [1]: https://prow.ci.openshift.org/view/gs/origin-ci-test/pr-logs/pull/openshift_cluster-version-operator/562/pull-ci-openshift-cluster-version-operator-master-e2e-agnostic/1393324758669463552
Right, I still see this happening with verbose logging: I0517 08:45:05.816392 1 rbac.go:37] Updating ClusterRoleBinding csi-snapshot-controller-operator-role due to diff: &v1.ClusterRoleBinding{ TypeMeta: v1.TypeMeta{ - Kind: "", + Kind: "ClusterRoleBinding", - APIVersion: "", + APIVersion: "rbac.authorization.k8s.io/v1", }, ObjectMeta: v1.ObjectMeta{ ... // 2 identical fields Namespace: "openshift-cluster-storage-operator", SelfLink: "", - UID: "b20b7ed4-9b69-4818-9df6-5731b3a90a7f", + UID: "", - ResourceVersion: "1669", + ResourceVersion: "", Generation: 0, - CreationTimestamp: v1.Time{Time: s"2021-05-17 08:32:29 +0000 UTC"}, + CreationTimestamp: v1.Time{}, DeletionTimestamp: nil, DeletionGracePeriodSeconds: nil, ... // 3 identical fields Finalizers: nil, ClusterName: "", - ManagedFields: []v1.ManagedFieldsEntry{ - { - Manager: "cluster-version-operator", - Operation: "Update", - APIVersion: "rbac.authorization.k8s.io/v1", - Time: s"2021-05-17 08:32:29 +0000 UTC", - FieldsType: "FieldsV1", - FieldsV1: s`{"f:metadata":{"f:annotations":{".":{},"f:include.release.opensh`..., - }, - }, + ManagedFields: nil, }, Subjects: {{Kind: "ServiceAccount", Name: "csi-snapshot-controller-operator", Namespace: "openshift-cluster-storage-operator"}}, RoleRef: v1.RoleRef{ - APIGroup: "rbac.authorization.k8s.io", + APIGroup: "", Kind: "ClusterRole", Name: "cluster-admin", }, } however now I'm not sure why
We don't seem to have that ClusterRoleBinding in the must-gather. I've filed bug 1961317 to get it collected going forward.
Ah, they are setting "namespace" for ClusterRoleBinding, which is a cluster-wide resource. Filed https://bugzilla.redhat.com/show_bug.cgi?id=1961538 to track that
We have discussed this bug within the team and it is not a release blocker.
Auditing now that bug 1961538 is ON_QA, assuming that a Cluster kind prefix is a good sign that the resource is cluster-scoped: $ oc adm release extract --to manifests registry.ci.openshift.org/ocp/release:4.8.0-0.nightly-2021-05-18-164623 $ for X in manifests/*.yaml; do yaml2json < "${X}" | jq -r '.[] | select((.kind // "" | startswith("Cluster")) and .metadata.namespace != null) | .kind + " " + (.metadata | .namespace + " " + .name)'; done ClusterRoleBinding openshift-cluster-storage-operator csi-snapshot-controller-operator-role ClusterServiceVersion openshift-operator-lifecycle-manager packageserver ClusterOperator openshift-marketplace marketplace
I've stuffed the CSI fix into the existing bug 1961538 for now. Will get PRs up for the other two under their own bugs...
I've opened [1] for the marketplace ClusterOperator. I don't think the CVO cares about that, though, so no bug for it, and we'll get it into 4.9. And it turns out that ClusterServiceVersion is namespaced [2], so no issues there. Once bug 1961538 gets back to being ON_QA, we should have no ClusterRoleBindings looping left. [1]: https://github.com/operator-framework/operator-marketplace/pull/401 [2]: https://github.com/openshift/operator-framework-olm/blob/b133dce55dd3b0cbedee5cc51fcf73db23c01b29/manifests/0000_50_olm_00-clusterserviceversions.crd.yaml#L23
Bug 1961538 is back to MODIFIED. We'll get swept into ON_QA together.
Attempting to reproduce it with 4.7.11 # oc get clusterversion NAME VERSION AVAILABLE PROGRESSING SINCE STATUS version 4.7.11 True False 3h46m Cluster version is 4.7.11 # oc -n openshift-kube-apiserver get po | grep -i running kube-apiserver-yangyang0520-m2sfl-master-0.c.openshift-qe.internal 5/5 Running 0 3h52m kube-apiserver-yangyang0520-m2sfl-master-1.c.openshift-qe.internal 5/5 Running 0 3h56m kube-apiserver-yangyang0520-m2sfl-master-2.c.openshift-qe.internal 5/5 Running 0 3h49m # oc -n openshift-kube-apiserver rsh kube-apiserver-yangyang0520-m2sfl-master-1.c.openshift-qe.internal sh-4.4# zgrep -h '"verb":"update".*"resource":".*rolebindings"' /var/log/kube-apiserver/audit.log 2>/dev/null | jq -r '.user.username + " " + (.objectRef | .resource + " " + .namespace + " " + .name) + " " + .stageTimestamp + " " + (.responseStatus | tostring)' | grep clusterrolebindings |grep cluster-version | sort system:serviceaccount:openshift-cluster-version:default clusterrolebindings cluster-node-tuning-operator 2021-05-20T07:12:31.840159Z {"metadata":{},"code":200} system:serviceaccount:openshift-cluster-version:default clusterrolebindings cluster-node-tuning-operator 2021-05-20T07:15:50.602433Z {"metadata":{},"code":200} system:serviceaccount:openshift-cluster-version:default clusterrolebindings cluster-node-tuning-operator 2021-05-20T07:19:09.367632Z {"metadata":{},"code":200} system:serviceaccount:openshift-cluster-version:default clusterrolebindings cluster-node-tuning-operator 2021-05-20T07:22:28.238765Z {"metadata":{},"code":200} system:serviceaccount:openshift-cluster-version:default clusterrolebindings cluster-node-tuning:tuned 2021-05-20T07:12:32.240097Z {"metadata":{},"code":200} system:serviceaccount:openshift-cluster-version:default clusterrolebindings cluster-node-tuning:tuned 2021-05-20T07:15:51.002674Z {"metadata":{},"code":200} system:serviceaccount:openshift-cluster-version:default clusterrolebindings cluster-node-tuning:tuned 2021-05-20T07:19:09.767590Z {"metadata":{},"code":200} system:serviceaccount:openshift-cluster-version:default clusterrolebindings cluster-node-tuning:tuned 2021-05-20T07:22:28.636673Z {"metadata":{},"code":200} system:serviceaccount:openshift-cluster-version:default clusterrolebindings cluster-samples-operator 2021-05-20T07:12:31.689667Z {"metadata":{},"code":200} system:serviceaccount:openshift-cluster-version:default clusterrolebindings cluster-samples-operator 2021-05-20T07:15:50.452469Z {"metadata":{},"code":200} system:serviceaccount:openshift-cluster-version:default clusterrolebindings cluster-samples-operator 2021-05-20T07:19:09.216353Z {"metadata":{},"code":200} system:serviceaccount:openshift-cluster-version:default clusterrolebindings cluster-samples-operator 2021-05-20T07:22:27.891297Z {"metadata":{},"code":200} system:serviceaccount:openshift-cluster-version:default clusterrolebindings cluster-samples-operator-proxy-reader 2021-05-20T07:12:30.889335Z {"metadata":{},"code":200} system:serviceaccount:openshift-cluster-version:default clusterrolebindings cluster-samples-operator-proxy-reader 2021-05-20T07:15:49.653945Z {"metadata":{},"code":200} system:serviceaccount:openshift-cluster-version:default clusterrolebindings cluster-samples-operator-proxy-reader 2021-05-20T07:19:08.419344Z {"metadata":{},"code":200} system:serviceaccount:openshift-cluster-version:default clusterrolebindings cluster-samples-operator-proxy-reader 2021-05-20T07:22:27.090689Z {"metadata":{},"code":200} system:serviceaccount:openshift-cluster-version:default clusterrolebindings cluster-version-operator 2021-05-20T07:12:49.239837Z {"metadata":{},"code":200} system:serviceaccount:openshift-cluster-version:default clusterrolebindings cluster-version-operator 2021-05-20T07:16:08.102473Z {"metadata":{},"code":200} system:serviceaccount:openshift-cluster-version:default clusterrolebindings cluster-version-operator 2021-05-20T07:19:26.720125Z {"metadata":{},"code":200} system:serviceaccount:openshift-cluster-version:default clusterrolebindings cluster-version-operator 2021-05-20T07:22:45.488084Z {"metadata":{},"code":200} system:serviceaccount:openshift-cluster-version:default clusterrolebindings insights-operator 2021-05-20T07:12:43.339656Z {"metadata":{},"code":200} system:serviceaccount:openshift-cluster-version:default clusterrolebindings insights-operator 2021-05-20T07:16:02.103008Z {"metadata":{},"code":200} system:serviceaccount:openshift-cluster-version:default clusterrolebindings insights-operator 2021-05-20T07:19:20.866358Z {"metadata":{},"code":200} system:serviceaccount:openshift-cluster-version:default clusterrolebindings insights-operator 2021-05-20T07:22:39.644448Z {"metadata":{},"code":200} system:serviceaccount:openshift-cluster-version:default clusterrolebindings insights-operator-auth 2021-05-20T07:12:42.839945Z {"metadata":{},"code":200} system:serviceaccount:openshift-cluster-version:default clusterrolebindings insights-operator-auth 2021-05-20T07:16:01.604727Z {"metadata":{},"code":200} system:serviceaccount:openshift-cluster-version:default clusterrolebindings insights-operator-auth 2021-05-20T07:19:20.366687Z {"metadata":{},"code":200} system:serviceaccount:openshift-cluster-version:default clusterrolebindings insights-operator-auth 2021-05-20T07:22:39.137288Z {"metadata":{},"code":200} system:serviceaccount:openshift-cluster-version:default clusterrolebindings insights-operator-gather 2021-05-20T07:12:43.640219Z {"metadata":{},"code":200} system:serviceaccount:openshift-cluster-version:default clusterrolebindings insights-operator-gather 2021-05-20T07:16:02.402516Z {"metadata":{},"code":200} system:serviceaccount:openshift-cluster-version:default clusterrolebindings insights-operator-gather 2021-05-20T07:19:21.167922Z {"metadata":{},"code":200} system:serviceaccount:openshift-cluster-version:default clusterrolebindings insights-operator-gather 2021-05-20T07:22:39.941207Z {"metadata":{},"code":200} system:serviceaccount:openshift-cluster-version:default clusterrolebindings insights-operator-gather-reader 2021-05-20T07:12:43.838750Z {"metadata":{},"code":200} system:serviceaccount:openshift-cluster-version:default clusterrolebindings insights-operator-gather-reader 2021-05-20T07:16:02.605045Z {"metadata":{},"code":200} system:serviceaccount:openshift-cluster-version:default clusterrolebindings insights-operator-gather-reader 2021-05-20T07:19:21.366886Z {"metadata":{},"code":200} system:serviceaccount:openshift-cluster-version:default clusterrolebindings insights-operator-gather-reader 2021-05-20T07:22:40.093956Z {"metadata":{},"code":200} system:serviceaccount:openshift-cluster-version:default clusterrolebindings system:openshift:operator:authentication 2021-05-20T07:12:36.388850Z {"metadata":{},"code":200} system:serviceaccount:openshift-cluster-version:default clusterrolebindings system:openshift:operator:authentication 2021-05-20T07:15:55.152671Z {"metadata":{},"code":200} system:serviceaccount:openshift-cluster-version:default clusterrolebindings system:openshift:operator:authentication 2021-05-20T07:19:13.916522Z {"metadata":{},"code":200} system:serviceaccount:openshift-cluster-version:default clusterrolebindings system:openshift:operator:authentication 2021-05-20T07:22:32.440589Z {"metadata":{},"code":200} system:serviceaccount:openshift-cluster-version:default clusterrolebindings system:openshift:operator:cluster-kube-scheduler-operator 2021-05-20T07:12:40.992219Z {"metadata":{},"code":200} system:serviceaccount:openshift-cluster-version:default clusterrolebindings system:openshift:operator:cluster-kube-scheduler-operator 2021-05-20T07:15:59.752343Z {"metadata":{},"code":200} system:serviceaccount:openshift-cluster-version:default clusterrolebindings system:openshift:operator:cluster-kube-scheduler-operator 2021-05-20T07:19:18.518366Z {"metadata":{},"code":200} system:serviceaccount:openshift-cluster-version:default clusterrolebindings system:openshift:operator:cluster-kube-scheduler-operator 2021-05-20T07:22:37.287259Z {"metadata":{},"code":200} system:serviceaccount:openshift-cluster-version:default clusterrolebindings system:openshift:operator:etcd-operator 2021-05-20T07:12:33.691237Z {"metadata":{},"code":200} system:serviceaccount:openshift-cluster-version:default clusterrolebindings system:openshift:operator:etcd-operator 2021-05-20T07:15:52.452665Z {"metadata":{},"code":200} system:serviceaccount:openshift-cluster-version:default clusterrolebindings system:openshift:operator:etcd-operator 2021-05-20T07:19:11.215971Z {"metadata":{},"code":200} system:serviceaccount:openshift-cluster-version:default clusterrolebindings system:openshift:operator:etcd-operator 2021-05-20T07:22:29.790855Z {"metadata":{},"code":200} system:serviceaccount:openshift-cluster-version:default clusterrolebindings system:openshift:operator:kube-apiserver-operator 2021-05-20T07:12:48.390882Z {"metadata":{},"code":200} system:serviceaccount:openshift-cluster-version:default clusterrolebindings system:openshift:operator:kube-apiserver-operator 2021-05-20T07:16:07.202177Z {"metadata":{},"code":200} system:serviceaccount:openshift-cluster-version:default clusterrolebindings system:openshift:operator:kube-apiserver-operator 2021-05-20T07:19:26.075248Z {"metadata":{},"code":200} system:serviceaccount:openshift-cluster-version:default clusterrolebindings system:openshift:operator:kube-apiserver-operator 2021-05-20T07:22:44.640873Z {"metadata":{},"code":200} system:serviceaccount:openshift-cluster-version:default clusterrolebindings system:openshift:operator:kube-controller-manager-operator 2021-05-20T07:12:39.539720Z {"metadata":{},"code":200} system:serviceaccount:openshift-cluster-version:default clusterrolebindings system:openshift:operator:kube-controller-manager-operator 2021-05-20T07:15:58.302587Z {"metadata":{},"code":200} system:serviceaccount:openshift-cluster-version:default clusterrolebindings system:openshift:operator:kube-controller-manager-operator 2021-05-20T07:19:17.117010Z {"metadata":{},"code":200} system:serviceaccount:openshift-cluster-version:default clusterrolebindings system:openshift:operator:kube-controller-manager-operator 2021-05-20T07:22:35.839730Z {"metadata":{},"code":200} system:serviceaccount:openshift-cluster-version:default clusterrolebindings system:openshift:operator:kube-storage-version-migrator-operator 2021-05-20T07:12:42.888943Z {"metadata":{},"code":200} system:serviceaccount:openshift-cluster-version:default clusterrolebindings system:openshift:operator:kube-storage-version-migrator-operator 2021-05-20T07:16:01.652914Z {"metadata":{},"code":200} system:serviceaccount:openshift-cluster-version:default clusterrolebindings system:openshift:operator:kube-storage-version-migrator-operator 2021-05-20T07:19:20.416263Z {"metadata":{},"code":200} system:serviceaccount:openshift-cluster-version:default clusterrolebindings system:openshift:operator:kube-storage-version-migrator-operator 2021-05-20T07:22:39.189026Z {"metadata":{},"code":200} system:serviceaccount:openshift-cluster-version:default clusterrolebindings system:openshift:operator:openshift-apiserver-operator 2021-05-20T07:12:37.492034Z {"metadata":{},"code":200} system:serviceaccount:openshift-cluster-version:default clusterrolebindings system:openshift:operator:openshift-apiserver-operator 2021-05-20T07:15:56.253500Z {"metadata":{},"code":200} system:serviceaccount:openshift-cluster-version:default clusterrolebindings system:openshift:operator:openshift-apiserver-operator 2021-05-20T07:19:15.018106Z {"metadata":{},"code":200} system:serviceaccount:openshift-cluster-version:default clusterrolebindings system:openshift:operator:openshift-apiserver-operator 2021-05-20T07:22:33.738843Z {"metadata":{},"code":200} system:serviceaccount:openshift-cluster-version:default clusterrolebindings system:openshift:operator:openshift-config-operator 2021-05-20T07:12:30.181099Z {"metadata":{},"code":200} system:serviceaccount:openshift-cluster-version:default clusterrolebindings system:openshift:operator:openshift-config-operator 2021-05-20T07:15:48.955865Z {"metadata":{},"code":200} system:serviceaccount:openshift-cluster-version:default clusterrolebindings system:openshift:operator:openshift-config-operator 2021-05-20T07:19:07.714165Z {"metadata":{},"code":200} system:serviceaccount:openshift-cluster-version:default clusterrolebindings system:openshift:operator:openshift-config-operator 2021-05-20T07:22:26.500276Z {"metadata":{},"code":200} system:serviceaccount:openshift-cluster-version:default clusterrolebindings system:openshift:operator:openshift-controller-manager-operator 2021-05-20T07:12:42.040040Z {"metadata":{},"code":200} system:serviceaccount:openshift-cluster-version:default clusterrolebindings system:openshift:operator:openshift-controller-manager-operator 2021-05-20T07:16:00.801841Z {"metadata":{},"code":200} system:serviceaccount:openshift-cluster-version:default clusterrolebindings system:openshift:operator:openshift-controller-manager-operator 2021-05-20T07:19:19.567865Z {"metadata":{},"code":200} system:serviceaccount:openshift-cluster-version:default clusterrolebindings system:openshift:operator:openshift-controller-manager-operator 2021-05-20T07:22:38.339100Z {"metadata":{},"code":200} system:serviceaccount:openshift-cluster-version:default clusterrolebindings system:openshift:operator:service-ca-operator 2021-05-20T07:12:29.851141Z {"metadata":{},"code":200} system:serviceaccount:openshift-cluster-version:default clusterrolebindings system:openshift:operator:service-ca-operator 2021-05-20T07:15:48.617437Z {"metadata":{},"code":200} system:serviceaccount:openshift-cluster-version:default clusterrolebindings system:openshift:operator:service-ca-operator 2021-05-20T07:19:07.395002Z {"metadata":{},"code":200} system:serviceaccount:openshift-cluster-version:default clusterrolebindings system:openshift:operator:service-ca-operator 2021-05-20T07:22:26.169031Z {"metadata":{},"code":200} system:serviceaccount:openshift-cluster-version:default clusterrolebindings openshift-cluster-storage-operator cluster-storage-operator-role 2021-05-20T07:12:49.389950Z {"metadata":{},"code":200} system:serviceaccount:openshift-cluster-version:default clusterrolebindings openshift-cluster-storage-operator cluster-storage-operator-role 2021-05-20T07:16:08.052554Z {"metadata":{},"code":200} system:serviceaccount:openshift-cluster-version:default clusterrolebindings openshift-cluster-storage-operator cluster-storage-operator-role 2021-05-20T07:19:26.973179Z {"metadata":{},"code":200} system:serviceaccount:openshift-cluster-version:default clusterrolebindings openshift-cluster-storage-operator cluster-storage-operator-role 2021-05-20T07:22:45.738823Z {"metadata":{},"code":200} system:serviceaccount:openshift-cluster-version:default clusterrolebindings openshift-cluster-storage-operator csi-snapshot-controller-operator-role 2021-05-20T07:12:33.539709Z {"metadata":{},"code":200} system:serviceaccount:openshift-cluster-version:default clusterrolebindings openshift-cluster-storage-operator csi-snapshot-controller-operator-role 2021-05-20T07:15:52.303728Z {"metadata":{},"code":200} system:serviceaccount:openshift-cluster-version:default clusterrolebindings openshift-cluster-storage-operator csi-snapshot-controller-operator-role 2021-05-20T07:19:11.067122Z {"metadata":{},"code":200} system:serviceaccount:openshift-cluster-version:default clusterrolebindings openshift-cluster-storage-operator csi-snapshot-controller-operator-role 2021-05-20T07:22:30.038899Z {"metadata":{},"code":200} sh-4.4# Seems like there are several loops on above clusterrolebindings.
Attempting to verify it with 4.8.0-0.nightly-2021-05-19-123944 # oc get clusterversion NAME VERSION AVAILABLE PROGRESSING SINCE STATUS version 4.8.0-0.nightly-2021-05-19-123944 True False 3h45m Cluster version is 4.8.0-0.nightly-2021-05-19-123944 # oc -n openshift-kube-apiserver rsh kube-apiserver-yangyang0520-1-dvx2j-master-0.c.openshift-qe.internal sh-4.4# zgrep -h '"verb":"update".*"resource":".*rolebindings"' /var/log/kube-apiserver/audit.log 2>/dev/null | jq -r '.user.username + " " + (.objectRef | .resource + " " + .namespace + " " + .name + " " + .apiGroup) + " " + .stageTimestamp + " " + (.responseStatus | tostring)' | grep clusterrolebindings |grep cluster-version | sort system:serviceaccount:openshift-cluster-version:default clusterrolebindings openshift-cluster-storage-operator csi-snapshot-controller-operator-role rbac.authorization.k8s.io 2021-05-20T07:31:28.827878Z {"metadata":{},"code":200} system:serviceaccount:openshift-cluster-version:default clusterrolebindings openshift-cluster-storage-operator csi-snapshot-controller-operator-role rbac.authorization.k8s.io 2021-05-20T07:34:46.249775Z {"metadata":{},"code":200} system:serviceaccount:openshift-cluster-version:default clusterrolebindings openshift-cluster-storage-operator csi-snapshot-controller-operator-role rbac.authorization.k8s.io 2021-05-20T07:38:03.770977Z {"metadata":{},"code":200} system:serviceaccount:openshift-cluster-version:default clusterrolebindings openshift-cluster-storage-operator csi-snapshot-controller-operator-role rbac.authorization.k8s.io 2021-05-20T07:41:21.384696Z {"metadata":{},"code":200} system:serviceaccount:openshift-cluster-version:default clusterrolebindings openshift-cluster-storage-operator csi-snapshot-controller-operator-role rbac.authorization.k8s.io 2021-05-20T07:44:38.699337Z {"metadata":{},"code":200} system:serviceaccount:openshift-cluster-version:default clusterrolebindings openshift-cluster-storage-operator csi-snapshot-controller-operator-role rbac.authorization.k8s.io 2021-05-20T07:47:56.218493Z {"metadata":{},"code":200} system:serviceaccount:openshift-cluster-version:default clusterrolebindings openshift-cluster-storage-operator csi-snapshot-controller-operator-role rbac.authorization.k8s.io 2021-05-20T07:51:13.685024Z {"metadata":{},"code":200} system:serviceaccount:openshift-cluster-version:default clusterrolebindings openshift-cluster-storage-operator csi-snapshot-controller-operator-role rbac.authorization.k8s.io 2021-05-20T07:54:31.207362Z {"metadata":{},"code":200} system:serviceaccount:openshift-cluster-version:default clusterrolebindings openshift-cluster-storage-operator csi-snapshot-controller-operator-role rbac.authorization.k8s.io 2021-05-20T07:57:48.618923Z {"metadata":{},"code":200} system:serviceaccount:openshift-cluster-version:default clusterrolebindings openshift-cluster-storage-operator csi-snapshot-controller-operator-role rbac.authorization.k8s.io 2021-05-20T08:01:06.088497Z {"metadata":{},"code":200} Loops only appears on csi-snapshot-controller-operator-role clusterrolebindings. Seems the latest nightly build does not include the PR [1]. Pending the available payload to verify it. [1] https://github.com/openshift/cluster-csi-snapshot-controller-operator/pull/87
The fix should be included in https://amd64.ocp.releases.ci.openshift.org/releasestream/4.8.0-0.nightly/release/4.8.0-0.nightly-2021-05-21-101954, could you give it a try?
Verifying with 4.8.0-0.nightly-2021-05-21-233425 # oc get clusterversion NAME VERSION AVAILABLE PROGRESSING SINCE STATUS version 4.8.0-0.nightly-2021-05-21-233425 True False 16m Cluster version is 4.8.0-0.nightly-2021-05-21-233425 sh-4.4# zgrep -h '"verb":"update".*"resource":".*rolebindings"' /var/log/kube-apiserver/audit.log 2>/dev/null | jq -r '.user.username + " " + (.objectRef | .resource + " " + .namespace + " " + .name + " " + .apiGroup) + " " + .stageTimestamp + " " + (.responseStatus | tostring)' | grep clusterrolebindings |grep cluster-version | sort null No clusterrolebindings loops found on user serviceaccount:openshift-cluster-version.
sh-4.4# zgrep -h '"verb":"update".*"resource":".*rolebindings"' /var/log/kube-apiserver/audit.log 2>/dev/null | jq -r '.user.username + " " + (.objectRef | .resource + " " + .namespace + " " + .name + " " + .apiGroup) + " " + .stageTimestamp + " " + (.responseStatus | tostring)' | grep clusterrolebindings | sort system:serviceaccount:openshift-machine-config-operator:default clusterrolebindings openshift-machine-config-operator machine-config-controller rbac.authorization.k8s.io 2021-05-24T02:31:17.029873Z {"metadata":{},"code":200} system:serviceaccount:openshift-machine-config-operator:default clusterrolebindings openshift-machine-config-operator machine-config-controller rbac.authorization.k8s.io 2021-05-24T02:31:39.146384Z {"metadata":{},"code":200} system:serviceaccount:openshift-machine-config-operator:default clusterrolebindings openshift-machine-config-operator machine-config-controller rbac.authorization.k8s.io 2021-05-24T02:36:14.043833Z {"metadata":{},"code":200} system:serviceaccount:openshift-machine-config-operator:default clusterrolebindings openshift-machine-config-operator machine-config-controller rbac.authorization.k8s.io 2021-05-24T02:36:51.699084Z {"metadata":{},"code":200} system:serviceaccount:openshift-machine-config-operator:default clusterrolebindings openshift-machine-config-operator machine-config-controller rbac.authorization.k8s.io 2021-05-24T02:36:58.611182Z {"metadata":{},"code":200} system:serviceaccount:openshift-machine-config-operator:default clusterrolebindings openshift-machine-config-operator machine-config-controller rbac.authorization.k8s.io 2021-05-24T02:39:51.322017Z {"metadata":{},"code":200} system:serviceaccount:openshift-machine-config-operator:default clusterrolebindings openshift-machine-config-operator machine-config-controller rbac.authorization.k8s.io 2021-05-24T02:39:58.241733Z {"metadata":{},"code":200} system:serviceaccount:openshift-machine-config-operator:default clusterrolebindings openshift-machine-config-operator machine-config-controller rbac.authorization.k8s.io 2021-05-24T02:42:04.284510Z {"metadata":{},"code":200} system:serviceaccount:openshift-machine-config-operator:default clusterrolebindings openshift-machine-config-operator machine-config-controller rbac.authorization.k8s.io 2021-05-24T02:42:11.231712Z {"metadata":{},"code":200} system:serviceaccount:openshift-machine-config-operator:default clusterrolebindings openshift-machine-config-operator machine-config-controller rbac.authorization.k8s.io 2021-05-24T02:43:11.100766Z {"metadata":{},"code":200} system:serviceaccount:openshift-machine-config-operator:default clusterrolebindings openshift-machine-config-operator machine-config-controller rbac.authorization.k8s.io 2021-05-24T02:43:18.008308Z {"metadata":{},"code":200} system:serviceaccount:openshift-machine-config-operator:default clusterrolebindings openshift-machine-config-operator machine-config-controller rbac.authorization.k8s.io 2021-05-24T02:45:03.926262Z {"metadata":{},"code":200} system:serviceaccount:openshift-machine-config-operator:default clusterrolebindings openshift-machine-config-operator machine-config-controller rbac.authorization.k8s.io 2021-05-24T02:47:16.817830Z {"metadata":{},"code":200} system:serviceaccount:openshift-machine-config-operator:default clusterrolebindings openshift-machine-config-operator machine-config-controller rbac.authorization.k8s.io 2021-05-24T02:47:23.735631Z {"metadata":{},"code":200} system:serviceaccount:openshift-machine-config-operator:default clusterrolebindings openshift-machine-config-operator machine-config-daemon rbac.authorization.k8s.io 2021-05-24T02:31:15.079134Z {"metadata":{},"code":200} system:serviceaccount:openshift-machine-config-operator:default clusterrolebindings openshift-machine-config-operator machine-config-daemon rbac.authorization.k8s.io 2021-05-24T02:31:37.187222Z {"metadata":{},"code":200} system:serviceaccount:openshift-machine-config-operator:default clusterrolebindings openshift-machine-config-operator machine-config-daemon rbac.authorization.k8s.io 2021-05-24T02:36:12.079546Z {"metadata":{},"code":200} system:serviceaccount:openshift-machine-config-operator:default clusterrolebindings openshift-machine-config-operator machine-config-daemon rbac.authorization.k8s.io 2021-05-24T02:36:49.750957Z {"metadata":{},"code":200} system:serviceaccount:openshift-machine-config-operator:default clusterrolebindings openshift-machine-config-operator machine-config-daemon rbac.authorization.k8s.io 2021-05-24T02:36:56.646920Z {"metadata":{},"code":200} system:serviceaccount:openshift-machine-config-operator:default clusterrolebindings openshift-machine-config-operator machine-config-daemon rbac.authorization.k8s.io 2021-05-24T02:39:49.358183Z {"metadata":{},"code":200} system:serviceaccount:openshift-machine-config-operator:default clusterrolebindings openshift-machine-config-operator machine-config-daemon rbac.authorization.k8s.io 2021-05-24T02:39:56.277073Z {"metadata":{},"code":200} system:serviceaccount:openshift-machine-config-operator:default clusterrolebindings openshift-machine-config-operator machine-config-daemon rbac.authorization.k8s.io 2021-05-24T02:42:02.324230Z {"metadata":{},"code":200} system:serviceaccount:openshift-machine-config-operator:default clusterrolebindings openshift-machine-config-operator machine-config-daemon rbac.authorization.k8s.io 2021-05-24T02:42:09.267678Z {"metadata":{},"code":200} system:serviceaccount:openshift-machine-config-operator:default clusterrolebindings openshift-machine-config-operator machine-config-daemon rbac.authorization.k8s.io 2021-05-24T02:43:09.138121Z {"metadata":{},"code":200} system:serviceaccount:openshift-machine-config-operator:default clusterrolebindings openshift-machine-config-operator machine-config-daemon rbac.authorization.k8s.io 2021-05-24T02:43:16.044792Z {"metadata":{},"code":200} system:serviceaccount:openshift-machine-config-operator:default clusterrolebindings openshift-machine-config-operator machine-config-daemon rbac.authorization.k8s.io 2021-05-24T02:45:01.964753Z {"metadata":{},"code":200} system:serviceaccount:openshift-machine-config-operator:default clusterrolebindings openshift-machine-config-operator machine-config-daemon rbac.authorization.k8s.io 2021-05-24T02:47:14.874808Z {"metadata":{},"code":200} system:serviceaccount:openshift-machine-config-operator:default clusterrolebindings openshift-machine-config-operator machine-config-daemon rbac.authorization.k8s.io 2021-05-24T02:47:21.773127Z {"metadata":{},"code":200} system:serviceaccount:openshift-machine-config-operator:default clusterrolebindings openshift-machine-config-operator machine-config-server rbac.authorization.k8s.io 2021-05-24T02:31:19.088481Z {"metadata":{},"code":200} system:serviceaccount:openshift-machine-config-operator:default clusterrolebindings openshift-machine-config-operator machine-config-server rbac.authorization.k8s.io 2021-05-24T02:31:41.203659Z {"metadata":{},"code":200} system:serviceaccount:openshift-machine-config-operator:default clusterrolebindings openshift-machine-config-operator machine-config-server rbac.authorization.k8s.io 2021-05-24T02:36:16.105600Z {"metadata":{},"code":200} system:serviceaccount:openshift-machine-config-operator:default clusterrolebindings openshift-machine-config-operator machine-config-server rbac.authorization.k8s.io 2021-05-24T02:36:53.756207Z {"metadata":{},"code":200} system:serviceaccount:openshift-machine-config-operator:default clusterrolebindings openshift-machine-config-operator machine-config-server rbac.authorization.k8s.io 2021-05-24T02:37:00.670284Z {"metadata":{},"code":200} system:serviceaccount:openshift-machine-config-operator:default clusterrolebindings openshift-machine-config-operator machine-config-server rbac.authorization.k8s.io 2021-05-24T02:39:53.383299Z {"metadata":{},"code":200} system:serviceaccount:openshift-machine-config-operator:default clusterrolebindings openshift-machine-config-operator machine-config-server rbac.authorization.k8s.io 2021-05-24T02:40:00.297370Z {"metadata":{},"code":200} system:serviceaccount:openshift-machine-config-operator:default clusterrolebindings openshift-machine-config-operator machine-config-server rbac.authorization.k8s.io 2021-05-24T02:42:06.361900Z {"metadata":{},"code":200} system:serviceaccount:openshift-machine-config-operator:default clusterrolebindings openshift-machine-config-operator machine-config-server rbac.authorization.k8s.io 2021-05-24T02:42:13.289405Z {"metadata":{},"code":200} system:serviceaccount:openshift-machine-config-operator:default clusterrolebindings openshift-machine-config-operator machine-config-server rbac.authorization.k8s.io 2021-05-24T02:43:13.158302Z {"metadata":{},"code":200} system:serviceaccount:openshift-machine-config-operator:default clusterrolebindings openshift-machine-config-operator machine-config-server rbac.authorization.k8s.io 2021-05-24T02:43:20.067081Z {"metadata":{},"code":200} system:serviceaccount:openshift-machine-config-operator:default clusterrolebindings openshift-machine-config-operator machine-config-server rbac.authorization.k8s.io 2021-05-24T02:45:05.982635Z {"metadata":{},"code":200} system:serviceaccount:openshift-machine-config-operator:default clusterrolebindings openshift-machine-config-operator machine-config-server rbac.authorization.k8s.io 2021-05-24T02:47:18.883512Z {"metadata":{},"code":200} system:serviceaccount:openshift-machine-config-operator:default clusterrolebindings openshift-machine-config-operator machine-config-server rbac.authorization.k8s.io 2021-05-24T02:47:26.209500Z {"metadata":{},"code":200} system:serviceaccount:openshift-network-operator:default clusterrolebindings metrics-daemon-sa-rolebinding rbac.authorization.k8s.io 2021-05-24T02:32:22.656259Z {"metadata":{},"code":200} system:serviceaccount:openshift-network-operator:default clusterrolebindings metrics-daemon-sa-rolebinding rbac.authorization.k8s.io 2021-05-24T02:35:23.615317Z {"metadata":{},"code":200} system:serviceaccount:openshift-network-operator:default clusterrolebindings metrics-daemon-sa-rolebinding rbac.authorization.k8s.io 2021-05-24T02:38:24.685856Z {"metadata":{},"code":200} system:serviceaccount:openshift-network-operator:default clusterrolebindings metrics-daemon-sa-rolebinding rbac.authorization.k8s.io 2021-05-24T02:41:25.752633Z {"metadata":{},"code":200} system:serviceaccount:openshift-network-operator:default clusterrolebindings metrics-daemon-sa-rolebinding rbac.authorization.k8s.io 2021-05-24T02:44:26.837924Z {"metadata":{},"code":200} system:serviceaccount:openshift-network-operator:default clusterrolebindings metrics-daemon-sa-rolebinding rbac.authorization.k8s.io 2021-05-24T02:47:27.895516Z {"metadata":{},"code":200} system:serviceaccount:openshift-network-operator:default clusterrolebindings metrics-daemon-sa-rolebinding rbac.authorization.k8s.io 2021-05-24T02:50:28.965264Z {"metadata":{},"code":200} Vadim, with latest build, loops appear on above clusterrolebindings. Is it something related to CVO? Does it need to be fixed?
We have a test PR [1] to show diff why objects are being updated - and in the log [2] I don't see CRB hotlooping anymore Seems the originator - first column - is MCO/Network operator SAs, so it should be a new bug filed against these components [1] https://github.com/openshift/cluster-version-operator/pull/561 [2] https://gcsweb-ci.apps.ci.l2s4.p1.openshiftapps.com/gcs/origin-ci-test/pr-logs/pull/openshift_cluster-version-operator/561/pull-ci-openshift-cluster-version-operator-master-e2e-agnostic/1396748541996568576/artifacts/e2e-agnostic/gather-extra/artifacts/pods/openshift-cluster-version_cluster-version-operator-86b9cd5d6b-vrrpl_cluster-version-operator.log
Based on comment#27, there are no hotloops against CVO SA. Moving it to verified.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Moderate: OpenShift Container Platform 4.8.2 bug fix and security update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2021:2438