1881484 – CVO hotloops on deployment manifests

Bug 1881484 - CVO hotloops on deployment manifests

Summary: CVO hotloops on deployment manifests

Keywords:
Status:	CLOSED ERRATA
Alias:	None
Product:	OpenShift Container Platform
Classification:	Red Hat
Component:	Cluster Version Operator
Sub Component:
Version:	4.6
Hardware:	Unspecified
OS:	Unspecified
Priority:	medium
Severity:	medium
Target Milestone:	---
Target Release:	4.8.0
Assignee:	Jack Ottofaro
QA Contact:	Yang Yang
Docs Contact:
URL:
Whiteboard:
Duplicates (1):	1964003 (view as bug list)
Depends On:
Blocks:
TreeView+	depends on / blocked

Reported:	2020-09-22 13:58 UTC by Stefan Schimanski
Modified:	2021-07-27 22:33 UTC (History)
CC List:	7 users (show)
Fixed In Version:
Doc Type:	No Doc Update
Doc Text:
Clone Of:
Environment:
Last Closed:	2021-07-27 22:33:27 UTC
Target Upstream Version:
Embargoed:

Attachments	(Terms of Use)

Links
System	ID	Private	Priority	Status	Summary	Last Updated
Github	openshift cluster-version-operator pull 559	0	None	open	Bug 1881484: Set defaults in deployment	2021-05-27 18:58:37 UTC
Red Hat Product Errata	RHSA-2021:2438	0	None	None	None	2021-07-27 22:33:45 UTC

Description Stefan Schimanski 2020-09-22 13:58:28 UTC

For unknown reason (probably missing defaulting logic in the apply funcs), CVO continuously updates deployments, giving deployment controllers unnecessary work to do:

{"count":74,"path":"/apis/apps/v1/namespaces/openshift-apiserver-operator/deployments/openshift-apiserver-operator"}
{"count":76,"path":"/apis/apps/v1/namespaces/openshift-authentication-operator/deployments/authentication-operator"}
{"count":76,"path":"/apis/apps/v1/namespaces/openshift-cloud-credential-operator/deployments/cloud-credential-operator"}
{"count":76,"path":"/apis/apps/v1/namespaces/openshift-cluster-machine-approver/deployments/machine-approver"}
{"count":76,"path":"/apis/apps/v1/namespaces/openshift-cluster-node-tuning-operator/deployments/cluster-node-tuning-operator"}
{"count":76,"path":"/apis/apps/v1/namespaces/openshift-cluster-samples-operator/deployments/cluster-samples-operator"}
{"count":77,"path":"/apis/apps/v1/namespaces/openshift-cluster-storage-operator/deployments/cluster-storage-operator"}
{"count":76,"path":"/apis/apps/v1/namespaces/openshift-cluster-storage-operator/deployments/csi-snapshot-controller-operator"}
{"count":76,"path":"/apis/apps/v1/namespaces/openshift-cluster-version/deployments/cluster-version-operator"}
{"count":78,"path":"/apis/apps/v1/namespaces/openshift-config-operator/deployments/openshift-config-operator"}
{"count":77,"path":"/apis/apps/v1/namespaces/openshift-console-operator/deployments/console-operator"}
{"count":77,"path":"/apis/apps/v1/namespaces/openshift-console/deployments/downloads"}
{"count":75,"path":"/apis/apps/v1/namespaces/openshift-controller-manager-operator/deployments/openshift-controller-manager-operator"}
{"count":77,"path":"/apis/apps/v1/namespaces/openshift-dns-operator/deployments/dns-operator"}
{"count":77,"path":"/apis/apps/v1/namespaces/openshift-etcd-operator/deployments/etcd-operator"}
{"count":77,"path":"/apis/apps/v1/namespaces/openshift-etcd/deployments/etcd-quorum-guard"}
{"count":75,"path":"/apis/apps/v1/namespaces/openshift-image-registry/deployments/cluster-image-registry-operator"}
{"count":76,"path":"/apis/apps/v1/namespaces/openshift-ingress-operator/deployments/ingress-operator"}
{"count":78,"path":"/apis/apps/v1/namespaces/openshift-insights/deployments/insights-operator"}
{"count":76,"path":"/apis/apps/v1/namespaces/openshift-kube-apiserver-operator/deployments/kube-apiserver-operator"}
{"count":75,"path":"/apis/apps/v1/namespaces/openshift-kube-controller-manager-operator/deployments/kube-controller-manager-operator"}
{"count":77,"path":"/apis/apps/v1/namespaces/openshift-kube-scheduler-operator/deployments/openshift-kube-scheduler-operator"}
{"count":76,"path":"/apis/apps/v1/namespaces/openshift-kube-storage-version-migrator-operator/deployments/kube-storage-version-migrator-operator"}
{"count":75,"path":"/apis/apps/v1/namespaces/openshift-machine-api/deployments/cluster-autoscaler-operator"}
{"count":75,"path":"/apis/apps/v1/namespaces/openshift-machine-api/deployments/machine-api-operator"}
{"count":78,"path":"/apis/apps/v1/namespaces/openshift-machine-config-operator/deployments/machine-config-operator"}
{"count":77,"path":"/apis/apps/v1/namespaces/openshift-marketplace/deployments/marketplace-operator"}
{"count":76,"path":"/apis/apps/v1/namespaces/openshift-monitoring/deployments/cluster-monitoring-operator"}
{"count":77,"path":"/apis/apps/v1/namespaces/openshift-network-operator/deployments/network-operator"}
{"count":75,"path":"/apis/apps/v1/namespaces/openshift-operator-lifecycle-manager/deployments/catalog-operator"}
{"count":75,"path":"/apis/apps/v1/namespaces/openshift-operator-lifecycle-manager/deployments/olm-operator"}

Comment 1 W. Trevor King 2020-09-22 21:36:29 UTC

Stefan suggests possibly waiting until API-server support for server-side apply [1] goes GA and rerolling the CVO's apply logic to use that instead of client-side merging, which might help here.  And bug 1879184 might end up with a [Late] CI guard based on the audit logs.  But whatever is going on here is unlikely to be new in 4.6, so punting to 4.7.

[1]: https://kubernetes.io/blog/2020/04/01/kubernetes-1.18-feature-server-side-apply-beta-2/

Comment 2 Stefan Schimanski 2020-09-23 13:12:52 UTC

I don't think #c1 reflect what I meant. I meant that it is an infinite game with lots of chance for mistakes and failure in writing the perfect client-side merging funcs for all types. Instead the right solution is to triage these bugs, fix the manifests for now and add an e2e test that uncovers the issues before new manifests merge.

Comment 3 Stefan Schimanski 2020-09-23 13:21:50 UTC

I double checked whether the objects change and in the samples I looked at they don't. So I think this is pure load-issue without consequences other than stressing the API server and etcd.

Comment 5 W. Trevor King 2020-09-23 16:57:32 UTC

> ... fix the manifests for now...

If that's what this bug is about, it should be assigned to the samples team, right?

> ... and add an e2e test that uncovers the issues before new manifests merge.

This is bug 1879184, right?

Punting back to 4.7, because I don't see any new-in-4.6 regressions here, and it's really late in the 4.6 cycle to make new 4.6 blockers unless we have a solid story around why this is a critical issue.

Comment 6 W. Trevor King 2020-10-02 23:12:06 UTC

It's end of sprint, and this is not going to get fixed in the next few hours.  Hopefully we will at least get the Late audit guard from bug 1879184 in next sprint, and then we'll see which team should fix this issue.

Comment 7 Jack Ottofaro 2020-10-23 19:00:48 UTC

Adding UpcomingSprint as we have reached the end of the current sprint and pushing this bug to the next sprint.

Comment 8 Jack Ottofaro 2021-05-24 16:46:05 UTC

*** Bug 1964003 has been marked as a duplicate of this bug. ***

Comment 10 Yang Yang 2021-06-07 05:43:42 UTC

Reproducing with 4.8.0-fc.3

# oc get clusterversion
NAME      VERSION      AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.8.0-fc.3   True        False         15m     Cluster version is 4.8.0-fc.3

# masters=$(oc get no -l node-role.kubernetes.io/master | sed '1d' | awk '{print $1}')

# oc adm node-logs $masters --path=kube-apiserver/audit.log --raw | zgrep -h '"verb":"update".*"resource":".*deployments"' 2>/dev/null | jq -r '.user.username + " " + (.objectRef | .resource + " " + .namespace + " " + .name + " " + .apiGroup) + " " + .stageTimestamp + " " + (.responseStatus | tostring)' | grep deployments | grep 'serviceaccount:openshift-cluster-version' | sort
system:serviceaccount:openshift-cluster-version:default deployments openshift-apiserver-operator openshift-apiserver-operator apps 2021-06-07T03:45:24.276038Z {"metadata":{},"code":200}
system:serviceaccount:openshift-cluster-version:default deployments openshift-apiserver-operator openshift-apiserver-operator apps 2021-06-07T05:31:31.851728Z {"metadata":{},"code":200}
system:serviceaccount:openshift-cluster-version:default deployments openshift-apiserver-operator openshift-apiserver-operator apps 2021-06-07T05:34:50.970118Z {"metadata":{},"code":200}
system:serviceaccount:openshift-cluster-version:default deployments openshift-apiserver-operator openshift-apiserver-operator apps 2021-06-07T05:38:09.900916Z {"metadata":{},"code":200}
system:serviceaccount:openshift-cluster-version:default deployments openshift-apiserver-operator openshift-apiserver-operator apps 2021-06-07T05:41:28.917213Z {"metadata":{},"code":200}
system:serviceaccount:openshift-cluster-version:default deployments openshift-authentication-operator authentication-operator apps 2021-06-07T03:45:25.601754Z {"metadata":{},"code":200}
system:serviceaccount:openshift-cluster-version:default deployments openshift-authentication-operator authentication-operator apps 2021-06-07T05:31:15.102163Z {"metadata":{},"code":200}
system:serviceaccount:openshift-cluster-version:default deployments openshift-authentication-operator authentication-operator apps 2021-06-07T05:34:34.198281Z {"metadata":{},"code":200}
system:serviceaccount:openshift-cluster-version:default deployments openshift-authentication-operator authentication-operator apps 2021-06-07T05:37:53.199502Z {"metadata":{},"code":200}
system:serviceaccount:openshift-cluster-version:default deployments openshift-authentication-operator authentication-operator apps 2021-06-07T05:41:12.230721Z {"metadata":{},"code":200}
system:serviceaccount:openshift-cluster-version:default deployments openshift-cloud-credential-operator cloud-credential-operator apps 2021-06-07T03:45:26.600311Z {"metadata":{},"code":200}
system:serviceaccount:openshift-cluster-version:default deployments openshift-cloud-credential-operator cloud-credential-operator apps 2021-06-07T05:31:29.100584Z {"metadata":{},"code":200}
system:serviceaccount:openshift-cluster-version:default deployments openshift-cloud-credential-operator cloud-credential-operator apps 2021-06-07T05:34:48.071451Z {"metadata":{},"code":200}
system:serviceaccount:openshift-cluster-version:default deployments openshift-cloud-credential-operator cloud-credential-operator apps 2021-06-07T05:38:07.251233Z {"metadata":{},"code":200}
system:serviceaccount:openshift-cluster-version:default deployments openshift-cloud-credential-operator cloud-credential-operator apps 2021-06-07T05:41:26.267258Z {"metadata":{},"code":200}
system:serviceaccount:openshift-cluster-version:default deployments openshift-cluster-machine-approver machine-approver apps 2021-06-07T03:45:23.401765Z {"metadata":{},"code":200}
system:serviceaccount:openshift-cluster-version:default deployments openshift-cluster-machine-approver machine-approver apps 2021-06-07T05:31:24.851803Z {"metadata":{},"code":200}
system:serviceaccount:openshift-cluster-version:default deployments openshift-cluster-machine-approver machine-approver apps 2021-06-07T05:34:43.970960Z {"metadata":{},"code":200}
system:serviceaccount:openshift-cluster-version:default deployments openshift-cluster-machine-approver machine-approver apps 2021-06-07T05:38:03.000841Z {"metadata":{},"code":200}
system:serviceaccount:openshift-cluster-version:default deployments openshift-cluster-machine-approver machine-approver apps 2021-06-07T05:41:22.018392Z {"metadata":{},"code":200}
system:serviceaccount:openshift-cluster-version:default deployments openshift-cluster-node-tuning-operator cluster-node-tuning-operator apps 2021-06-07T03:45:29.199563Z {"metadata":{},"code":200}
system:serviceaccount:openshift-cluster-version:default deployments openshift-cluster-node-tuning-operator cluster-node-tuning-operator apps 2021-06-07T05:31:27.149407Z {"metadata":{},"code":200}
system:serviceaccount:openshift-cluster-version:default deployments openshift-cluster-node-tuning-operator cluster-node-tuning-operator apps 2021-06-07T05:34:46.270037Z {"metadata":{},"code":200}
system:serviceaccount:openshift-cluster-version:default deployments openshift-cluster-node-tuning-operator cluster-node-tuning-operator apps 2021-06-07T05:38:05.250627Z {"metadata":{},"code":200}
system:serviceaccount:openshift-cluster-version:default deployments openshift-cluster-node-tuning-operator cluster-node-tuning-operator apps 2021-06-07T05:41:24.266155Z {"metadata":{},"code":200}
system:serviceaccount:openshift-cluster-version:default deployments openshift-cluster-samples-operator cluster-samples-operator apps 2021-06-07T03:45:29.380140Z {"metadata":{},"code":200}
system:serviceaccount:openshift-cluster-version:default deployments openshift-cluster-samples-operator cluster-samples-operator apps 2021-06-07T05:31:17.406398Z {"metadata":{},"code":200}
system:serviceaccount:openshift-cluster-version:default deployments openshift-cluster-samples-operator cluster-samples-operator apps 2021-06-07T05:34:36.470267Z {"metadata":{},"code":200}
system:serviceaccount:openshift-cluster-version:default deployments openshift-cluster-samples-operator cluster-samples-operator apps 2021-06-07T05:37:55.501941Z {"metadata":{},"code":200}
system:serviceaccount:openshift-cluster-version:default deployments openshift-cluster-samples-operator cluster-samples-operator apps 2021-06-07T05:41:14.517012Z {"metadata":{},"code":200}
system:serviceaccount:openshift-cluster-version:default deployments openshift-cluster-storage-operator cluster-storage-operator apps 2021-06-07T03:45:30.026026Z {"metadata":{},"code":200}
system:serviceaccount:openshift-cluster-version:default deployments openshift-cluster-storage-operator cluster-storage-operator apps 2021-06-07T05:31:18.800018Z {"metadata":{},"code":200}
system:serviceaccount:openshift-cluster-version:default deployments openshift-cluster-storage-operator cluster-storage-operator apps 2021-06-07T05:34:37.869930Z {"metadata":{},"code":200}
system:serviceaccount:openshift-cluster-version:default deployments openshift-cluster-storage-operator cluster-storage-operator apps 2021-06-07T05:37:56.902133Z {"metadata":{},"code":200}
system:serviceaccount:openshift-cluster-version:default deployments openshift-cluster-storage-operator cluster-storage-operator apps 2021-06-07T05:41:15.917454Z {"metadata":{},"code":200}
system:serviceaccount:openshift-cluster-version:default deployments openshift-cluster-storage-operator csi-snapshot-controller-operator apps 2021-06-07T03:45:28.748924Z {"metadata":{},"code":200}
system:serviceaccount:openshift-cluster-version:default deployments openshift-cluster-storage-operator csi-snapshot-controller-operator apps 2021-06-07T05:31:30.999243Z {"metadata":{},"code":200}
system:serviceaccount:openshift-cluster-version:default deployments openshift-cluster-storage-operator csi-snapshot-controller-operator apps 2021-06-07T05:34:50.019140Z {"metadata":{},"code":200}
system:serviceaccount:openshift-cluster-version:default deployments openshift-cluster-storage-operator csi-snapshot-controller-operator apps 2021-06-07T05:38:09.150049Z {"metadata":{},"code":200}
system:serviceaccount:openshift-cluster-version:default deployments openshift-cluster-storage-operator csi-snapshot-controller-operator apps 2021-06-07T05:41:28.165946Z {"metadata":{},"code":200}
system:serviceaccount:openshift-cluster-version:default deployments openshift-cluster-version cluster-version-operator apps 2021-06-07T03:45:20.750886Z {"metadata":{},"code":200}
system:serviceaccount:openshift-cluster-version:default deployments openshift-cluster-version cluster-version-operator apps 2021-06-07T05:31:33.500463Z {"metadata":{},"code":200}
system:serviceaccount:openshift-cluster-version:default deployments openshift-cluster-version cluster-version-operator apps 2021-06-07T05:34:52.520606Z {"metadata":{},"code":200}
system:serviceaccount:openshift-cluster-version:default deployments openshift-cluster-version cluster-version-operator apps 2021-06-07T05:38:11.602789Z {"metadata":{},"code":200}
system:serviceaccount:openshift-cluster-version:default deployments openshift-cluster-version cluster-version-operator apps 2021-06-07T05:41:30.616033Z {"metadata":{},"code":200}
system:serviceaccount:openshift-cluster-version:default deployments openshift-config-operator openshift-config-operator apps 2021-06-07T03:45:25.326161Z {"metadata":{},"code":200}
system:serviceaccount:openshift-cluster-version:default deployments openshift-config-operator openshift-config-operator apps 2021-06-07T05:31:23.750077Z {"metadata":{},"code":200}
system:serviceaccount:openshift-cluster-version:default deployments openshift-config-operator openshift-config-operator apps 2021-06-07T05:34:42.870842Z {"metadata":{},"code":200}
system:serviceaccount:openshift-cluster-version:default deployments openshift-config-operator openshift-config-operator apps 2021-06-07T05:38:01.901223Z {"metadata":{},"code":200}
system:serviceaccount:openshift-cluster-version:default deployments openshift-config-operator openshift-config-operator apps 2021-06-07T05:41:20.917350Z {"metadata":{},"code":200}
system:serviceaccount:openshift-cluster-version:default deployments openshift-console-operator console-operator apps 2021-06-07T03:45:31.699927Z {"metadata":{},"code":200}
system:serviceaccount:openshift-cluster-version:default deployments openshift-console-operator console-operator apps 2021-06-07T05:31:40.150098Z {"metadata":{},"code":200}
system:serviceaccount:openshift-cluster-version:default deployments openshift-console-operator console-operator apps 2021-06-07T05:34:59.171186Z {"metadata":{},"code":200}
system:serviceaccount:openshift-cluster-version:default deployments openshift-console-operator console-operator apps 2021-06-07T05:38:18.200986Z {"metadata":{},"code":200}
system:serviceaccount:openshift-cluster-version:default deployments openshift-controller-manager-operator openshift-controller-manager-operator apps 2021-06-07T03:45:24.251611Z {"metadata":{},"code":200}
system:serviceaccount:openshift-cluster-version:default deployments openshift-controller-manager-operator openshift-controller-manager-operator apps 2021-06-07T05:31:20.399832Z {"metadata":{},"code":200}
system:serviceaccount:openshift-cluster-version:default deployments openshift-controller-manager-operator openshift-controller-manager-operator apps 2021-06-07T05:34:39.470552Z {"metadata":{},"code":200}
system:serviceaccount:openshift-cluster-version:default deployments openshift-controller-manager-operator openshift-controller-manager-operator apps 2021-06-07T05:37:58.502471Z {"metadata":{},"code":200}
system:serviceaccount:openshift-cluster-version:default deployments openshift-controller-manager-operator openshift-controller-manager-operator apps 2021-06-07T05:41:17.518188Z {"metadata":{},"code":200}
system:serviceaccount:openshift-cluster-version:default deployments openshift-dns-operator dns-operator apps 2021-06-07T03:45:22.549695Z {"metadata":{},"code":200}
system:serviceaccount:openshift-cluster-version:default deployments openshift-dns-operator dns-operator apps 2021-06-07T05:31:36.399082Z {"metadata":{},"code":200}
system:serviceaccount:openshift-cluster-version:default deployments openshift-dns-operator dns-operator apps 2021-06-07T05:34:55.420451Z {"metadata":{},"code":200}
system:serviceaccount:openshift-cluster-version:default deployments openshift-dns-operator dns-operator apps 2021-06-07T05:38:14.500618Z {"metadata":{},"code":200}
system:serviceaccount:openshift-cluster-version:default deployments openshift-etcd-operator etcd-operator apps 2021-06-07T03:45:25.277629Z {"metadata":{},"code":200}
system:serviceaccount:openshift-cluster-version:default deployments openshift-etcd-operator etcd-operator apps 2021-06-07T05:31:39.249976Z {"metadata":{},"code":200}
system:serviceaccount:openshift-cluster-version:default deployments openshift-etcd-operator etcd-operator apps 2021-06-07T05:34:58.370566Z {"metadata":{},"code":200}
system:serviceaccount:openshift-cluster-version:default deployments openshift-etcd-operator etcd-operator apps 2021-06-07T05:38:17.356739Z {"metadata":{},"code":200}
system:serviceaccount:openshift-cluster-version:default deployments openshift-image-registry cluster-image-registry-operator apps 2021-06-07T03:45:30.152350Z {"metadata":{},"code":200}
system:serviceaccount:openshift-cluster-version:default deployments openshift-image-registry cluster-image-registry-operator apps 2021-06-07T05:31:24.700505Z {"metadata":{},"code":200}
system:serviceaccount:openshift-cluster-version:default deployments openshift-image-registry cluster-image-registry-operator apps 2021-06-07T05:34:43.720934Z {"metadata":{},"code":200}
system:serviceaccount:openshift-cluster-version:default deployments openshift-image-registry cluster-image-registry-operator apps 2021-06-07T05:38:02.751361Z {"metadata":{},"code":200}
system:serviceaccount:openshift-cluster-version:default deployments openshift-image-registry cluster-image-registry-operator apps 2021-06-07T05:41:21.768527Z {"metadata":{},"code":200}
system:serviceaccount:openshift-cluster-version:default deployments openshift-ingress-operator ingress-operator apps 2021-06-07T03:45:29.125916Z {"metadata":{},"code":200}
system:serviceaccount:openshift-cluster-version:default deployments openshift-ingress-operator ingress-operator apps 2021-06-07T05:31:35.200551Z {"metadata":{},"code":200}
system:serviceaccount:openshift-cluster-version:default deployments openshift-ingress-operator ingress-operator apps 2021-06-07T05:34:54.219887Z {"metadata":{},"code":200}
system:serviceaccount:openshift-cluster-version:default deployments openshift-ingress-operator ingress-operator apps 2021-06-07T05:38:13.301770Z {"metadata":{},"code":200}
system:serviceaccount:openshift-cluster-version:default deployments openshift-ingress-operator ingress-operator apps 2021-06-07T05:41:32.318984Z {"metadata":{},"code":200}
system:serviceaccount:openshift-cluster-version:default deployments openshift-insights insights-operator apps 2021-06-07T03:45:30.680450Z {"metadata":{},"code":200}
system:serviceaccount:openshift-cluster-version:default deployments openshift-insights insights-operator apps 2021-06-07T05:31:35.650392Z {"metadata":{},"code":200}
system:serviceaccount:openshift-cluster-version:default deployments openshift-insights insights-operator apps 2021-06-07T05:34:54.771045Z {"metadata":{},"code":200}
system:serviceaccount:openshift-cluster-version:default deployments openshift-insights insights-operator apps 2021-06-07T05:38:13.750309Z {"metadata":{},"code":200}
system:serviceaccount:openshift-cluster-version:default deployments openshift-insights insights-operator apps 2021-06-07T05:41:32.768723Z {"metadata":{},"code":200}
system:serviceaccount:openshift-cluster-version:default deployments openshift-kube-apiserver-operator kube-apiserver-operator apps 2021-06-07T03:45:29.873967Z {"metadata":{},"code":200}
system:serviceaccount:openshift-cluster-version:default deployments openshift-kube-apiserver-operator kube-apiserver-operator apps 2021-06-07T05:31:22.749788Z {"metadata":{},"code":200}
system:serviceaccount:openshift-cluster-version:default deployments openshift-kube-apiserver-operator kube-apiserver-operator apps 2021-06-07T05:34:41.871450Z {"metadata":{},"code":200}
system:serviceaccount:openshift-cluster-version:default deployments openshift-kube-apiserver-operator kube-apiserver-operator apps 2021-06-07T05:38:00.901385Z {"metadata":{},"code":200}
system:serviceaccount:openshift-cluster-version:default deployments openshift-kube-apiserver-operator kube-apiserver-operator apps 2021-06-07T05:41:19.917678Z {"metadata":{},"code":200}
system:serviceaccount:openshift-cluster-version:default deployments openshift-kube-controller-manager-operator kube-controller-manager-operator apps 2021-06-07T03:45:23.076537Z {"metadata":{},"code":200}
system:serviceaccount:openshift-cluster-version:default deployments openshift-kube-controller-manager-operator kube-controller-manager-operator apps 2021-06-07T05:31:17.750037Z {"metadata":{},"code":200}
system:serviceaccount:openshift-cluster-version:default deployments openshift-kube-controller-manager-operator kube-controller-manager-operator apps 2021-06-07T05:34:36.870170Z {"metadata":{},"code":200}
system:serviceaccount:openshift-cluster-version:default deployments openshift-kube-controller-manager-operator kube-controller-manager-operator apps 2021-06-07T05:37:55.901014Z {"metadata":{},"code":200}
system:serviceaccount:openshift-cluster-version:default deployments openshift-kube-controller-manager-operator kube-controller-manager-operator apps 2021-06-07T05:41:14.916353Z {"metadata":{},"code":200}
system:serviceaccount:openshift-cluster-version:default deployments openshift-kube-scheduler-operator openshift-kube-scheduler-operator apps 2021-06-07T03:45:23.024916Z {"metadata":{},"code":200}
system:serviceaccount:openshift-cluster-version:default deployments openshift-kube-scheduler-operator openshift-kube-scheduler-operator apps 2021-06-07T05:31:32.200809Z {"metadata":{},"code":200}
system:serviceaccount:openshift-cluster-version:default deployments openshift-kube-scheduler-operator openshift-kube-scheduler-operator apps 2021-06-07T05:34:51.219865Z {"metadata":{},"code":200}
system:serviceaccount:openshift-cluster-version:default deployments openshift-kube-scheduler-operator openshift-kube-scheduler-operator apps 2021-06-07T05:38:10.350464Z {"metadata":{},"code":200}
system:serviceaccount:openshift-cluster-version:default deployments openshift-kube-scheduler-operator openshift-kube-scheduler-operator apps 2021-06-07T05:41:29.365662Z {"metadata":{},"code":200}
system:serviceaccount:openshift-cluster-version:default deployments openshift-kube-storage-version-migrator-operator kube-storage-version-migrator-operator apps 2021-06-07T03:45:23.754261Z {"metadata":{},"code":200}
system:serviceaccount:openshift-cluster-version:default deployments openshift-kube-storage-version-migrator-operator kube-storage-version-migrator-operator apps 2021-06-07T05:31:22.600288Z {"metadata":{},"code":200}
system:serviceaccount:openshift-cluster-version:default deployments openshift-kube-storage-version-migrator-operator kube-storage-version-migrator-operator apps 2021-06-07T05:34:41.618764Z {"metadata":{},"code":200}
system:serviceaccount:openshift-cluster-version:default deployments openshift-kube-storage-version-migrator-operator kube-storage-version-migrator-operator apps 2021-06-07T05:38:00.651259Z {"metadata":{},"code":200}
system:serviceaccount:openshift-cluster-version:default deployments openshift-kube-storage-version-migrator-operator kube-storage-version-migrator-operator apps 2021-06-07T05:41:19.667379Z {"metadata":{},"code":200}
system:serviceaccount:openshift-cluster-version:default deployments openshift-machine-api cluster-autoscaler-operator apps 2021-06-07T03:45:28.826008Z {"metadata":{},"code":200}
system:serviceaccount:openshift-cluster-version:default deployments openshift-machine-api cluster-autoscaler-operator apps 2021-06-07T05:31:14.980399Z {"metadata":{},"code":200}
system:serviceaccount:openshift-cluster-version:default deployments openshift-machine-api cluster-autoscaler-operator apps 2021-06-07T05:34:34.057253Z {"metadata":{},"code":200}
system:serviceaccount:openshift-cluster-version:default deployments openshift-machine-api cluster-autoscaler-operator apps 2021-06-07T05:37:53.081746Z {"metadata":{},"code":200}
system:serviceaccount:openshift-cluster-version:default deployments openshift-machine-api cluster-autoscaler-operator apps 2021-06-07T05:41:12.084375Z {"metadata":{},"code":200}
system:serviceaccount:openshift-cluster-version:default deployments openshift-machine-api cluster-baremetal-operator apps 2021-06-07T03:45:26.699924Z {"metadata":{},"code":200}
system:serviceaccount:openshift-cluster-version:default deployments openshift-machine-api cluster-baremetal-operator apps 2021-06-07T05:31:16.800097Z {"metadata":{},"code":200}
system:serviceaccount:openshift-cluster-version:default deployments openshift-machine-api cluster-baremetal-operator apps 2021-06-07T05:34:35.870242Z {"metadata":{},"code":200}
system:serviceaccount:openshift-cluster-version:default deployments openshift-machine-api cluster-baremetal-operator apps 2021-06-07T05:37:54.900788Z {"metadata":{},"code":200}
system:serviceaccount:openshift-cluster-version:default deployments openshift-machine-api cluster-baremetal-operator apps 2021-06-07T05:41:13.916059Z {"metadata":{},"code":200}
system:serviceaccount:openshift-cluster-version:default deployments openshift-machine-api machine-api-operator apps 2021-06-07T03:45:32.050383Z {"metadata":{},"code":200}
system:serviceaccount:openshift-cluster-version:default deployments openshift-machine-api machine-api-operator apps 2021-06-07T05:31:42.502659Z {"metadata":{},"code":200}
system:serviceaccount:openshift-cluster-version:default deployments openshift-machine-api machine-api-operator apps 2021-06-07T05:35:01.670926Z {"metadata":{},"code":200}
system:serviceaccount:openshift-cluster-version:default deployments openshift-machine-api machine-api-operator apps 2021-06-07T05:38:20.702118Z {"metadata":{},"code":200}
system:serviceaccount:openshift-cluster-version:default deployments openshift-machine-config-operator machine-config-operator apps 2021-06-07T03:45:27.421145Z {"metadata":{},"code":200}
system:serviceaccount:openshift-cluster-version:default deployments openshift-machine-config-operator machine-config-operator apps 2021-06-07T05:31:29.849653Z {"metadata":{},"code":200}
system:serviceaccount:openshift-cluster-version:default deployments openshift-machine-config-operator machine-config-operator apps 2021-06-07T05:34:48.971010Z {"metadata":{},"code":200}
system:serviceaccount:openshift-cluster-version:default deployments openshift-machine-config-operator machine-config-operator apps 2021-06-07T05:38:07.902335Z {"metadata":{},"code":200}
system:serviceaccount:openshift-cluster-version:default deployments openshift-machine-config-operator machine-config-operator apps 2021-06-07T05:41:26.917331Z {"metadata":{},"code":200}
system:serviceaccount:openshift-cluster-version:default deployments openshift-marketplace marketplace-operator apps 2021-06-07T03:45:24.450903Z {"metadata":{},"code":200}
system:serviceaccount:openshift-cluster-version:default deployments openshift-marketplace marketplace-operator apps 2021-06-07T05:31:21.066453Z {"metadata":{},"code":200}
system:serviceaccount:openshift-cluster-version:default deployments openshift-marketplace marketplace-operator apps 2021-06-07T05:34:40.121484Z {"metadata":{},"code":200}
system:serviceaccount:openshift-cluster-version:default deployments openshift-marketplace marketplace-operator apps 2021-06-07T05:37:59.152549Z {"metadata":{},"code":200}
system:serviceaccount:openshift-cluster-version:default deployments openshift-marketplace marketplace-operator apps 2021-06-07T05:41:18.168381Z {"metadata":{},"code":200}
system:serviceaccount:openshift-cluster-version:default deployments openshift-monitoring cluster-monitoring-operator apps 2021-06-07T03:45:27.376161Z {"metadata":{},"code":200}
system:serviceaccount:openshift-cluster-version:default deployments openshift-monitoring cluster-monitoring-operator apps 2021-06-07T05:31:19.650310Z {"metadata":{},"code":200}
system:serviceaccount:openshift-cluster-version:default deployments openshift-monitoring cluster-monitoring-operator apps 2021-06-07T05:34:38.720417Z {"metadata":{},"code":200}
system:serviceaccount:openshift-cluster-version:default deployments openshift-monitoring cluster-monitoring-operator apps 2021-06-07T05:37:57.751995Z {"metadata":{},"code":200}
system:serviceaccount:openshift-cluster-version:default deployments openshift-monitoring cluster-monitoring-operator apps 2021-06-07T05:41:16.766419Z {"metadata":{},"code":200}
system:serviceaccount:openshift-cluster-version:default deployments openshift-network-operator network-operator apps 2021-06-07T03:45:22.699854Z {"metadata":{},"code":200}
system:serviceaccount:openshift-cluster-version:default deployments openshift-network-operator network-operator apps 2021-06-07T05:31:27.849372Z {"metadata":{},"code":200}
system:serviceaccount:openshift-cluster-version:default deployments openshift-network-operator network-operator apps 2021-06-07T05:34:46.820994Z {"metadata":{},"code":200}
system:serviceaccount:openshift-cluster-version:default deployments openshift-network-operator network-operator apps 2021-06-07T05:38:05.900564Z {"metadata":{},"code":200}
system:serviceaccount:openshift-cluster-version:default deployments openshift-network-operator network-operator apps 2021-06-07T05:41:24.917817Z {"metadata":{},"code":200}
system:serviceaccount:openshift-cluster-version:default deployments openshift-operator-lifecycle-manager catalog-operator apps 2021-06-07T03:45:28.226132Z {"metadata":{},"code":200}
system:serviceaccount:openshift-cluster-version:default deployments openshift-operator-lifecycle-manager catalog-operator apps 2021-06-07T05:31:42.450600Z {"metadata":{},"code":200}
system:serviceaccount:openshift-cluster-version:default deployments openshift-operator-lifecycle-manager catalog-operator apps 2021-06-07T05:35:01.421574Z {"metadata":{},"code":200}
system:serviceaccount:openshift-cluster-version:default deployments openshift-operator-lifecycle-manager catalog-operator apps 2021-06-07T05:38:20.451824Z {"metadata":{},"code":200}
system:serviceaccount:openshift-cluster-version:default deployments openshift-operator-lifecycle-manager olm-operator apps 2021-06-07T03:45:27.301906Z {"metadata":{},"code":200}
system:serviceaccount:openshift-cluster-version:default deployments openshift-operator-lifecycle-manager olm-operator apps 2021-06-07T05:31:42.149728Z {"metadata":{},"code":200}
system:serviceaccount:openshift-cluster-version:default deployments openshift-operator-lifecycle-manager olm-operator apps 2021-06-07T05:35:01.121076Z {"metadata":{},"code":200}
system:serviceaccount:openshift-cluster-version:default deployments openshift-operator-lifecycle-manager olm-operator apps 2021-06-07T05:38:20.150900Z {"metadata":{},"code":200}
system:serviceaccount:openshift-cluster-version:default deployments openshift-service-ca-operator service-ca-operator apps 2021-06-07T03:45:21.023871Z {"metadata":{},"code":200}
system:serviceaccount:openshift-cluster-version:default deployments openshift-service-ca-operator service-ca-operator apps 2021-06-07T05:31:14.907619Z {"metadata":{},"code":200}
system:serviceaccount:openshift-cluster-version:default deployments openshift-service-ca-operator service-ca-operator apps 2021-06-07T05:34:33.977938Z {"metadata":{},"code":200}
system:serviceaccount:openshift-cluster-version:default deployments openshift-service-ca-operator service-ca-operator apps 2021-06-07T05:37:53.004246Z {"metadata":{},"code":200}
system:serviceaccount:openshift-cluster-version:default deployments openshift-service-ca-operator service-ca-operator apps 2021-06-07T05:41:12.031695Z {"metadata":{},"code":200}

CVO hotloops on above deployments
 
Verifying with 4.8.0-0.nightly-2021-06-06-164529

# oc get clusterversion
NAME      VERSION                             AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.8.0-0.nightly-2021-06-06-164529   True        False         66m     Cluster version is 4.8.0-0.nightly-2021-06-06-164529

# masters=$(oc get no -l node-role.kubernetes.io/master | sed '1d' | awk '{print $1}')

# oc adm node-logs $masters --path=kube-apiserver/audit.log --raw | zgrep -h '"verb":"update".*"resource":".*deployments"' 2>/dev/null | jq -r '.user.username + " " + (.objectRef | .resource + " " + .namespace + " " + .name + " " + .apiGroup) + " " + .stageTimestamp + " " + (.responseStatus | tostring)' | grep deployments | grep 'serviceaccount:openshift-cluster-version' | sort

null

CVO does not constantly update deployments. Moving it to verified state.

Comment 13 errata-xmlrpc 2021-07-27 22:33:27 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: OpenShift Container Platform 4.8.2 bug fix and security update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2021:2438

Note You need to log in before you can comment on or make changes to this bug.