Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

Bug 2015052

Summary: OLM (and little less auth operator) heavily access apiserver, WAS: Massive audit log generation /var/log/kube-apiserver/
Product: OpenShift Container Platform Reporter: Gianluca Villani <gvillani>
Component: OLMAssignee: Kevin Rizza <krizza>
OLM sub component: OLM QA Contact: Jian Zhang <jiazha>
Status: CLOSED NOTABUG Docs Contact:
Severity: high    
Priority: high CC: aos-bugs, krizza, mfojtik, sttts, xxia
Version: 4.8   
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2021-11-01 17:57:12 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Comment 10 Stefan Schimanski 2021-10-21 10:03:44 UTC 
Here is another analysis, this time by client of the top resources (with >1 request per second, summed up for all apiservers):

for f in *.yaml; do if [ $(yq eval -j $f | jq '.status.currentHour.requestCount > 3600') = "true" ]; then echo "$(yq eval -j $f | jq '.status.currentHour.byNode | max_by(.requestCount).requestCount') $f $(yq eval -j $f | jq -r '.status.currentHour.byNode | max_by(.requestCount).byUser[0] | "\(.username) \(.requestCount)"')"; fi; done | sort -nr

57331 clusterserviceversions.v1alpha1.operators.coreos.com.yaml system:serviceaccount:openshift-operator-lifecycle-manager:olm-operator-serviceaccount 57272
32584 serviceaccounts.v1.yaml system:serviceaccount:openshift-operator-lifecycle-manager:olm-operator-serviceaccount 29701
28791 secrets.v1.yaml system:serviceaccount:openshift-authentication-operator:authentication-operator 11406
25475 configmaps.v1.yaml system:serviceaccount:openshift-kube-controller-manager-operator:kube-controller-manager-operator 8787
19451 clusteroperators.v1.config.openshift.io.yaml system:serviceaccount:openshift-operator-lifecycle-manager:olm-operator-serviceaccount 18704
16017 openshiftcontrollermanagers.v1.operator.openshift.io.yaml system:serviceaccount:openshift-controller-manager-operator:openshift-controller-manager-operator 16001
14280 subjectaccessreviews.v1.authorization.k8s.io.yaml system:serviceaccount:openshift-apiserver:openshift-apiserver-sa 10500
14260 clusterrolebindings.v1.rbac.authorization.k8s.io.yaml system:serviceaccount:openshift-kube-storage-version-migrator-operator:kube-storage-version-migrator-operator 7403
13055 namespaces.v1.yaml system:serviceaccount:openshift-kube-storage-version-migrator-operator:kube-storage-version-migrator-operator 7402
10798 deployments.v1.apps.yaml system:serviceaccount:openshift-kube-storage-version-migrator-operator:kube-storage-version-migrator-operator 7411
9871 leases.v1.coordination.k8s.io.yaml system:serviceaccount:openshift-storage:ocs-operator 4335
7410 kubestorageversionmigrators.v1.operator.openshift.io.yaml system:serviceaccount:openshift-kube-storage-version-migrator-operator:kube-storage-version-migrator-operator 7409
7162 services.v1.yaml system:serviceaccount:openshift-operators-redhat:elasticsearch-operator 5353
7057 apirequestcounts.v1.apiserver.openshift.io.yaml system:apiserver 7050
5515 rolebindings.v1.rbac.authorization.k8s.io.yaml system:serviceaccount:openshift-kube-scheduler-operator:openshift-kube-scheduler-operator 3445
5353 clusterroles.v1.rbac.authorization.k8s.io.yaml system:serviceaccount:openshift-operators-redhat:elasticsearch-operator 3572
4366 tokenreviews.v1.authentication.k8s.io.yaml system:kube-scheduler 663
3428 roles.v1.rbac.authorization.k8s.io.yaml system:serviceaccount:openshift-operators-redhat:elasticsearch-operator 1612
3137 prometheusrules.v1.monitoring.coreos.com.yaml system:serviceaccount:openshift-kube-apiserver-operator:kube-apiserver-operator 2792
2759 persistentvolumeclaims.v1.yaml system:serviceaccount:openshift-operators-redhat:elasticsearch-operator 2680
2673 servicemonitors.v1.monitoring.coreos.com.yaml system:serviceaccount:openshift-operators-redhat:elasticsearch-operator 1968
2610 nodes.v1.yaml system:node:ocp4azexp2-mqn4f-ocs-northeurope2-g6hb6 314
1636 pods.v1.yaml system:serviceaccount:openshift-authentication-operator:authentication-operator 325

By far most request come from OLM. Moving over to OLM team, but also informing auth team about their top3 entry.
Comment 14 Michal Fojtik 2021-11-08 09:23:42 UTC 
A follow-up BZ has been created to investigate the kube-storage-migrator requests: https://bugzilla.redhat.com/show_bug.cgi?id=2021067