Bug 1961561

Summary:	The encryption controllers send lots of request to an API server
Product:	OpenShift Container Platform	Reporter:	Lukasz Szaszkiewicz <lszaszki>
Component:	kube-apiserver	Assignee:	Lukasz Szaszkiewicz <lszaszki>
Status:	CLOSED ERRATA	QA Contact:	Ke Wang <kewang>
Severity:	high	Docs Contact:
Priority:	high
Version:	4.8	CC:	aos-bugs, mfojtik, xxia
Target Milestone:	---
Target Release:	4.8.0
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2021-07-27 23:08:55 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description Lukasz Szaszkiewicz 2021-05-18 09:22:38 UTC

Synchronizing encryption controllers is expensive because they pull data directly from the servers to get the most recent data.

By default, the controllers resync every 60 seconds. However, tighter loops can be enforced on dependencies. For example, the authentication operator reconciles its resource every 20 seconds.

We provided a precondition checker [1] that determines if encryption controllers should synchronize. This helps to avoid sending requests to the API servers if there is no work to do.


The precondition checker must be pulled into kas-o, oas-o and the authentication operator.


[1] - https://github.com/openshift/library-go/pull/1059

Comment 2 Lukasz Szaszkiewicz 2021-05-25 15:43:40 UTC

I downloaded the audit-logs and used https://github.com/openshift/cluster-debug-tools to get requests for encryption-config secret made by the cluster-openshift-apiserver-operator.

I haven't found any requests for the secret after applying the fix.

after: ./kubectl-dev_tool audit -f /Users/lszaszki/workspace/Downloads/audit-logs/registry-build01-ci-openshift-org-ci-op-d3lj2xcr-stable-sha256-b26af26229ab63635a788809f1997f11bbe81d2a5a815203f7002c67711f68fb/audit_logs/kube-apiserver --by=resource --user=system:serviceaccount:openshift-apiserver-operator:openshift-apiserver-operator --verb=list,get -otop=50 -owide | grep encryption

before: ./kubectl-dev_tool audit -f /Users/lszaszki/workspace/Downloads/audit-logs-before/registry-build01-ci-openshift-org-ci-op-k9msd634-stable-sha256-b26af26229ab63635a788809f1997f11bbe81d2a5a815203f7002c67711f68fb/audit_logs/kube-apiserver --by=resource --user=system:serviceaccount:openshift-apiserver-operator:openshift-apiserver-operator --verb=list,get -otop=50 -owide | grep encryption


after: https://prow.ci.openshift.org/view/gs/origin-ci-test/pr-logs/pull/openshift_cluster-openshift-apiserver-operator/451/pull-ci-openshift-cluster-openshift-apiserver-operator-master-e2e-aws/1394239157840121856

before: https://prow.ci.openshift.org/view/gs/origin-ci-test/pr-logs/pull/openshift_cluster-openshift-apiserver-operator/448/pull-ci-openshift-cluster-openshift-apiserver-operator-master-e2e-aws/1393074832035287040


It would be nice if we could do similar tests for kas-o and the authentication operator

Comment 4 Ke Wang 2021-06-10 11:38:11 UTC

Refer to Comment #2, checked all audit.log files on master, 

$ oc get clusterversion
NAME      VERSION                             AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.8.0-0.nightly-2021-06-09-142759   True        False         9h      Cluster version is 4.8.0-0.nightly-2021-06-09-142759

$ masters=$(oc get no -l node-role.kubernetes.io/master | sed '1d' | awk '{print $1}')
$ oc adm node-logs $masters --path=kube-apiserver/audit.log > kas-audit.log
$ oc adm node-logs $masters --path=openshift-apiserver/audit.log > oas-audit.log;oc adm node-logs $masters --path=oauth-apiserver/audit.log > oauth-audit.log

Downloaded and used https://github.com/openshift/cluster-debug-tools to check if there are requests for encryption-config secret made by the cluster-openshift-apiserver-operator, openshift-kube-apiserver-operator and openshift-authentication-operator, 
$./kubectl-dev_tool audit -f oas-audit.log --by=resource --user=system:serviceaccount:openshift-apiserver-operator:openshift-apiserver-operator --verb=list,get -otop=50 -owide | grep encryption

$./kubectl-dev_tool audit -f kas-audit.log --by=resource --user=system:serviceaccount:openshift-kube-apiserver-operator:kube-apiserver-operator --verb=list,get -otop=50 -owide | grep encryption

$./kubectl-dev_tool audit -f oauth-audit.log --by=resource --user=system:serviceaccount:openshift-authentication-operator:authentication-operator --verb=list,get -otop=50 -owide | grep encryption

After checked, no any requests for the secret can be found, it is as expected, so move the bug VERIFIED.

Comment 7 errata-xmlrpc 2021-07-27 23:08:55 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: OpenShift Container Platform 4.8.2 bug fix and security update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2021:2438