Bug 1795412

Summary:	OpenAPI spec regenerates once every 3 minutes on each kube-apiserver, increasing CPU and memory use
Product:	OpenShift Container Platform	Reporter:	Clayton Coleman <ccoleman>
Component:	Cluster Version Operator	Assignee:	Stefan Schimanski <sttts>
Status:	CLOSED ERRATA	QA Contact:	Ke Wang <kewang>
Severity:	high	Docs Contact:
Priority:	high
Version:	4.4	CC:	aos-bugs, jiajliu, jokerman, lszaszki, mfojtik, sttts, xxia
Target Milestone:	---
Target Release:	4.4.0
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2020-05-04 11:27:25 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description Clayton Coleman 2020-01-27 21:55:14 UTC

Looking at idle cluster kube-apiservers, openapi regeneration is still ~25% of CPU and ~16% of in use memory and 30% of allocated objects.

I looked at the logs and see this every 3 minutes, which seems to indicate we are regenerating the core openapi schemas repeatedly?

I0127 21:38:58.618423       1 aggregator.go:226] Updating OpenAPI spec because k8s_internal_local_delegation_chain_0000000002 is updated
I0127 21:42:06.521407       1 aggregator.go:226] Updating OpenAPI spec because k8s_internal_local_delegation_chain_0000000002 is updated
I0127 21:42:13.187747       1 aggregator.go:226] Updating OpenAPI spec because k8s_internal_local_delegation_chain_0000000002 is updated
I0127 21:45:21.099039       1 aggregator.go:226] Updating OpenAPI spec because k8s_internal_local_delegation_chain_0000000002 is updated
I0127 21:45:28.814451       1 aggregator.go:226] Updating OpenAPI spec because k8s_internal_local_delegation_chain_0000000002 is updated
I0127 21:48:36.571656       1 aggregator.go:226] Updating OpenAPI spec because k8s_internal_local_delegation_chain_0000000002 is updated
I0127 21:48:43.411851       1 aggregator.go:226] Updating OpenAPI spec because k8s_internal_local_delegation_chain_0000000002 is updated

I see that on each apiserver.

High because this wastes idle CPU.

Comment 3 liujia 2020-02-06 06:24:51 UTC

Hi Stefan Schimanski 

According to https://github.com/openshift/cluster-version-operator/pull/258#issuecomment-582349256, it seems this pr is only for adding more logs for debugging the increase of usage of the apiservers. So I wonder if there is another pr to fix the issue about cpu&mem usage increasing actually? or else, how show qe verify the bug? It's enough to  check logs added?

Comment 7 Ke Wang 2020-02-14 09:18:58 UTC

Confirmed with Dev that the similar msg ‘Updating CRD ...’ won't be seen in the kube-apiserver related log if fixed.

Verified with OCP 4.4.0-0.nightly-2020-02-13-185851, steps as below,

$ apiserver_node=$(oc get po -o wide -n openshift-apiserver | grep apiserver | awk '{print $7}' | head -1)
$ oc debug node/$apiserver_node

After logged it,
sh-4.2# chroot /host

sh-4.4# cd /var/log/pods

sh-4.4# grep -rn 'Updating CRD' *apiserver*

There is no any output.

Comment 9 errata-xmlrpc 2020-05-04 11:27:25 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:0581