Bug 1795412 - OpenAPI spec regenerates once every 3 minutes on each kube-apiserver, increasing CPU and memory use
Summary: OpenAPI spec regenerates once every 3 minutes on each kube-apiserver, increas...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Cluster Version Operator
Version: 4.4
Hardware: Unspecified
OS: Unspecified
high
high
Target Milestone: ---
: 4.4.0
Assignee: Stefan Schimanski
QA Contact: Ke Wang
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2020-01-27 21:55 UTC by Clayton Coleman
Modified: 2020-05-04 11:27 UTC (History)
7 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2020-05-04 11:27:25 UTC
Target Upstream Version:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Github openshift cluster-config-operator pull 115 0 None closed Bug 1795412: Only install config-operator config/v1 manifests 2020-12-19 15:56:02 UTC
Github openshift cluster-version-operator pull 258 0 None closed Bug 1795412: Print whether CRDs are updated 2020-12-19 15:56:04 UTC
Red Hat Product Errata RHBA-2020:0581 0 None None None 2020-05-04 11:27:50 UTC

Description Clayton Coleman 2020-01-27 21:55:14 UTC
Looking at idle cluster kube-apiservers, openapi regeneration is still ~25% of CPU and ~16% of in use memory and 30% of allocated objects.

I looked at the logs and see this every 3 minutes, which seems to indicate we are regenerating the core openapi schemas repeatedly?

I0127 21:38:58.618423       1 aggregator.go:226] Updating OpenAPI spec because k8s_internal_local_delegation_chain_0000000002 is updated
I0127 21:42:06.521407       1 aggregator.go:226] Updating OpenAPI spec because k8s_internal_local_delegation_chain_0000000002 is updated
I0127 21:42:13.187747       1 aggregator.go:226] Updating OpenAPI spec because k8s_internal_local_delegation_chain_0000000002 is updated
I0127 21:45:21.099039       1 aggregator.go:226] Updating OpenAPI spec because k8s_internal_local_delegation_chain_0000000002 is updated
I0127 21:45:28.814451       1 aggregator.go:226] Updating OpenAPI spec because k8s_internal_local_delegation_chain_0000000002 is updated
I0127 21:48:36.571656       1 aggregator.go:226] Updating OpenAPI spec because k8s_internal_local_delegation_chain_0000000002 is updated
I0127 21:48:43.411851       1 aggregator.go:226] Updating OpenAPI spec because k8s_internal_local_delegation_chain_0000000002 is updated

I see that on each apiserver.

High because this wastes idle CPU.

Comment 3 liujia 2020-02-06 06:24:51 UTC
Hi Stefan Schimanski 

According to https://github.com/openshift/cluster-version-operator/pull/258#issuecomment-582349256, it seems this pr is only for adding more logs for debugging the increase of usage of the apiservers. So I wonder if there is another pr to fix the issue about cpu&mem usage increasing actually? or else, how show qe verify the bug? It's enough to  check logs added?

Comment 7 Ke Wang 2020-02-14 09:18:58 UTC
Confirmed with Dev that the similar msg ‘Updating CRD ...’ won't be seen in the kube-apiserver related log if fixed.

Verified with OCP 4.4.0-0.nightly-2020-02-13-185851, steps as below,

$ apiserver_node=$(oc get po -o wide -n openshift-apiserver | grep apiserver | awk '{print $7}' | head -1)
$ oc debug node/$apiserver_node

After logged it,
sh-4.2# chroot /host

sh-4.4# cd /var/log/pods

sh-4.4# grep -rn 'Updating CRD' *apiserver*

There is no any output.

Comment 9 errata-xmlrpc 2020-05-04 11:27:25 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:0581


Note You need to log in before you can comment on or make changes to this bug.