Looking at idle cluster kube-apiservers, openapi regeneration is still ~25% of CPU and ~16% of in use memory and 30% of allocated objects. I looked at the logs and see this every 3 minutes, which seems to indicate we are regenerating the core openapi schemas repeatedly? I0127 21:38:58.618423 1 aggregator.go:226] Updating OpenAPI spec because k8s_internal_local_delegation_chain_0000000002 is updated I0127 21:42:06.521407 1 aggregator.go:226] Updating OpenAPI spec because k8s_internal_local_delegation_chain_0000000002 is updated I0127 21:42:13.187747 1 aggregator.go:226] Updating OpenAPI spec because k8s_internal_local_delegation_chain_0000000002 is updated I0127 21:45:21.099039 1 aggregator.go:226] Updating OpenAPI spec because k8s_internal_local_delegation_chain_0000000002 is updated I0127 21:45:28.814451 1 aggregator.go:226] Updating OpenAPI spec because k8s_internal_local_delegation_chain_0000000002 is updated I0127 21:48:36.571656 1 aggregator.go:226] Updating OpenAPI spec because k8s_internal_local_delegation_chain_0000000002 is updated I0127 21:48:43.411851 1 aggregator.go:226] Updating OpenAPI spec because k8s_internal_local_delegation_chain_0000000002 is updated I see that on each apiserver. High because this wastes idle CPU.
Hi Stefan Schimanski According to https://github.com/openshift/cluster-version-operator/pull/258#issuecomment-582349256, it seems this pr is only for adding more logs for debugging the increase of usage of the apiservers. So I wonder if there is another pr to fix the issue about cpu&mem usage increasing actually? or else, how show qe verify the bug? It's enough to check logs added?
Confirmed with Dev that the similar msg ‘Updating CRD ...’ won't be seen in the kube-apiserver related log if fixed. Verified with OCP 4.4.0-0.nightly-2020-02-13-185851, steps as below, $ apiserver_node=$(oc get po -o wide -n openshift-apiserver | grep apiserver | awk '{print $7}' | head -1) $ oc debug node/$apiserver_node After logged it, sh-4.2# chroot /host sh-4.4# cd /var/log/pods sh-4.4# grep -rn 'Updating CRD' *apiserver* There is no any output.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2020:0581