Bug 1747479 - kube-apiserver: excessive logging "OpenAPI AggregationController: Processing item" [NEEDINFO]
Summary: kube-apiserver: excessive logging "OpenAPI AggregationController: Processing ...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: kube-apiserver
Version: 4.2.0
Hardware: Unspecified
OS: Unspecified
unspecified
medium
Target Milestone: ---
: 4.4.0
Assignee: Stefan Schimanski
QA Contact: Ke Wang
URL:
Whiteboard:
: 1747268 (view as bug list)
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2019-08-30 14:21 UTC by Seth Jennings
Modified: 2020-05-04 11:14 UTC (History)
7 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2020-05-04 11:13:32 UTC
Target Upstream Version:
fshaikh: needinfo? (sttts)
fshaikh: needinfo? (sttts)


Attachments (Terms of Use)


Links
System ID Priority Status Summary Last Updated
Github openshift origin pull 24497 None closed Bug 1747479: UPSTREAM: 87813: kube-aggregator: increase log level of AggregationController API group logging 2020-11-15 13:55:42 UTC
Red Hat Knowledge Base (Solution) 4666231 None None None 2019-12-17 21:43:40 UTC
Red Hat Product Errata RHBA-2020:0581 None None None 2020-05-04 11:14:08 UTC

Description Seth Jennings 2019-08-30 14:21:03 UTC
https://storage.googleapis.com/origin-ci-test/logs/release-openshift-ocp-installer-e2e-aws-serial-4.2/450/artifacts/e2e-aws/pods/openshift-kube-apiserver_kube-apiserver-ip-10-0-130-108.ec2.internal_kube-apiserver-6.log

~15 msgs every minute

I0830 11:56:43.701053       1 controller.go:107] OpenAPI AggregationController: Processing item v1.user.openshift.io
I0830 11:56:43.706210       1 controller.go:107] OpenAPI AggregationController: Processing item v1.security.openshift.io
I0830 11:56:48.657864       1 controller.go:107] OpenAPI AggregationController: Processing item v1.oauth.openshift.io
I0830 11:56:56.144497       1 controller.go:107] OpenAPI AggregationController: Processing item v1.image.openshift.io
I0830 11:56:56.147784       1 controller.go:107] OpenAPI AggregationController: Processing item v1.quota.openshift.io
I0830 11:56:56.150903       1 controller.go:107] OpenAPI AggregationController: Processing item v1.apps.openshift.io
I0830 11:56:56.153554       1 controller.go:107] OpenAPI AggregationController: Processing item v1.project.openshift.io
I0830 11:57:00.170983       1 controller.go:107] OpenAPI AggregationController: Processing item v1.route.openshift.io
I0830 11:57:00.173671       1 controller.go:107] OpenAPI AggregationController: Processing item v1.packages.operators.coreos.com
I0830 11:57:00.176051       1 controller.go:107] OpenAPI AggregationController: Processing item v1.template.openshift.io
I0830 11:57:00.177193       1 controller.go:107] OpenAPI AggregationController: Processing item v1.authorization.openshift.io
I0830 11:57:00.178204       1 controller.go:107] OpenAPI AggregationController: Processing item v1.build.openshift.io
I0830 11:57:26.791091       1 controller.go:107] OpenAPI AggregationController: Processing item v1beta1.metrics.k8s.io
E0830 11:57:26.794934       1 controller.go:114] loading OpenAPI spec for "v1beta1.metrics.k8s.io" failed with: OpenAPI spec does not exist
I0830 11:57:26.794955       1 controller.go:127] OpenAPI AggregationController: action for item v1beta1.metrics.k8s.io: Rate Limited Requeue.

Related:
https://bugzilla.redhat.com/show_bug.cgi?id=1747268

Comment 2 Michal Fojtik 2019-11-06 20:21:13 UTC
*** Bug 1747268 has been marked as a duplicate of this bug. ***

Comment 3 Fatima 2019-11-29 09:51:18 UTC
Hi team,

We have a customer who is facing the same issue on OCP 4.2.7

Excessive logging (36 msgs/second) of messages below...

...I1125 14:02:12.550102       1 controller.go:102] OpenAPI AggregationController: Processing item k8s_internal_local_delegation_chain_0000000001

As of now, he has re-deployed the cluster and is not facing the issue in the new one.

Thanks,
Fatima

Comment 6 Fatima 2020-02-03 10:18:18 UTC
Hi,

Since OCP 4.3 is released is this bug fixed there?

Any updates would be appreciated.

Thanks,
Fatima

Comment 7 Seth Jennings 2020-02-03 16:03:16 UTC
This still happens in 4.3.0

I0203 16:00:18.926067       1 controller.go:107] OpenAPI AggregationController: Processing item v1.project.openshift.io
I0203 16:00:20.980164       1 controller.go:107] OpenAPI AggregationController: Processing item v1.build.openshift.io
I0203 16:00:23.470490       1 controller.go:107] OpenAPI AggregationController: Processing item v1.image.openshift.io
I0203 16:00:25.661778       1 controller.go:107] OpenAPI AggregationController: Processing item v1.security.openshift.io
I0203 16:00:27.330009       1 controller.go:107] OpenAPI AggregationController: Processing item v1.packages.operators.coreos.com
I0203 16:00:29.283581       1 controller.go:107] OpenAPI AggregationController: Processing item v1.apps.openshift.io
I0203 16:00:31.461177       1 controller.go:107] OpenAPI AggregationController: Processing item v1.route.openshift.io
I0203 16:00:33.648494       1 controller.go:107] OpenAPI AggregationController: Processing item v1.quota.openshift.io
I0203 16:00:35.687155       1 controller.go:107] OpenAPI AggregationController: Processing item v1.template.openshift.io
I0203 16:00:38.080755       1 controller.go:107] OpenAPI AggregationController: Processing item v1.oauth.openshift.io
I0203 16:00:40.981177       1 controller.go:107] OpenAPI AggregationController: Processing item v1.user.openshift.io
I0203 16:00:42.906006       1 controller.go:107] OpenAPI AggregationController: Processing item v1.authorization.openshift.io
I0203 16:01:02.518120       1 controller.go:107] OpenAPI AggregationController: Processing item v1beta1.metrics.k8s.io
E0203 16:01:02.538167       1 controller.go:114] loading OpenAPI spec for "v1beta1.metrics.k8s.io" failed with: OpenAPI spec does not exist
I0203 16:01:02.538341       1 controller.go:127] OpenAPI AggregationController: action for item v1beta1.metrics.k8s.io: Rate Limited Requeue.
I0203 16:01:18.930814       1 controller.go:107] OpenAPI AggregationController: Processing item v1.project.openshift.io

Comment 9 Ke Wang 2020-02-13 15:05:29 UTC
Verified with OCP build 4.4.0-0.nightly-2020-02-13-103342, searching keyword 'OpenAPI AggregationController' in 
https://storage.googleapis.com/origin-ci-test/logs/release-openshift-ocp-installer-e2e-openstack-serial-4.4/813/artifacts/e2e-openstack-serial/pods/openshift-kube-apiserver_kube-apiserver-rtvq59zz-bc4a8-9zq6p-master-0_kube-apiserver.log, 
the similar msg 'OpenAPI AggregationController: Processing item ...' cannot be found.

Comment 10 Oscar Casal Sanchez 2020-04-02 16:27:40 UTC
Hello,

I know that this issue is at this moment to be fixed in OCP 4.4, but at this point, people using 4.2 hadn't upgrade path dues the BZ1810036. Could be this backported to 4.2? This behaviour severely limits the ability to use the centralized logging feature since it's flooding all the logs.

Thank you in advance.

Comment 12 errata-xmlrpc 2020-05-04 11:13:32 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:0581


Note You need to log in before you can comment on or make changes to this bug.