Bug 1820468

Summary: Lots of log spam 'loading OpenAPI spec for "v1beta1.metrics.k8s.io" failed with: OpenAPI spec does not exist" annoying
Product: OpenShift Container Platform Reporter: Xingxing Xia <xxia>
Component: MonitoringAssignee: Sergiusz Urbaniak <surbania>
Status: CLOSED DUPLICATE QA Contact: Junqi Zhao <juzhao>
Severity: low Docs Contact:
Priority: unspecified    
Version: 4.4CC: alegrand, anpicker, erooth, juzhao, kakkoyun, lcosic, mloibl, mnewby, pkrupa, surbania, xxia
Target Milestone: ---   
Target Release: 4.4.z   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: 1819053 Environment:
Last Closed: 2020-04-20 07:50:24 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 1819053    
Bug Blocks:    

Description Xingxing Xia 2020-04-03 07:07:04 UTC
As shown in the comment, 4.4 also reproduces. The log spam is so much, seen in daily build test, that I think it is worthy to address in 4.4 too, not must within 4.4.0, even to be addressed in 4.4.z is very expected. Thus cloning a 4.4.z one.

+++ This bug was initially created as a clone of Bug #1819053 +++

Clusters deployed with the 1.18 origin rebase PR [1] have the following in the apiserver logs every 60s:

controller.go:114] loading OpenAPI spec for "v1beta1.metrics.k8s.io" failed with: OpenAPI spec does not exist
controller.go:127] OpenAPI AggregationController: action for item v1beta1.metrics.k8s.io: Rate Limited Requeue.

This problem doesn't appear to be limited to the rebase PR, as per a longstanding github issue [2].

Beyond eliminating the log spew, it would be helpful to know what functional impact - if any - these logged errors represent.

1: https://github.com/openshift/origin/pull/24719
2: https://github.com/coreos/kube-prometheus/issues/304

--- Additional comment from Xingxing Xia on 2020-03-31 17:08:18 CST ---

4.4.0-0.nightly-2020-03-29-132004 also reproduces:
$ oc logs -n openshift-kube-apiserver kube-apiserver-ip-10-0-134-38.ap-northeast-2.compute.internal -c kube-apiserver
...
I0331 02:35:46.842201       1 aggregator.go:226] Updating OpenAPI spec because v1.route.openshift.io is updated
I0331 02:35:47.317974       1 aggregator.go:229] Finished OpenAPI spec generation after 475.760307ms
I0331 02:35:47.318047       1 aggregator.go:226] Updating OpenAPI spec because v1beta1.metrics.k8s.io is updated
I0331 02:35:47.818875       1 aggregator.go:229] Finished OpenAPI spec generation after 500.807236ms
I0331 02:35:47.818956       1 aggregator.go:226] Updating OpenAPI spec because v1.user.openshift.io is updated
I0331 02:35:48.273538       1 aggregator.go:229] Finished OpenAPI spec generation after 454.560076ms
...snipped...
I0331 02:35:59.218097       1 aggregator.go:226] Updating OpenAPI spec because v1.route.openshift.io is updated
I0331 02:36:00.378834       1 aggregator.go:229] Finished OpenAPI spec generation after 1.160710014s
E0331 02:36:00.382542       1 controller.go:114] loading OpenAPI spec for "v1beta1.metrics.k8s.io" failed with: OpenAPI spec does not exist
I0331 02:36:00.382560       1 controller.go:127] OpenAPI AggregationController: action for item v1beta1.metrics.k8s.io: Rate Limited Requeue.
I0331 02:36:00.649455       1 aggregator.go:226] Updating OpenAPI spec because v1.user.openshift.io is updated
...snipped...
I0331 02:36:08.140496       1 aggregator.go:226] Updating OpenAPI spec because v1.template.openshift.io is updated                     
I0331 02:36:09.401918       1 aggregator.go:229] Finished OpenAPI spec generation after 1.261388097s                                   
E0331 02:37:00.386352       1 controller.go:114] loading OpenAPI spec for "v1beta1.metrics.k8s.io" failed with: OpenAPI spec does not exist
I0331 02:37:00.386371       1 controller.go:127] OpenAPI AggregationController: action for item v1beta1.metrics.k8s.io: Rate Limited Requeue.
E0331 02:39:00.397969       1 controller.go:114] loading OpenAPI spec for "v1beta1.metrics.k8s.io" failed with: OpenAPI spec does not exist
I0331 02:39:00.397986       1 controller.go:127] OpenAPI AggregationController: action for item v1beta1.metrics.k8s.io: Rate Limited Requeue.
E0331 02:40:44.528108       1 controller.go:114] loading OpenAPI spec for "v1beta1.metrics.k8s.io" failed with: OpenAPI spec does not exist
I0331 02:40:44.528125       1 controller.go:127] OpenAPI AggregationController: action for item v1beta1.metrics.k8s.io: Rate Limited Requeue.
E0331 02:41:44.536408       1 controller.go:114] loading OpenAPI spec for "v1beta1.metrics.k8s.io" failed with: OpenAPI spec does not exist
I0331 02:41:44.536428       1 controller.go:127] OpenAPI AggregationController: action for item v1beta1.metrics.k8s.io: Rate Limited Requeue.
E0331 02:43:44.543085       1 controller.go:114] loading OpenAPI spec for "v1beta1.metrics.k8s.io" failed with: OpenAPI spec does not exist
...all following logs are only above spams now...

--- Additional comment from Xingxing Xia on 2020-03-31 17:12:34 CST ---

Like bug 1537270#c33 / bug 1819103 log spams annoying customer, this bug also annoys I think. Thus updating bug field.