Bug 1819053 - loading OpenAPI spec for "v1beta1.metrics.k8s.io" failed with: OpenAPI spec does not exist
Summary: loading OpenAPI spec for "v1beta1.metrics.k8s.io" failed with: OpenAPI spec d...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Monitoring
Version: 4.5
Hardware: Unspecified
OS: Unspecified
high
low
Target Milestone: ---
: 4.7.0
Assignee: Sergiusz Urbaniak
QA Contact: hongyan li
URL:
Whiteboard: groom
: 1820468 (view as bug list)
Depends On:
Blocks: 1820468
TreeView+ depends on / blocked
 
Reported: 2020-03-31 05:14 UTC by Maru Newby
Modified: 2021-07-26 09:12 UTC (History)
21 users (show)

Fixed In Version:
Doc Type: Release Note
Doc Text:
prometheus-adapter did not implement the OpenAPI spec. Thus, the API server logs a message every 60 seconds that the OpenAPI does not exist while the Prometheus Adapter is deployed into the cluster. The `KubeAPIErrorsHigh` alert might also fire due to the errors in the logs. There is no known degraded functionality to the monitoring stack due to this known issue. This bugzilla fixes it by introducing the OpenAPI spec into prometheus-adapter to comply with other core API resources within Openshift.
Clone Of:
: 1820468 (view as bug list)
Environment:
Last Closed: 2021-02-24 15:10:58 UTC
Target Upstream Version:

Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Github openshift k8s-prometheus-adapter pull 32 0 None closed Bug 1819053: bump to v0.8.0 2021-02-21 07:17:29 UTC
Red Hat Knowledge Base (Solution) 4621371 0 None None None 2020-11-20 20:22:09 UTC
Red Hat Product Errata RHSA-2020:5633 0 None None None 2021-02-24 15:11:53 UTC

Description Maru Newby 2020-03-31 05:14:00 UTC 
Clusters deployed with the 1.18 origin rebase PR [1] have the following in the apiserver logs every 60s:

controller.go:114] loading OpenAPI spec for "v1beta1.metrics.k8s.io" failed with: OpenAPI spec does not exist
controller.go:127] OpenAPI AggregationController: action for item v1beta1.metrics.k8s.io: Rate Limited Requeue.

This problem doesn't appear to be limited to the rebase PR, as per a longstanding github issue [2].

Beyond eliminating the log spew, it would be helpful to know what functional impact - if any - these logged errors represent.

1: https://github.com/openshift/origin/pull/24719
2: https://github.com/coreos/kube-prometheus/issues/304
Comment 1 Xingxing Xia 2020-03-31 09:08:18 UTC 
4.4.0-0.nightly-2020-03-29-132004 also reproduces:
$ oc logs -n openshift-kube-apiserver kube-apiserver-ip-10-0-134-38.ap-northeast-2.compute.internal -c kube-apiserver
...
I0331 02:35:46.842201       1 aggregator.go:226] Updating OpenAPI spec because v1.route.openshift.io is updated
I0331 02:35:47.317974       1 aggregator.go:229] Finished OpenAPI spec generation after 475.760307ms
I0331 02:35:47.318047       1 aggregator.go:226] Updating OpenAPI spec because v1beta1.metrics.k8s.io is updated
I0331 02:35:47.818875       1 aggregator.go:229] Finished OpenAPI spec generation after 500.807236ms
I0331 02:35:47.818956       1 aggregator.go:226] Updating OpenAPI spec because v1.user.openshift.io is updated
I0331 02:35:48.273538       1 aggregator.go:229] Finished OpenAPI spec generation after 454.560076ms
...snipped...
I0331 02:35:59.218097       1 aggregator.go:226] Updating OpenAPI spec because v1.route.openshift.io is updated
I0331 02:36:00.378834       1 aggregator.go:229] Finished OpenAPI spec generation after 1.160710014s
E0331 02:36:00.382542       1 controller.go:114] loading OpenAPI spec for "v1beta1.metrics.k8s.io" failed with: OpenAPI spec does not exist
I0331 02:36:00.382560       1 controller.go:127] OpenAPI AggregationController: action for item v1beta1.metrics.k8s.io: Rate Limited Requeue.
I0331 02:36:00.649455       1 aggregator.go:226] Updating OpenAPI spec because v1.user.openshift.io is updated
...snipped...
I0331 02:36:08.140496       1 aggregator.go:226] Updating OpenAPI spec because v1.template.openshift.io is updated                     
I0331 02:36:09.401918       1 aggregator.go:229] Finished OpenAPI spec generation after 1.261388097s                                   
E0331 02:37:00.386352       1 controller.go:114] loading OpenAPI spec for "v1beta1.metrics.k8s.io" failed with: OpenAPI spec does not exist
I0331 02:37:00.386371       1 controller.go:127] OpenAPI AggregationController: action for item v1beta1.metrics.k8s.io: Rate Limited Requeue.
E0331 02:39:00.397969       1 controller.go:114] loading OpenAPI spec for "v1beta1.metrics.k8s.io" failed with: OpenAPI spec does not exist
I0331 02:39:00.397986       1 controller.go:127] OpenAPI AggregationController: action for item v1beta1.metrics.k8s.io: Rate Limited Requeue.
E0331 02:40:44.528108       1 controller.go:114] loading OpenAPI spec for "v1beta1.metrics.k8s.io" failed with: OpenAPI spec does not exist
I0331 02:40:44.528125       1 controller.go:127] OpenAPI AggregationController: action for item v1beta1.metrics.k8s.io: Rate Limited Requeue.
E0331 02:41:44.536408       1 controller.go:114] loading OpenAPI spec for "v1beta1.metrics.k8s.io" failed with: OpenAPI spec does not exist
I0331 02:41:44.536428       1 controller.go:127] OpenAPI AggregationController: action for item v1beta1.metrics.k8s.io: Rate Limited Requeue.
E0331 02:43:44.543085       1 controller.go:114] loading OpenAPI spec for "v1beta1.metrics.k8s.io" failed with: OpenAPI spec does not exist
...all following logs are only above spams now...
Comment 2 Xingxing Xia 2020-03-31 09:12:34 UTC 
Like bug 1537270#c33 / bug 1819103 log spams annoying customer, this bug also annoys I think. Thus updating bug field.
Comment 3 Sergiusz Urbaniak 2020-04-20 07:50:24 UTC 
*** Bug 1820468 has been marked as a duplicate of this bug. ***
Comment 17 Russell Bryant 2020-08-25 19:43:43 UTC 
FWIW, I just hit this issue in my metal UPI cluster after upgrading from 4.5.5 to 4.5.7.  My cluster is firing the KubeAPIErrorsHigh alert, and the kube-apiserver log has the same messages from this report.
Comment 18 Russell Bryant 2020-08-25 19:46:05 UTC 
(In reply to Russell Bryant from comment #17)
> FWIW, I just hit this issue in my metal UPI cluster after upgrading from
> 4.5.5 to 4.5.7.  My cluster is firing the KubeAPIErrorsHigh alert, and the
> kube-apiserver log has the same messages from this report.

actually it's not clear the alert and this log spam is related.  I just happened to see them mentioned together in a slack conversation, which led me to this bug.
Comment 19 Sergiusz Urbaniak 2020-09-08 09:37:30 UTC 
KubeAPIErrorsHigh and this issue are not related.
Comment 20 Sergiusz Urbaniak 2020-09-09 07:40:40 UTC 
Setting priority to high, leaving severity to low as there is no functional impact.

Priority is high as this also affects upstream functionality on a broader sense, the metrics-server API library is used in many places.
Comment 26 Junqi Zhao 2020-10-09 00:50:51 UTC 
reproduced on 4.6.0-0.nightly-2020-10-08-182439
# oc -n openshift-kube-apiserver logs kube-apiserver-ip-10-0-131-187.eu-west-1.compute.internal -c kube-apiserver | grep "v1beta1.metrics.k8s.io" | tail -n 5
I1009 00:45:33.398180      17 controller.go:129] OpenAPI AggregationController: action for item v1beta1.metrics.k8s.io: Rate Limited Requeue.
E1009 00:47:33.401701      17 controller.go:116] loading OpenAPI spec for "v1beta1.metrics.k8s.io" failed with: OpenAPI spec does not exist
I1009 00:47:33.401718      17 controller.go:129] OpenAPI AggregationController: action for item v1beta1.metrics.k8s.io: Rate Limited Requeue.
E1009 00:49:33.396989      17 controller.go:116] loading OpenAPI spec for "v1beta1.metrics.k8s.io" failed with: OpenAPI spec does not exist
I1009 00:49:33.397006      17 controller.go:129] OpenAPI AggregationController: action for item v1beta1.metrics.k8s.io: Rate Limited Requeue.
Comment 32 hongyan li 2020-10-27 08:28:59 UTC 
Still not in any payload by now.
Comment 33 hongyan li 2020-10-28 06:08:33 UTC 
Issue exists in payload 4.7.0-0.nightly-2020-10-27-051128
Comment 34 Junqi Zhao 2020-11-05 07:47:16 UTC 
tested with 4.7.0-0.nightly-2020-11-05-010603, no "OpenAPI spec does not exist" error now
# for i in $(oc -n openshift-kube-apiserver get pod | grep kube-apiserver | awk '{print $1}'); do echo $i; oc -n openshift-kube-apiserver logs $i -c kube-apiserver | grep "OpenAPI spec does not exist" | tail -n 5; echo -e "\n";done
kube-apiserver-ip-10-0-158-63.ap-southeast-2.compute.internal


kube-apiserver-ip-10-0-180-210.ap-southeast-2.compute.internal


kube-apiserver-ip-10-0-206-75.ap-southeast-2.compute.internal
Comment 43 errata-xmlrpc 2021-02-24 15:10:58 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: OpenShift Container Platform 4.7.0 security, bug fix, and enhancement update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2020:5633
Note You need to log in before you can comment on or make changes to this bug.