Bug 2091902

Description of problem:

Unable to delete any project/ns with following error:

unable to retrieve the complete list of server APIs: metrics.k8s.io/v1beta1: the server has received too many requests and has asked us to try again later


kube-apiserver-master01
...
2022-05-25T13:53:36.576108316Z E0525 13:53:36.575877      16 controller.go:116] loading OpenAPI spec for "v1beta1.metrics.k8s.io" failed with: failed to retrieve openAPI spec, http error: ResponseCode: 429, Body: Too many requests, please try again later.
2022-05-25T13:53:36.576108316Z I0525 13:53:36.575912      16 controller.go:129] OpenAPI AggregationController: action for item v1beta1.metrics.k8s.io: Rate Limited Requeue.
2022-05-25T13:54:36.585753602Z E0525 13:54:36.585663      16 controller.go:116] loading OpenAPI spec for "v1beta1.metrics.k8s.io" failed with: failed to retrieve openAPI spec, http error: ResponseCode: 429, Body: Too many requests, please try again later.
2022-05-25T13:54:36.585753602Z I0525 13:54:36.585680      16 controller.go:129] OpenAPI AggregationController: action for item v1beta1.metrics.k8s.io: Rate Limited Requeue.

kube-controller-manager-master01
...
2022-05-28T03:54:34.326992167Z E0528 03:54:34.326928       1 resource_quota_controller.go:413] unable to retrieve the complete list of server APIs: metrics.k8s.io/v1beta1: the server has received too many requests and has asked us to try again later
2022-05-28T03:54:35.175781100Z E0528 03:54:35.175731       1 memcache.go:101] couldn't get resource list for metrics.k8s.io/v1beta1: the server has received too many requests and has asked us to try again later


# Unable to access raw pod metrics API endpoint

+ oc get --v=9 --raw /apis/metrics.k8s.io/v1beta1/pods
I0530 11:53:35.984102   24912 loader.go:372] Config loaded from file:  .../kubeconfig
I0530 11:53:35.986361   24912 round_trippers.go:435] curl -k -v -XGET  -H "Accept: application/json, */*" -H "User-Agent: oc/4.8.0 (linux/amd64) kubernetes/a0c12be" 'https://api.ocp1-example.se:6443/apis/metrics.k8s.io/v1beta1/pods'
I0530 11:54:36.001605   24912 round_trippers.go:454] GET https://api.ocp1-example.se:6443/apis/metrics.k8s.io/v1beta1/pods  in 60015 milliseconds
I0530 11:54:36.001641   24912 round_trippers.go:460] Response Headers:
I0530 11:54:36.001773   24912 helpers.go:234] Connection error: Get https://api.ocp1-example.se:6443/apis/metrics.k8s.io/v1beta1/pods: stream error: stream ID 1; INTERNAL_ERROR
F0530 11:54:36.001818   24912 helpers.go:115] Unable to connect to the server: stream error: stream ID 1; INTERNAL_ERROR
goroutine 1 [running]:
k8s.io/klog/v2.stacks(0xc000010001, 0xc000d92840, 0x79, 0x28c)
        /go/src/github.com/openshift/oc/vendor/k8s.io/klog/v2/klog.go:1021 +0xb9
k8s.io/klog/v2.(*loggingT).output(0x5c17580, 0xc000000003, 0x0, 0x0, 0xc00077ad90, 0x4a41f6c, 0xa, 0x73, 0x11e0400)
        /go/src/github.com/openshift/oc/vendor/k8s.io/klog/v2/klog.go:970 +0x191
k8s.io/klog/v2.(*loggingT).printDepth(0x5c17580, 0xc000000003, 0x0, 0x0, 0x0, 0x0, 0x2, 0xc000a4e290, 0x1, 0x1)
        /go/src/github.com/openshift/oc/vendor/k8s.io/klog/v2/klog.go:733 +0x16f
k8s.io/klog/v2.FatalDepth(...)
        /go/src/github.com/openshift/oc/vendor/k8s.io/klog/v2/klog.go:1495
k8s.io/kubectl/pkg/cmd/util.fatal(0xc000e1c8c0, 0x4a, 0x1)
        /go/src/github.com/openshift/oc/vendor/k8s.io/kubectl/pkg/cmd/util/helpers.go:93 +0x288
k8s.io/kubectl/pkg/cmd/util.checkErr(0x42bc240, 0xc000542a80, 0x3fb1878)
        /go/src/github.com/openshift/oc/vendor/k8s.io/kubectl/pkg/cmd/util/helpers.go:188 +0x935
k8s.io/kubectl/pkg/cmd/util.CheckErr(...)
        /go/src/github.com/openshift/oc/vendor/k8s.io/kubectl/pkg/cmd/util/helpers.go:115
k8s.io/kubectl/pkg/cmd/get.NewCmdGet.func1(0xc000bc9180, 0xc000d95950, 0x0, 0x3)
        /go/src/github.com/openshift/oc/vendor/k8s.io/kubectl/pkg/cmd/get/get.go:167 +0x159
github.com/spf13/cobra.(*Command).execute(0xc000bc9180, 0xc000d95920, 0x3, 0x3, 0xc000bc9180, 0xc000d95920)
        /go/src/github.com/openshift/oc/vendor/github.com/spf13/cobra/command.go:856 +0x2c2
github.com/spf13/cobra.(*Command).ExecuteC(0xc000724000, 0x2, 0xc000724000, 0x2)
        /go/src/github.com/openshift/oc/vendor/github.com/spf13/cobra/command.go:960 +0x375
github.com/spf13/cobra.(*Command).Execute(...)
        /go/src/github.com/openshift/oc/vendor/github.com/spf13/cobra/command.go:897
main.main()
        /go/src/github.com/openshift/oc/cmd/oc/oc.go:93 +0x645

# Unable to get podmetrics from all namespaces

+ oc get podmetrics --all-namespaces
Unable to connect to the server: stream error: stream ID 1; INTERNAL_ERROR

# Access the nodes metrics API endpoint is  OK  


+ oc get --raw /apis/metrics.k8s.io/v1beta1/nodes
{"kind":"NodeMetricsList","apiVersion":"metrics.k8s.io/v1beta1","metadata":{},"items":[{"metadata":{"name":"master01-ocp.com","creationTimestamp":"2022-05-30T09:54:36Z","labels":{"beta.kubernetes.io/arch":"amd64","beta.kubernetes.io/os":"linux","kubernetes.io/arch":"amd64","kubernetes.io/hostname":"master01-ocp.com","kubernetes.io/os":"linux","node-role.kubernetes.io/master":"","node-role.kubernetes.io/worker":"","node.openshift.io/os_id":"rhcos","pod":"controller","pod":"non-controller"}},"timestamp":"2022-05-30T09:54:36Z","window":"1m0s","usage":{"cpu":"9922m","memory":"67077808Ki"}},{"metadata":{"name":"master02-ocp.com","creationTimestamp":"2022-05-30T09:54:36Z","labels":{"beta.kubernetes.io/arch":"amd64","beta.kubernetes.io/os":"linux","kubernetes.io/arch":"amd64","kubernetes.io/hostname":"master02-ocp.com","kubernetes.io/os":"linux","node-role.kubernetes.io/master":"","node-role.kubernetes.io/worker":"","node.openshift.io/os_id":"rhcos","pod":"controller","pod":"non-controller"}},"timestamp":"2022-05-30T09:54:36Z","window":"1m0s","usage":{"cpu":"7112m","memory":"44586456Ki"}},{"metadata":{"name":"master03-ocp.com","creationTimestamp":"2022-05-30T09:54:36Z","labels":{"beta.kubernetes.io/arch":"amd64","beta.kubernetes.io/os":"linux","kubernetes.io/arch":"amd64","kubernetes.io/hostname":"master03-ocp.com","kubernetes.io/os":"linux","node-role.kubernetes.io/master":"","node-role.kubernetes.io/worker":"","node.openshift.io/os_id":"rhcos","pcc-mm-pod":"non-controller","pcc-sm-pod":"non-controller"}},"timestamp":"2022-05-30T09:54:36Z","window":"1m0s","usage":{"cpu":"9436m","memory":"51704216Ki"}},{"metadata":{"name":"worker01-ocp.com","creationTimestamp":"2022-05-30T09:54:36Z","labels":{"beta.kubernetes.io/arch":"amd64","beta.kubernetes.io/os":"linux","kubernetes.io/arch":"amd64","kubernetes.io/hostname":"worker01-ocp.com","kubernetes.io/os":"linux","node-role.kubernetes.io/standard":"","node-role.kubernetes.io/worker":"","node.openshift.io/os_id":"rhcos","pod":"non-controller","pod":"non-controller"}},"timestamp":"2022-05-30T09:54:36Z","window":"1m0s","usage":{"cpu":"5223m","memory":"49409820Ki"}}....

After restart of `prometheus-adapter` Pods the deletion of project/ns is working OK however still unable to access podmetrics raw endpoint.



Version-Release number of selected component (if applicable):


How reproducible:


Steps to Reproduce:
Undefined

Actual results:


Expected results:

Metrics API do not impact project/ns creation/deletion process


Additional info: