Bug 1948703 - p&f: probes should not get 429s
Summary: p&f: probes should not get 429s
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: kube-apiserver
Version: 4.7
Hardware: Unspecified
OS: Unspecified
Target Milestone: ---
: 4.8.0
Assignee: Abu Kashem
QA Contact: Ke Wang
Depends On: 1956606
Blocks: 1937916
TreeView+ depends on / blocked
Reported: 2021-04-12 18:49 UTC by Abu Kashem
Modified: 2021-07-27 22:59 UTC (History)
9 users (show)

Fixed In Version:
Doc Type: No Doc Update
Doc Text:
Clone Of: 1937916
Last Closed: 2021-07-27 22:59:27 UTC
Target Upstream Version:

Attachments (Terms of Use)

System ID Private Priority Status Summary Last Updated
Github openshift kubernetes pull 656 0 None open BUG Bug 1948703: UPSTREAM: 100678: apf: exempt probes /healthz /livez /readyz 2021-04-12 19:00:30 UTC
Red Hat Product Errata RHSA-2021:2438 0 None None None 2021-07-27 22:59:43 UTC

Comment 1 Abu Kashem 2021-04-12 19:02:49 UTC
I don't see any '4.9' yet, so using '4.8' for now for target release.

Comment 2 W. Trevor King 2021-04-12 19:58:50 UTC
> I don't see any '4.9' yet, so using '4.8' for now for target release.

4.8 is in feature freeze, but release-4.8 branches are still being fast-forwarded to track main/master until closer to code freeze.  So everything that is landing now is still going out with 4.8 when it GAs, and 4.8.0 as a target release is appropriate.

Comment 4 Ke Wang 2021-04-16 06:42:39 UTC
$ oc get clusterversion
NAME      VERSION                             AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.8.0-0.nightly-2021-04-15-202330   True        False         118m    Cluster version is 4.8.0-0.nightly-2021-04-15-202330

$ oc get flowschema | grep probe
probes                              exempt                              2                    <none>                142m   False

$ oc edit  kubeapiserver/cluster # change the loglevel to TraceAll
kubeapiserver.operator.openshift.io/cluster edited

After the kube-apiservers fnished the restarting, make some readyz requests to the apiserver,

$ for i in {1..30}; do curl -k https://api.kewang-man1681.qe.gcp.devcluster.openshift.com:6443/readyz;done

$ kas_pods=$(oc get pods -n openshift-kube-apiserver | grep 'kube-apiserver' | awk '{print $1}'); for pod in $kas_pods; do oc -n openshift-kube-apiserver logs $pod -c kube-apiserver | grep 'exempt' | grep 'readyz' | head -1;done

I0416 06:05:33.220514      20 apf_controller.go:792] startRequest(RequestDigest{RequestInfo: &request.RequestInfo{IsResourceRequest:false, Path:"/readyz", Verb:"get", APIPrefix:"", APIGroup:"", APIVersion:"", Namespace:"", Resource:"", Subresource:"", Name:"", Parts:[]string(nil)}, User: &user.DefaultInfo{Name:"system:anonymous", UID:"", Groups:[]string{"system:unauthenticated"}, Extra:map[string][]string(nil)}}) => fsName="probes", distMethod=(*v1beta1.FlowDistinguisherMethod)(nil), plName="exempt", immediate

The new flowschema probes works as expected, so move the bug VERIFIED.

Comment 5 Abu Kashem 2021-05-14 14:11:52 UTC
https://github.com/openshift/cluster-kube-apiserver-operator/pull/1099 must get in first, we need to revert the flow schema from the 'Manifests' folder first, otherwise we don't know who is creating the flow schema object, is it cvo or kube-apiserer?

(This is due to the fact that we did not wait for the upstream PR to merge, we went ahead and added a similar flowschema in manifests folder)

Comment 6 W. Trevor King 2021-05-14 18:28:01 UTC
> ... we need to revert the flow schema from the 'Manifests' folder first, otherwise we don't know who is creating the flow schema...

No need to block on 1099, because, as bug 1956606 points out, the manifest is being ignored by the CVO and has been since it landed.

Comment 9 errata-xmlrpc 2021-07-27 22:59:27 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: OpenShift Container Platform 4.8.2 bug fix and security update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.


Note You need to log in before you can comment on or make changes to this bug.