Created attachment 1811132 [details] must-gather logs Created attachment 1811132 [details] must-gather logs Created attachment 1811132 [details] must-gather logs Created attachment 1811132 [details] must-gather logs Description of problem: We want to ensure that metrics scraping can always work if the scraper can reach the target, even if the kube-apiserver is unavailable or unreachable. But this is not the fact in my testing, see below. Version-Release number of selected component (if applicable): Latest 4.9 nightly payload How reproducible: Always Steps to Reproduce: 1. Get the certificate: oc extract secret/metrics-client-certs -n openshift-monitoring 2. Get IPs of metrics targets in advance for use in later steps: oc get endpoints -A > filename.txt oc get node -o wide > nodes_ip.txt 3. ssh to all masters and try below on all masters: sudo mv /etc/kubernetes/manifests/kube-apiserver-pod.yaml /tmp Wait for a while (about 135s) for the kube-apiserver container to shutdown by watching: sudo crictl ps -a --name="^kube-apiserver$" 4. Copy tls.crt and tls.key on /tmp of all masters. 5. Given all kube-apiservers are unavailable after previous step, gather metrics by using cert instead of token. Take the target openshift-apiserver for instance, get its IPs from previous filename.txt, and try below on one master: # curl -k --key /tmp/tls.key --cert /tmp/tls.crt "https://<openshift-apiserver pod IP>:8443/metrics" > /tmp/metrics.txt Try similarly for targets of below namespaces: openshift-controller-manager, openshift-kube-scheduler, openshift-kube-controller-manager, openshift-etcd Actual results: 5. It still tries to reach kube-apiserver for authorization like below. This means local authorization does not work. # curl -k --key /tmp/tls.key --cert /tmp/tls.crt "https://10.128.0.41:8443/metrics" Internal Server Error: "/metrics": Post "https://172.30.0.1:443/apis/authorization.k8s.io/v1/subjectaccessreviews?timeout=10s": dial tcp 172.30.0.1:443: connect: connection refused Expected results: Able to gather metrics Additional info:
reassigning to workloads team as these components are not owned by auth.
*** Bug 1991900 has been marked as a duplicate of this bug. ***
to clarify: to fully implement client cert based metrics scraping, both subjectaccessreview (replaced with a local static authorizer) and tokenreview (replaced with client certs) must be implemented.
Hi @ravig, Please update as soon as possible. If the bug is not fixed, this epic can't be said it implements the static local function well. Please provide the fix for bug as it is blocker for 4.9 release.
Based on comment5 setting Flag to blocker?
this is not a blocker, there is no degradation in functionality.
Targeting this BZ to 4.10 as this is not a blocker for 4.9
This bug hasn't had any activity in the last 30 days. Maybe the problem got resolved, was a duplicate of something else, or became less pressing for some reason - or maybe it's still relevant but just hasn't been looked at yet. As such, we're marking this bug as "LifecycleStale" and decreasing the severity/priority. If you have further information on the current state of the bug, please update it, otherwise this bug can be closed in about 7 days. The information can be, for example, that the problem still occurs, that you still want the feature, that more information is needed, or that the bug is (for whatever reason) no longer relevant. Additionally, you can add LifecycleFrozen into Whiteboard if you think this bug should never be marked as stale. Please consult with bug assignee before you do that.
Hi Michal, The bug is not fixed. @ravig: Please update
@rgudimet Please update on this.
I did not get enough time to work on this during this release considering this is not a blocker.
Hi Rahul, > We want to ensure that metrics scraping can always work if the scraper can reach the target, even if the kube-apiserver is unavailable or unreachable. > If the bug is not fixed, this epic can't be said it implements the static local function well would you please point me to the epic?
Ported into https://issues.redhat.com/browse/WRKLDS-648