Bug 1916392

Summary: scrape priority and fairness endpoints for must-gather
Product: OpenShift Container Platform Reporter: Lukasz Szaszkiewicz <lszaszki>
Component: kube-apiserverAssignee: Robin Cernin <rcernin>
Status: CLOSED ERRATA QA Contact: Ke Wang <kewang>
Severity: medium Docs Contact:
Priority: medium    
Version: 4.8CC: aos-bugs, mfojtik, rcernin, xxia
Target Milestone: ---   
Target Release: 4.8.0   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: No Doc Update
Doc Text:
Added must-gather script to scrape priority and fairness debug endpoints.
 Story Points: ---
Clone Of: Environment:
Last Closed: 2021-07-27 22:36:14 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Lukasz Szaszkiewicz 2021-01-14 16:58:58 UTC 
API Priority and Fairness exposes a set of debug endpoints (HTTP) that could help us troubleshoot p&f related issues.
During must-gather we could scrape these endpoints and save the output to a file(s) and include it in the final must-gather bundle.

The full list of endpoints can be found at https://kubernetes.io/docs/concepts/cluster-administration/flow-control/#debug-endpoints
Comment 1 Lukasz Szaszkiewicz 2021-01-15 10:27:14 UTC 
@Robin feel free to assign it to you if you are interested.
Comment 2 Robin Cernin 2021-01-15 12:14:18 UTC 
Just added relevant endpoints to the Must-gather https://github.com/openshift/must-gather/pull/204

All debug endpoints, not the detailed ones, do you want detailed ones too?
Comment 3 Lukasz Szaszkiewicz 2021-01-15 14:55:27 UTC 
(In reply to Robin Cernin from comment #2)
> Just added relevant endpoints to the Must-gather
> https://github.com/openshift/must-gather/pull/204
> 
> All debug endpoints, not the detailed ones, do you want detailed ones too?

The ones you added are the ones I wanted. What do you mean by detailed endpoints?
Comment 5 Ke Wang 2021-03-17 07:05:47 UTC 
Verified with the following OCP payload,

$ oc get clusterversion
NAME      VERSION                             AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.8.0-0.nightly-2021-03-16-221720   True        False         151m    Cluster version is 4.8.0-0.nightly-2021-03-16-221720

$ oc adm must-gather | grep  priority_and_fairness
[must-gather-rtcjb] POD INFO: Collecting /debug/api_priority_and_fairness endpoint from kube-apiserver-ip-10-0-140-229.us-east-2.compute.internal
[must-gather-rtcjb] POD INFO: Collecting /debug/api_priority_and_fairness endpoint from kube-apiserver-ip-10-0-176-115.us-east-2.compute.internal
[must-gather-rtcjb] POD INFO: Collecting /debug/api_priority_and_fairness endpoint from kube-apiserver-ip-10-0-221-102.us-east-2.compute.internal
[must-gather-rtcjb] OUT namespaces/openshift-kube-apiserver/pods/kube-apiserver-ip-10-0-140-229.us-east-2.compute.internal/kube-apiserver/kube-apiserver/api_priority_and_fairness/
[must-gather-rtcjb] OUT namespaces/openshift-kube-apiserver/pods/kube-apiserver-ip-10-0-140-229.us-east-2.compute.internal/kube-apiserver/kube-apiserver/api_priority_and_fairness/priority_levels
[must-gather-rtcjb] OUT namespaces/openshift-kube-apiserver/pods/kube-apiserver-ip-10-0-140-229.us-east-2.compute.internal/kube-apiserver/kube-apiserver/api_priority_and_fairness/queues
[must-gather-rtcjb] OUT namespaces/openshift-kube-apiserver/pods/kube-apiserver-ip-10-0-140-229.us-east-2.compute.internal/kube-apiserver/kube-apiserver/api_priority_and_fairness/requests
[must-gather-rtcjb] OUT namespaces/openshift-kube-apiserver/pods/kube-apiserver-ip-10-0-176-115.us-east-2.compute.internal/kube-apiserver/kube-apiserver/api_priority_and_fairness/
[must-gather-rtcjb] OUT namespaces/openshift-kube-apiserver/pods/kube-apiserver-ip-10-0-176-115.us-east-2.compute.internal/kube-apiserver/kube-apiserver/api_priority_and_fairness/priority_levels
[must-gather-rtcjb] OUT namespaces/openshift-kube-apiserver/pods/kube-apiserver-ip-10-0-176-115.us-east-2.compute.internal/kube-apiserver/kube-apiserver/api_priority_and_fairness/queues
[must-gather-rtcjb] OUT namespaces/openshift-kube-apiserver/pods/kube-apiserver-ip-10-0-176-115.us-east-2.compute.internal/kube-apiserver/kube-apiserver/api_priority_and_fairness/requests
[must-gather-rtcjb] OUT namespaces/openshift-kube-apiserver/pods/kube-apiserver-ip-10-0-221-102.us-east-2.compute.internal/kube-apiserver/kube-apiserver/api_priority_and_fairness/
[must-gather-rtcjb] OUT namespaces/openshift-kube-apiserver/pods/kube-apiserver-ip-10-0-221-102.us-east-2.compute.internal/kube-apiserver/kube-apiserver/api_priority_and_fairness/priority_levels
[must-gather-rtcjb] OUT namespaces/openshift-kube-apiserver/pods/kube-apiserver-ip-10-0-221-102.us-east-2.compute.internal/kube-apiserver/kube-apiserver/api_priority_and_fairness/queues
[must-gather-rtcjb] OUT namespaces/openshift-kube-apiserver/pods/kube-apiserver-ip-10-0-221-102.us-east-2.compute.internal/kube-apiserver/kube-apiserver/api_priority_and_fairness/requests

From above, we can get the priority and fairness exposing a set of debug endpoints (HTTP) in must-gather bundle, so move the bug VERIFIED.
Comment 8 errata-xmlrpc 2021-07-27 22:36:14 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: OpenShift Container Platform 4.8.2 bug fix and security update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2021:2438