Bug 1798215 - Send apiserver request-in-flight metrics to telemeter
Summary: Send apiserver request-in-flight metrics to telemeter
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: openshift-apiserver
Version: 4.4
Hardware: Unspecified
OS: Unspecified
unspecified
unspecified
Target Milestone: ---
: 4.4.0
Assignee: Abu Kashem
QA Contact: Ke Wang
URL:
Whiteboard:
Depends On: 1798214 1799057
Blocks:
TreeView+ depends on / blocked
 
Reported: 2020-02-04 20:23 UTC by Abu Kashem
Modified: 2020-05-04 11:33 UTC (History)
4 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of: 1798214
Environment:
Last Closed: 2020-05-04 11:33:13 UTC
Target Upstream Version:


Attachments (Terms of Use)


Links
System ID Priority Status Summary Last Updated
Github openshift cluster-openshift-apiserver-operator pull 307 None closed Bug 1798215: Add a new label 'apiserver' to identify the source 2020-03-02 09:32:33 UTC
Red Hat Product Errata RHBA-2020:0581 None None None 2020-05-04 11:33:48 UTC

Description Abu Kashem 2020-02-04 20:23:25 UTC
Send apiserver request-in-flight metrics to telemeter

We want to have an idea of how loaded
our api server(s) are. Use the metric apiserver_current_inflight_requests to look at the peak of the number of requests in flight over time.

Similar request for kube-apiserver: Bug #1798214

Comment 2 Ke Wang 2020-03-05 04:25:15 UTC
Verified with the following OCP env,
$ oc version
Client Version: 4.4.0-202002282323-bc08a48
Server Version: 4.4.0-0.nightly-2020-03-04-143604
Kubernetes Version: v1.17.1

Verification steps,

1. Check if the code changes of PR https://github.com/openshift/cluster-openshift-apiserver-operator/pull/307 in,
$ oc get ServiceMonitor -n openshift-kube-apiserver -o yaml
apiVersion: v1
items:
- apiVersion: monitoring.coreos.com/v1
  kind: ServiceMonitor
  ...
relabelings:
      - action: replace
        replacement: openshift-apiserver
        targetLabel: apiserver
  ...

$ oc get PrometheusRule -n openshift-kube-apiserver -o yaml
...
   - name: apiserver-requests-in-flight
      rules:
      - expr: |
          max_over_time(sum(apiserver_current_inflight_requests{apiserver=~"openshift-apiserver|kube-apiserver"}) by (apiserver,requestKind)[2m:])
        record: cluster:apiserver_current_inflight_requests:sum:max_over_time:2m
...

$ oc -n openshift-monitoring get cm telemetry-config -oyaml | grep "cluster:apiserver_current_inflight_requests:sum:max_over_time:2m" 
    # cluster:apiserver_current_inflight_requests:sum:max_over_time:2m gives maximum number of requests in flight
    - '{__name__="cluster:apiserver_current_inflight_requests:sum:max_over_time:2m"}'

The code changes are checked as expected.

2. Check if the feature work fine with Metrics.

Open the OCP cluster web console, on the left panel, navigate to Monitoring-> Metrics,  enter the keyword ‘cluster:apiserver_current_inflight_requests:sum:max_over_time:2m’ in query textarea of displayed page , click on ‘Run  Queries’,
four items of openshift-apiserver and kube-apiserver are displayed, at column Value, we can see the requests number in 2 minutes.

Element 	                                                                                                             Value                                                                                                                                                                                      
cluster:apiserver_current_inflight_requests:sum:max_over_time:2m{apiserver="kube-apiserver",requestKind="mutating"}	      4
cluster:apiserver_current_inflight_requests:sum:max_over_time:2m{apiserver="kube-apiserver",requestKind="readOnly"}	      6
cluster:apiserver_current_inflight_requests:sum:max_over_time:2m{apiserver="openshift-apiserver",requestKind="mutating"}      1
cluster:apiserver_current_inflight_requests:sum:max_over_time:2m{apiserver="openshift-apiserver",requestKind="readOnly"}      3

We will see the feature work as expected.

Comment 4 errata-xmlrpc 2020-05-04 11:33:13 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:0581


Note You need to log in before you can comment on or make changes to this bug.