Bug 1951705 - kube-apiserver needs alerts on CPU utlization
Summary: kube-apiserver needs alerts on CPU utlization
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: kube-apiserver
Version: 4.8
Hardware: Unspecified
OS: Unspecified
high
high
Target Milestone: ---
: 4.8.0
Assignee: David Eads
QA Contact: Ke Wang
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2021-04-20 19:09 UTC by David Eads
Modified: 2021-07-27 23:02 UTC (History)
3 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2021-07-27 23:02:18 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Github openshift origin pull 26099 0 None open bug 1951705: allow HighOverallControlPlaneCPU during e2e runs 2021-04-22 13:47:46 UTC
Red Hat Product Errata RHSA-2021:2438 0 None None None 2021-07-27 23:02:45 UTC

Description David Eads 2021-04-20 19:09:48 UTC
This will give us warning before an individual kube-apiserver fails under load and if the cluster cannot sustain the load if a single member fails.

Comment 4 Ke Wang 2021-05-06 03:54:06 UTC
$ oc get clusterversion
NAME      VERSION                             AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.8.0-0.nightly-2021-04-30-201824   True        False         88m     Cluster version is 4.8.0-0.nightly-2021-04-30-201824

$ oc get prometheusrules.monitoring.coreos.com/cpu-utilization -n openshift-kube-apiserver -oyaml
apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  creationTimestamp: "2021-05-06T02:03:48Z"
  generation: 1
  managedFields:
  - apiVersion: monitoring.coreos.com/v1
    fieldsType: FieldsV1
    fieldsV1:
      f:spec:
        .: {}
        f:groups: {}
    manager: cluster-kube-apiserver-operator
    operation: Update
    time: "2021-05-06T02:03:48Z"
  name: cpu-utilization
  namespace: openshift-kube-apiserver
  resourceVersion: "17201"
  uid: ce838b0c-07b0-47e7-a915-4ab4ef4ce2a4
spec:
  groups:
  - name: control-plane-cpu-utilization
    rules:
    - alert: HighOverallControlPlaneCPU
      expr: |
        sum(
          100 - (avg by (instance) (rate(node_cpu_seconds_total{mode="idle"}[5m])) * 100)
          AND on (instance) label_replace( kube_node_role{role="master"}, "instance", "$1", "node", "(.+)" )
        )
        /
        count(kube_node_role{role="master"})
        > 60
      for: 10m
      labels:
        severity: warning
    - alert: ExtremelyHighIndividualControlPlaneCPU
      expr: |
        100 - (avg by (instance) (rate(node_cpu_seconds_total{mode="idle"}[5m])) * 100) > 90 AND on (instance) label_replace( kube_node_role{role="master"}, "instance", "$1", "node", "(.+)" )
      for: 5m
      labels:
        severity: critical

The HighOverallControlPlaneCPU has already been applied, also can be seen in web-console.

Comment 7 errata-xmlrpc 2021-07-27 23:02:18 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: OpenShift Container Platform 4.8.2 bug fix and security update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2021:2438


Note You need to log in before you can comment on or make changes to this bug.