Bug 1872337 - KubeletHealthState alert keeps firing [4.5.z]
Summary: KubeletHealthState alert keeps firing [4.5.z]
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Node
Version: 4.3.z
Hardware: Unspecified
OS: Unspecified
high
high
Target Milestone: ---
: 4.5.z
Assignee: Seth Jennings
QA Contact: Sunil Choudhary
URL:
Whiteboard:
Depends On: 1854009
Blocks: 1878798
TreeView+ depends on / blocked
 
Reported: 2020-08-25 14:09 UTC by OpenShift BugZilla Robot
Modified: 2023-12-15 19:00 UTC (History)
18 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Fixes an issue where once the KubeletHealthState fires from a particular kubelet, it stays active even after the kubelet becomes healthy again. Only a kubelet restart could clear the alert.
Clone Of:
Environment:
Last Closed: 2020-09-21 17:42:05 UTC
Target Upstream Version:
Embargoed:
palonsor: needinfo-


Attachments (Terms of Use)
metrics-fix-confirmation (149.48 KB, image/png)
2020-09-14 09:58 UTC, Sunil Choudhary
no flags Details


Links
System ID Private Priority Status Summary Last Updated
Github openshift machine-config-operator pull 2023 0 None closed [release-4.5] Bug 1872337: remove err cardinality from mcd_kubelet_state metrics 2021-02-11 14:56:11 UTC
Red Hat Product Errata RHBA-2020:3719 0 None None None 2020-09-21 17:42:21 UTC

Comment 1 Seth Jennings 2020-08-25 17:06:23 UTC
increasing severity to match CU ticket on parent bz1854009

Comment 2 Seth Jennings 2020-08-25 17:07:45 UTC
MCO PR https://github.com/openshift/machine-config-operator/pull/2023

Comment 5 Sunil Choudhary 2020-09-14 09:57:11 UTC
Verified on 4.5.0-0.nightly-2020-09-12-063044. Stopped kubelet service on one node and see alerts firing and they were cleared once kubelet was started again.

$ oc get clusterversion
NAME      VERSION                             AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.5.0-0.nightly-2020-09-12-063044   True        False         144m    Cluster version is 4.5.0-0.nightly-2020-09-12-063044

$ oc get nodes
NAME                                         STATUS   ROLES    AGE    VERSION
ip-10-0-130-237.us-east-2.compute.internal   Ready    worker   154m   v1.18.3+b0068a8
ip-10-0-152-249.us-east-2.compute.internal   Ready    master   165m   v1.18.3+b0068a8
ip-10-0-167-191.us-east-2.compute.internal   Ready    master   166m   v1.18.3+b0068a8
ip-10-0-190-64.us-east-2.compute.internal    Ready    worker   154m   v1.18.3+b0068a8
ip-10-0-200-110.us-east-2.compute.internal   Ready    master   166m   v1.18.3+b0068a8
ip-10-0-220-35.us-east-2.compute.internal    Ready    worker   154m   v1.18.3+b0068a8

$ token=`oc sa get-token prometheus-k8s -n openshift-monitoring`

$ oc -n openshift-monitoring exec -c prometheus prometheus-k8s-0 -- curl -k -g -H "Authorization: Bearer $token" 'https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?query=mcd_kubelet_state>2' | jq
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   334  100   334    0     0   4517      0 --:--:-- --:--:-- --:--:--  4575
{
  "status": "success",
  "data": {
    "resultType": "vector",
    "result": [
      {
        "metric": {
          "__name__": "mcd_kubelet_state",
          "endpoint": "metrics",
          "instance": "10.0.130.237:9001",
          "job": "machine-config-daemon",
          "namespace": "openshift-machine-config-operator",
          "pod": "machine-config-daemon-xqddl",
          "service": "machine-config-daemon"
        },
        "value": [
          1600077068.81,
          "3"
        ]
      }
    ]
  }
}

$ oc -n openshift-monitoring exec -c prometheus prometheus-k8s-0 -- curl -k -g -H "Authorization: Bearer $token" 'https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?query=mcd_kubelet_state>2' | jq
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100    63  100    63    0     0    768      0 --:--:-- --:--:-- --:--:--   777
{
  "status": "success",
  "data": {
    "resultType": "vector",
    "result": []
  }
}

Comment 6 Sunil Choudhary 2020-09-14 09:58:14 UTC
Created attachment 1714740 [details]
metrics-fix-confirmation

Comment 10 errata-xmlrpc 2020-09-21 17:42:05 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (OpenShift Container Platform 4.5.11 bug fix update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:3719


Note You need to log in before you can comment on or make changes to this bug.