Bug 1878798

Summary:

KubeletHealthState alert keeps firing [4.4.z]

Product:

OpenShift Container Platform

Reporter:

OpenShift BugZilla Robot <openshift-bugzilla-robot>

Component:

Node

Assignee:

Seth Jennings <sjenning>

Status:

CLOSED ERRATA

QA Contact:

Sunil Choudhary <schoudha>

Severity:

high

Docs Contact:

Priority:

medium

Version:

4.3.z

CC:

alegrand, anpicker, aos-bugs, erooth, jokerman, kakkoyun, kgarriso, lcosic, mloibl, palonsor, pkrupa, rrackow, schoudha, surbania, wking

Target Milestone:

---

Target Release:

4.4.z

Hardware:

Unspecified

OS:

Unspecified

Whiteboard:

Fixed In Version:

Doc Type:

Bug Fix

Doc Text:

Fixes an issue where once the KubeletHealthState fires from a particular kubelet, it stays active even after the kubelet becomes healthy again. Only a kubelet restart could clear the alert.

Story Points:

---

Clone Of:

Environment:

Last Closed:

2020-10-01 17:50:07 UTC

Type:

---

Regression:

---

Mount Type:

---

Documentation:

---

CRM:

Verified Versions:

Category:

---

oVirt Team:

---

RHEL 7.3 requirements from Atomic Host:

Cloudforms Team:

---

Target Upstream Version:

Embargoed:

Bug Depends On:

1872337

Bug Blocks:

1880207

Attachments:

Description	Flags
metrics-fix-confirmation	none

Comment 2 Sunil Choudhary 2020-09-22 07:02:55 UTC

Verified on 4.4.0-0.nightly-2020-09-20-175714. Stopped kubelet service on one node and see alerts firing and they were cleared once kubelet was started again.

$ oc get clusterversion
NAME      VERSION                             AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.4.0-0.nightly-2020-09-20-175714   True        False         57m     Cluster version is 4.4.0-0.nightly-2020-09-20-175714

$ oc get nodes
NAME                                         STATUS     ROLES    AGE   VERSION
ip-10-0-134-125.us-east-2.compute.internal   Ready      master   90m   v1.17.1+809da40
ip-10-0-150-228.us-east-2.compute.internal   NotReady   worker   78m   v1.17.1+809da40
ip-10-0-182-191.us-east-2.compute.internal   Ready      worker   78m   v1.17.1+809da40
ip-10-0-187-120.us-east-2.compute.internal   Ready      master   90m   v1.17.1+809da40
ip-10-0-200-13.us-east-2.compute.internal    Ready      master   90m   v1.17.1+809da40
ip-10-0-205-216.us-east-2.compute.internal   Ready      worker   78m   v1.17.1+809da40

$ token=`oc sa get-token prometheus-k8s -n openshift-monitoring`

$ oc -n openshift-monitoring exec -c prometheus prometheus-k8s-0 -- curl -k -g -H "Authorization: Bearer $token" 'https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?query=mcd_kubelet_state>2' | jq
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100   334  100   334    0     0   6769      0 --:--:-- --:--:-- --:--:--  6816
{
  "status": "success",
  "data": {
    "resultType": "vector",
    "result": [
      {
        "metric": {
          "__name__": "mcd_kubelet_state",
          "endpoint": "metrics",
          "instance": "10.0.150.228:9001",
          "job": "machine-config-daemon",
          "namespace": "openshift-machine-config-operator",
          "pod": "machine-config-daemon-th47l",
          "service": "machine-config-daemon"
        },
        "value": [
          1600757954.48,
          "3"
        ]
      }
    ]
  }
}

$ oc get nodes
NAME                                         STATUS   ROLES    AGE   VERSION
ip-10-0-134-125.us-east-2.compute.internal   Ready    master   91m   v1.17.1+809da40
ip-10-0-150-228.us-east-2.compute.internal   Ready    worker   79m   v1.17.1+809da40
ip-10-0-182-191.us-east-2.compute.internal   Ready    worker   79m   v1.17.1+809da40
ip-10-0-187-120.us-east-2.compute.internal   Ready    master   91m   v1.17.1+809da40
ip-10-0-200-13.us-east-2.compute.internal    Ready    master   91m   v1.17.1+809da40
ip-10-0-205-216.us-east-2.compute.internal   Ready    worker   79m   v1.17.1+809da40

$ oc -n openshift-monitoring exec -c prometheus prometheus-k8s-0 -- curl -k -g -H "Authorization: Bearer $token" 'https://prometheus-k8s.openshift-monitoring.svc:9091/api/v1/query?query=mcd_kubelet_state>2' | jq
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100    63  100    63    0     0   1164      0 --:--:-- --:--:-- --:--:--  1188
{
  "status": "success",
  "data": {
    "resultType": "vector",
    "result": []
  }
}

Comment 3 Sunil Choudhary 2020-09-22 07:03:32 UTC

Created attachment 1715639 [details]
metrics-fix-confirmation

Comment 6 errata-xmlrpc 2020-10-01 17:50:07 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (OpenShift Container Platform 4.4.26 bug fix update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:3764