Bug 1939227 - kube-apiserver liveness probe appears to be hitting /healthz, not /livez
Summary: kube-apiserver liveness probe appears to be hitting /healthz, not /livez
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: kube-apiserver
Version: 4.8
Hardware: Unspecified
OS: Unspecified
medium
medium
Target Milestone: ---
: 4.8.0
Assignee: David Eads
QA Contact: Xingxing Xia
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2021-03-15 18:51 UTC by Elana Hashman
Modified: 2021-07-27 22:54 UTC (History)
6 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2021-07-27 22:53:48 UTC
Target Upstream Version:

Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Github openshift cluster-kube-apiserver-operator pull 1066 0 None open bug 1939227: make liveness hit livez 2021-03-15 19:36:16 UTC
Red Hat Product Errata RHSA-2021:2438 0 None None None 2021-07-27 22:54:05 UTC

Description Elana Hashman 2021-03-15 18:51:58 UTC 
Description of problem:

When looking at logs of an unhealthy API server, I noticed it appears to be hitting /healthz instead of /livez for liveness probes.

Mar 15 03:01:38.268055 ip-10-0-146-200 hyperkube[1442]: I0315 03:01:38.268073    1442 prober.go:117] Liveness probe for "kube-apiserver-ip-10-0-146-200.us-west-1.compute.internal_openshift-kube-apiserver(22bd459e-677c-40ce-a715-9a263b663b2b):kube-apiserver" failed (failure): Get "https://10.0.146.200:6443/healthz": net/http: request canceled (Client.Timeout exceeded while awaiting headers)


See also must-gather here: https://gcsweb-ci.apps.ci.l2s4.p1.openshiftapps.com/gcs/origin-ci-test/logs/release-openshift-ocp-installer-e2e-aws-4.8/1371290082278903808/artifacts/e2e-aws/


Version-Release number of selected component (if applicable): 4.8? possibly more


How reproducible:


Steps to Reproduce:
1.
2.
3.

Actual results:


Expected results:


Additional info:
Comment 2 David Eads 2021-03-16 19:28:20 UTC 
livez and healthz are slightly different, but in our configuration the distinction isn't meaningful enough to backport (I don't think).
Comment 3 David Eads 2021-03-16 19:29:04 UTC 
huh, I can't seem to set targets.  fun.
Comment 5 Xingxing Xia 2021-04-07 10:27:11 UTC 
Verified in 4.8.0-0.nightly-2021-04-06-162113:
$ oc get po -n openshift-kube-apiserver kube-apiserver-ci-ln-qfw7ms2-002ac-p5jzx-master-0 -o yaml
...
    livenessProbe:
      failureThreshold: 3
      httpGet:
        path: livez
...
    readinessProbe:
      failureThreshold: 3
      httpGet:
        path: readyz
...

Liveness probe now uses "livez".

Run crictl stop with the kube-apiserver container on master, then:
$ oc describe po -n openshift-kube-apiserver kube-apiserver-ci-ln-qfw7ms2-002ac-p5jzx-master-0
...
  Warning  Unhealthy   6m14s (x2 over 6m24s)  kubelet  Liveness probe failed: Get "https://10.0.0.6:6443/livez": dial tcp 10.0.0.6:6443: connect: connection refused
  Warning  ProbeError  6m14s (x2 over 6m24s)  kubelet  Readiness probe error: Get "https://10.0.0.6:6443/readyz": dial tcp 10.0.0.6:6443: connect: connection refused
...

Same, liveness probe now uses "livez".
Comment 8 errata-xmlrpc 2021-07-27 22:53:48 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: OpenShift Container Platform 4.8.2 bug fix and security update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2021:2438
Note You need to log in before you can comment on or make changes to this bug.