Bug 1811169

Summary: /readyz should start reporting failure on shutdown initiation
Product: OpenShift Container Platform Reporter: Abu Kashem <akashem>
Component: kube-apiserverAssignee: Abu Kashem <akashem>
Status: CLOSED ERRATA QA Contact: Ke Wang <kewang>
Severity: high Docs Contact:
Priority: unspecified    
Version: 4.4CC: aos-bugs, mfojtik, sttts, xxia
Target Milestone: ---   
Target Release: 4.5.0   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: No Doc Update
Doc Text:
Story Points: ---
Clone Of:
: 1811198 1811200 1811202 (view as bug list) Environment:
Last Closed: 2020-07-13 17:18:56 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On:    
Bug Blocks: 1811198, 1821493, 1821494, 1821495    

Description Abu Kashem 2020-03-06 18:37:07 UTC 
Description of problem:

Currently, /readyz starts reporting failure after ShutdownDelayDuration elapses. The load balancer(s) uses /readyz for health check and are not aware of the shutdown initiation until ShutdownDelayDuration elapses. This does not give the load balancer(s) enough time to detect and react to it.

We expect /readyz to start returning failure as soon as apiserver shutdown is initiated(SIGTERM received). This gives the load balancer a window (defined by ShutdownDelayDuration) to detect that /readyz is red and stop sending traffic to this server.


How reproducible:
Always


upstream PR: https://github.com/kubernetes/kubernetes/pull/88911
Comment 4 Ke Wang 2020-03-16 05:54:43 UTC 
Verified with OCP build 4.5.0-0.nightly-2020-03-15-152626, detail see below,

- in one terminal:
  - exec into kube-apiserver pod of master 0
    $ oc rsh -n openshift-kube-apiserver <kube-apiserver pod name>
  - execute: while true; do curl -k https://localhost:6443/readyz; done
    sh-4.2# while true; do curl -k https://localhost:6443/readyz; done
    okokokokokok ...

- in other terminal:
  - oc debug node/<master-0>
  - chroot /host
  - bash
  - ps aux | grep " kube-apiserver "
  - kill -INT <pid-from-previous-output>
- in first terminal we can see:

[+]ping ok
[+]log ok
[+]etcd ok
[+]poststarthook/quota.openshift.io-clusterquotamapping ok
[+]poststarthook/openshift.io-startkubeinformers ok
[+]poststarthook/openshift.io-StartOAuthInformers ok
[+]poststarthook/start-kube-apiserver-admission-initializer ok
[+]poststarthook/generic-apiserver-start-informers ok
[+]poststarthook/start-apiextensions-informers ok
[+]poststarthook/start-apiextensions-controllers ok
[+]poststarthook/crd-discovery-available ok
[+]poststarthook/crd-informer-synced ok
[+]poststarthook/bootstrap-controller ok
[+]poststarthook/rbac/bootstrap-roles ok
[+]poststarthook/scheduling/bootstrap-system-priority-classes ok
[+]poststarthook/start-cluster-authentication-info-controller ok
[+]poststarthook/aggregator-reload-proxy-client-cert ok
[+]poststarthook/start-kube-aggregator-informers ok
[+]poststarthook/apiservice-registration-controller ok
[+]poststarthook/apiservice-status-available-controller ok
[+]poststarthook/apiservice-wait-for-first-sync ok
[+]poststarthook/kube-apiserver-autoregistration ok
[+]autoregister-completion ok
[+]poststarthook/apiservice-openapi-controller ok
[-]shutdown failed: reason withheld
healthz check failed


The endpoint of readyz will start returning failure as soon as apiserver shutdown is initiated.
Comment 6 errata-xmlrpc 2020-07-13 17:18:56 UTC 
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:2409