Bug 2089933 - Pods do not post readiness status on termination
Summary: Pods do not post readiness status on termination
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Node
Version: 4.9
Hardware: Unspecified
OS: Unspecified
high
high
Target Milestone: ---
: 4.11.0
Assignee: Ryan Phillips
QA Contact: Sunil Choudhary
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2022-05-24 18:13 UTC by Ryan Phillips
Modified: 2022-08-10 11:14 UTC (History)
2 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Cause: Code changes in 4.9 Consequence: Readiness probes were not running. Fix: Code change Result: Readiness probes now fire on pod termination.
Clone Of:
Environment:
Last Closed: 2022-08-10 11:14:04 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Github openshift kubernetes pull 1285 0 None open Bug 2089933: Backport 110191 Re-enable Kubelet Pod Readiness Probes on Termination and Pod probes should be handled by p... 2022-06-07 13:04:41 UTC
Red Hat Product Errata RHSA-2022:5069 0 None None None 2022-08-10 11:14:15 UTC

Description Ryan Phillips 2022-05-24 18:13:42 UTC
Description of problem:
4.9 introduced a pod refactor within the kubelet. This refactor stopped all probes on termination; however, eadiness probes are extremely useful to load balancers and controllers to know when a pod is not handling traffic anymore (to allow for a graceful handoff).

The kubelet should run readiness probes on termination. The documentation states that the kubelet should, but it does not.

Upstream Issue: https://github.com/kubernetes/kubernetes/issues/102537
Related Endpoints and EndpointSlices issue: https://github.com/kubernetes/kubernetes/pull/110115
PR to Fix: https://github.com/kubernetes/kubernetes/pull/110191

Version-Release number of selected component (if applicable):


How reproducible:


Steps to Reproduce:
1.
2.
3.

Actual results:


Expected results:


Additional info:

Comment 3 Sunil Choudhary 2022-06-17 10:55:02 UTC
% oc get clusterversion
NAME      VERSION                              AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.11.0-0.nightly-2022-06-15-222801   True        False         47m     Cluster version is 4.11.0-0.nightly-2022-06-15-222801

% cat epod.yaml 
apiVersion: v1
kind: Pod
metadata:
  name: twocontainers
  annotations:
    cpu-load-balancing.crio.io: disable
spec:
  containers:
  - name: sise
    image: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:aa8d1daf3432d8dedc5c56d94aeb1f25301bce6ccd7d5406fb03a00be97374ad
    command:
      - "bin/bash"
      - "-c"
      - "sleep 10000"
  resources:
    limits:
      cpu: “500m”
      memory: "500Mi"
    requests:
      cpu: “400m”
      memory: "400Mi"


% oc create -f epod.yaml                                     
Warning: would violate PodSecurity "restricted:latest": allowPrivilegeEscalation != false (container "sise" must set securityContext.allowPrivilegeEscalation=false), unrestricted capabilities (container "sise" must set securityContext.capabilities.drop=["ALL"]), runAsNonRoot != true (pod or container "sise" must set securityContext.runAsNonRoot=true), seccompProfile (pod or container "sise" must set securityContext.seccompProfile.type to "RuntimeDefault" or "Localhost")
pod/twocontainers created

% oc get pods
NAME            READY   STATUS    RESTARTS   AGE
twocontainers   1/1     Running   0          3s

On terminal 1, I deleted pod and on terminal 2 I see the pod status changed to Terminating immediately.

Terminal 1:

% oc get pods
NAME            READY   STATUS    RESTARTS   AGE
twocontainers   1/1     Running   0          7m39s
sunilc@schoudha-mac debug1 % oc delete pod twocontainers

Terminal 2:

% oc get pods -w
NAME            READY   STATUS    RESTARTS   AGE
twocontainers   1/1     Running   0          7m32s
twocontainers   1/1     Terminating   0          7m44s
twocontainers   0/1     Terminating   0          8m15s
twocontainers   0/1     Terminating   0          8m15s
twocontainers   0/1     Terminating   0          8m15s

Comment 5 errata-xmlrpc 2022-08-10 11:14:04 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Important: OpenShift Container Platform 4.11.0 bug fix and security update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2022:5069


Note You need to log in before you can comment on or make changes to this bug.