Bug 1466720 - Pod is not terminated promptly when user delete it without gracefully option [NEEDINFO]
Pod is not terminated promptly when user delete it without gracefully option
Product: OpenShift Container Platform
Classification: Red Hat
Component: Pod (Show other bugs)
Unspecified Unspecified
medium Severity medium
: ---
: ---
Assigned To: Derek Carr
DeShuai Ma
Depends On:
  Show dependency treegraph
Reported: 2017-06-30 06:14 EDT by XiaochuanWang
Modified: 2017-07-05 03 EDT (History)
6 users (show)

See Also:
Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of:
Last Closed: 2017-06-30 12:11:16 EDT
Type: Bug
Regression: ---
Mount Type: ---
Documentation: ---
Verified Versions:
Category: ---
oVirt Team: ---
RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: ---
dmoessne: needinfo? (decarr)

Attachments (Terms of Use)
cli side force delete Terminating pod promptly (44.92 KB, image/png)
2017-07-05 03:45 EDT, XiaochuanWang
no flags Details

  None (edit)
Description XiaochuanWang 2017-06-30 06:14:56 EDT
Description of problem:
Running a standalone pod, delete it and keep it in Terminating (could stop node before delete), try in both CLI and console sides, pod is still Terminating

Version-Release number of selected component (if applicable):
OpenShift Master:  v3.6.128
Kubernetes Master: v1.6.1+5115d708d7

How reproducible:

Steps to Reproduce:
1. Put a pod in Terminationg status:
oc run mypod --image=aosqe/hello-openshift --generator='run-pod/v1'
Stop node of the pod: systemctl stop atomic-openshift-node.service
Check when pod is in Terminating status
2. oc delete pod mypod --grace-period=0 # or use --now=true, which has same result
3. OR delete the pod in web-console and select "Delete pod immediately without waiting for the processes to terminate gracefully"

Actual results:
In both 2 and 3, Pod is still in Terminating.

Expected results:
Pod should be Terminated promptly

Additional info:
Pod keeps Terminating for at least 20 mins
Comment 1 Derek Carr 2017-06-30 11:40:56 EDT
this should work with

$ oc delete pods <name> --force --grace-period=0

verifying that locally.
Comment 2 Derek Carr 2017-06-30 12:11:16 EDT
for future reference, a normal delete flow of a pod is as follows:

1. user deletes a pod
2. pod deletion timestamp is set
3. kubelet observes pod update, and cleans up pod on node
4. kubelet sends final deletion request of pod
5. pod is removed

in this scenario, a pod stuck in terminating phase, will not have the final deletion sent by the kubelet.

in prior versions of the product, the node controller would have sent a force delete of a pod after enough time expired and delete the pod, but this was removed in https://github.com/kubernetes/kubernetes/pull/36017 due to safety concerns.

as a result, users must explicitly force delete the pod.

in order to force delete the pod, the user must pass two options:

$ oc delete pods <name> --force --grace-period=0

for example:

$ oc get nodes
NAME                    STATUS                     AGE       VERSION
openshift-master-node   Ready,SchedulingDisabled   5m        v1.6.1+5115d708d7
openshift-node-1        NotReady                   5m        v1.6.1+5115d708d7
openshift-node-2        Ready                      5m        v1.6.1+5115d708d7

$ oc get pods
NAME            READY     STATUS        RESTARTS   AGE
hello-1-57lkv   1/1       Running       0          1m
hello-1-vmwv5   1/1       Terminating   0          3m

the pod hello-1-vmwv5 is stuck terminating because openshift-node-1 is not running.

$ oc delete pods hello-1-vmwv5 --grace-period=0 --force
warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely.
pod "hello-1-vmwv5" deleted

$ oc get pods
hello-1-57lkv   1/1       Running   0          1m

the pod is now removed.
Comment 4 XiaochuanWang 2017-07-02 21:32:24 EDT
Thanks, that makes sense to QE when it's safe to use --force to confirm the final deletion.
Comment 5 XiaochuanWang 2017-07-02 21:55:58 EDT
CLI is cool, but this will cause a UI issue, QE will reopen https://bugzilla.redhat.com/show_bug.cgi?id=1462067
Comment 6 XiaochuanWang 2017-07-05 03:45 EDT
Created attachment 1294477 [details]
cli side force delete Terminating pod promptly

Note You need to log in before you can comment on or make changes to this bug.