Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

Bug 1466720

Summary: Pod is not terminated promptly when user delete it without gracefully option
Product: OpenShift Container Platform Reporter: XiaochuanWang <xiaocwan>
Component: NodeAssignee: Derek Carr <decarr>
Status: CLOSED NOTABUG QA Contact: DeShuai Ma <dma>
Severity: medium Docs Contact:
Priority: medium    
Version: 3.6.1CC: aos-bugs, decarr, dmoessne, jokerman, mmccomas, wmeng
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2017-06-30 16:11:16 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Attachments:
Description Flags
cli side force delete Terminating pod promptly none

Description XiaochuanWang 2017-06-30 10:14:56 UTC 
Description of problem:
Running a standalone pod, delete it and keep it in Terminating (could stop node before delete), try in both CLI and console sides, pod is still Terminating

Version-Release number of selected component (if applicable):
OpenShift Master:  v3.6.128
Kubernetes Master: v1.6.1+5115d708d7

How reproducible:
Always

Steps to Reproduce:
1. Put a pod in Terminationg status:
oc run mypod --image=aosqe/hello-openshift --generator='run-pod/v1'
Stop node of the pod: systemctl stop atomic-openshift-node.service
Check when pod is in Terminating status
2. oc delete pod mypod --grace-period=0 # or use --now=true, which has same result
3. OR delete the pod in web-console and select "Delete pod immediately without waiting for the processes to terminate gracefully"

Actual results:
In both 2 and 3, Pod is still in Terminating.

Expected results:
Pod should be Terminated promptly

Additional info:
Pod keeps Terminating for at least 20 mins
Comment 1 Derek Carr 2017-06-30 15:40:56 UTC 
this should work with

$ oc delete pods <name> --force --grace-period=0

verifying that locally.
Comment 2 Derek Carr 2017-06-30 16:11:16 UTC 
for future reference, a normal delete flow of a pod is as follows:

1. user deletes a pod
2. pod deletion timestamp is set
3. kubelet observes pod update, and cleans up pod on node
4. kubelet sends final deletion request of pod
5. pod is removed

in this scenario, a pod stuck in terminating phase, will not have the final deletion sent by the kubelet.

in prior versions of the product, the node controller would have sent a force delete of a pod after enough time expired and delete the pod, but this was removed in https://github.com/kubernetes/kubernetes/pull/36017 due to safety concerns.

as a result, users must explicitly force delete the pod.

in order to force delete the pod, the user must pass two options:

$ oc delete pods <name> --force --grace-period=0

for example:

$ oc get nodes
NAME                    STATUS                     AGE       VERSION
openshift-master-node   Ready,SchedulingDisabled   5m        v1.6.1+5115d708d7
openshift-node-1        NotReady                   5m        v1.6.1+5115d708d7
openshift-node-2        Ready                      5m        v1.6.1+5115d708d7

$ oc get pods
NAME            READY     STATUS        RESTARTS   AGE
hello-1-57lkv   1/1       Running       0          1m
hello-1-vmwv5   1/1       Terminating   0          3m

the pod hello-1-vmwv5 is stuck terminating because openshift-node-1 is not running.

$ oc delete pods hello-1-vmwv5 --grace-period=0 --force
warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely.
pod "hello-1-vmwv5" deleted

$ oc get pods
NAME            READY     STATUS    RESTARTS   AGE
hello-1-57lkv   1/1       Running   0          1m

the pod is now removed.
Comment 4 XiaochuanWang 2017-07-03 01:32:24 UTC 
Thanks, that makes sense to QE when it's safe to use --force to confirm the final deletion.
Comment 5 XiaochuanWang 2017-07-03 01:55:58 UTC 
CLI is cool, but this will cause a UI issue, QE will reopen https://bugzilla.redhat.com/show_bug.cgi?id=1462067
Comment 6 XiaochuanWang 2017-07-05 07:45:38 UTC 
Created attachment 1294477 [details]
cli side force delete Terminating pod promptly
Comment 7 Red Hat Bugzilla 2023-09-14 04:00:25 UTC 
The needinfo request[s] on this closed bug have been removed as they have been unresolved for 1000 days