Description of problem: Running a standalone pod, delete it and keep it in Terminating (could stop node before delete), try in both CLI and console sides, pod is still Terminating Version-Release number of selected component (if applicable): OpenShift Master: v3.6.128 Kubernetes Master: v1.6.1+5115d708d7 How reproducible: Always Steps to Reproduce: 1. Put a pod in Terminationg status: oc run mypod --image=aosqe/hello-openshift --generator='run-pod/v1' Stop node of the pod: systemctl stop atomic-openshift-node.service Check when pod is in Terminating status 2. oc delete pod mypod --grace-period=0 # or use --now=true, which has same result 3. OR delete the pod in web-console and select "Delete pod immediately without waiting for the processes to terminate gracefully" Actual results: In both 2 and 3, Pod is still in Terminating. Expected results: Pod should be Terminated promptly Additional info: Pod keeps Terminating for at least 20 mins
this should work with $ oc delete pods <name> --force --grace-period=0 verifying that locally.
for future reference, a normal delete flow of a pod is as follows: 1. user deletes a pod 2. pod deletion timestamp is set 3. kubelet observes pod update, and cleans up pod on node 4. kubelet sends final deletion request of pod 5. pod is removed in this scenario, a pod stuck in terminating phase, will not have the final deletion sent by the kubelet. in prior versions of the product, the node controller would have sent a force delete of a pod after enough time expired and delete the pod, but this was removed in https://github.com/kubernetes/kubernetes/pull/36017 due to safety concerns. as a result, users must explicitly force delete the pod. in order to force delete the pod, the user must pass two options: $ oc delete pods <name> --force --grace-period=0 for example: $ oc get nodes NAME STATUS AGE VERSION openshift-master-node Ready,SchedulingDisabled 5m v1.6.1+5115d708d7 openshift-node-1 NotReady 5m v1.6.1+5115d708d7 openshift-node-2 Ready 5m v1.6.1+5115d708d7 $ oc get pods NAME READY STATUS RESTARTS AGE hello-1-57lkv 1/1 Running 0 1m hello-1-vmwv5 1/1 Terminating 0 3m the pod hello-1-vmwv5 is stuck terminating because openshift-node-1 is not running. $ oc delete pods hello-1-vmwv5 --grace-period=0 --force warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. pod "hello-1-vmwv5" deleted $ oc get pods NAME READY STATUS RESTARTS AGE hello-1-57lkv 1/1 Running 0 1m the pod is now removed.
Thanks, that makes sense to QE when it's safe to use --force to confirm the final deletion.
CLI is cool, but this will cause a UI issue, QE will reopen https://bugzilla.redhat.com/show_bug.cgi?id=1462067
Created attachment 1294477 [details] cli side force delete Terminating pod promptly
The needinfo request[s] on this closed bug have been removed as they have been unresolved for 1000 days