Description of problem: When trying to evacuate pods from a node that was going on maintenance, some pods were stuck in "Terminating" state. After rebooting the node, those pods were in "Unknown" state. Trying to terminate the pods using `oc delete --force --grace-period=0`, didn't help. The pods vanished after some time (~1h). Version-Release number of selected component (if applicable): 3.5.5.31 How reproducible: At least once Steps to Reproduce: 1. evacuate pods, if pods in terminating state never die, reboot 2. node back, pods in unknown state 3. use oc delete --force --grace-period=0 to delete the pods in unknown state. Actual results: oc delete doesn't produce the expected results. Expected results: The pods in unknown state to be deleted. Additional info:
I believe the pods being on unknown state is the effect of the following proposal: https://github.com/kubernetes/community/blob/master/contributors/design-proposals/storage/pod-safety.md What doesn't seem to be working as expected is the "--force" parameter as per: https://github.com/kubernetes/kubernetes/pull/37263 kubectl delete pods <pod> --grace-period=0 --force should do the trick. As far as I have seen, oc delete will send any parameter directly to kubectl.
Avesh, PTAL
Hi Sergi Jimenez Romero, I looked into it and here is my understanding: First of all: oc delete --grace-period=0 --force should have done the job (though you tried oc delete --force --grace-period=0, but i think the order of force and grace-period does not matter). So now the question is why it did not, and it may be due to several reasons, for example, kubelet might be wedged on the node, or any other issues with the node. (There is a similar upstream issue: https://github.com/kubernetes/kubernetes/issues/43279.) As a next step, I'd suggest you to provide: 1) logs from node 2) logs from master 3) oc describe node 4) oc describe pod To really see what is going on.
Hi Sergi Jimenez Romero, I have been trying to reproduce this on my 1 master and 2-node cluster but unable too. Could you provide some details about the cluster in addition to the info I asked in https://bugzilla.redhat.com/show_bug.cgi?id=1505687#c3?
Since yesterday, I have been trying to reproduce but not able to: I have been running 50 pods rc: http://pastebin.test.redhat.com/527504 I have been running following commands several times for both nodes: oadm drain 192.168.122.186 --config=./openshift.local.config/master/admin.kubeconfig oadm drain 192.168.122.239 --config=./openshift.local.config/master/admin.kubeconfig oadm uncordon 192.168.122.186 --config=./openshift.local.config/master/admin.kubeconfig oadm uncordon 192.168.122.239 --config=./openshift.local.config/master/admin.kubeconfig I also tried Seth's suggestion to hold onto shell in mounted dirs on host for one of the pods basically by going into the pod's mounted dir and running a watch ls -al. But I dont see any pod stuck and drain is always successful.
But the bad news is that I can delete pods in unkown state with out any issue so Before: # oc get pods -a -o wide --config=./openshift.local.config/master/admin.kubeconfig |grep Unknown nginx8-009hz 0/1 Unknown 0 17m <none> 192.168.122.239 nginx8-0zgcp 0/1 Unknown 0 17m <none> 192.168.122.239 nginx8-1mksh 0/1 Unknown 0 17m <none> 192.168.122.239 nginx8-3lghb 0/1 Unknown 0 17m <none> 192.168.122.239 nginx8-4cp93 0/1 Unknown 0 17m <none> 192.168.122.239 nginx8-4n9sm 0/1 Unknown 0 17m <none> 192.168.122.239 nginx8-66tkm 0/1 Unknown 0 17m <none> 192.168.122.239 nginx8-84llr 0/1 Unknown 0 17m <none> 192.168.122.239 nginx8-cb4v3 0/1 Unknown 0 17m <none> 192.168.122.239 nginx8-cxb6q 0/1 Unknown 0 17m <none> 192.168.122.239 nginx8-d4726 0/1 Unknown 0 17m <none> 192.168.122.239 nginx8-frk9n 0/1 Unknown 0 17m <none> 192.168.122.239 nginx8-g65xt 0/1 Unknown 0 17m <none> 192.168.122.239 nginx8-gdjfz 0/1 Unknown 0 17m <none> 192.168.122.239 nginx8-ktx78 0/1 Unknown 0 17m <none> 192.168.122.239 nginx8-ljb2f 0/1 Unknown 0 17m <none> 192.168.122.239 nginx8-m6lb4 0/1 Unknown 0 17m <none> 192.168.122.239 nginx8-pg93t 0/1 Unknown 0 17m <none> 192.168.122.239 nginx8-rq46z 0/1 Unknown 0 17m <none> 192.168.122.239 nginx8-s13pd 0/1 Unknown 0 17m <none> 192.168.122.239 nginx8-s4222 0/1 Unknown 0 17m <none> 192.168.122.239 nginx8-stjd5 0/1 Unknown 0 17m <none> 192.168.122.239 nginx8-stl9z 0/1 Unknown 0 17m <none> 192.168.122.239 nginx8-vkwt4 0/1 Unknown 0 17m <none> 192.168.122.239 nginx8-wkj4q 0/1 Unknown 0 17m <none> 192.168.122.239 Now force delete one of the above: #oc delete --force --grace-period=0 pod nginx8-009hz --config=./openshift.local.config/master/admin.kubeconfig warning: Immediate deletion does not wait for confirmation that the running resource has been terminated. The resource may continue to run on the cluster indefinitely. pod "nginx8-009hz" deleted After: # oc get pods -a -o wide --config=./openshift.local.config/master/admin.kubeconfig |grep Unknown nginx8-0zgcp 0/1 Unknown 0 18m <none> 192.168.122.239 nginx8-1mksh 0/1 Unknown 0 18m <none> 192.168.122.239 nginx8-3lghb 0/1 Unknown 0 18m <none> 192.168.122.239 nginx8-4cp93 0/1 Unknown 0 18m <none> 192.168.122.239 nginx8-4n9sm 0/1 Unknown 0 18m <none> 192.168.122.239 nginx8-66tkm 0/1 Unknown 0 18m <none> 192.168.122.239 nginx8-84llr 0/1 Unknown 0 18m <none> 192.168.122.239 nginx8-cb4v3 0/1 Unknown 0 18m <none> 192.168.122.239 nginx8-cxb6q 0/1 Unknown 0 18m <none> 192.168.122.239 nginx8-d4726 0/1 Unknown 0 18m <none> 192.168.122.239 nginx8-frk9n 0/1 Unknown 0 18m <none> 192.168.122.239 nginx8-g65xt 0/1 Unknown 0 18m <none> 192.168.122.239 nginx8-gdjfz 0/1 Unknown 0 18m <none> 192.168.122.239 nginx8-ktx78 0/1 Unknown 0 18m <none> 192.168.122.239 nginx8-ljb2f 0/1 Unknown 0 18m <none> 192.168.122.239 nginx8-m6lb4 0/1 Unknown 0 18m <none> 192.168.122.239 nginx8-pg93t 0/1 Unknown 0 18m <none> 192.168.122.239 nginx8-rq46z 0/1 Unknown 0 18m <none> 192.168.122.239 nginx8-s13pd 0/1 Unknown 0 18m <none> 192.168.122.239 nginx8-s4222 0/1 Unknown 0 18m <none> 192.168.122.239 nginx8-stjd5 0/1 Unknown 0 18m <none> 192.168.122.239 nginx8-stl9z 0/1 Unknown 0 18m <none> 192.168.122.239 nginx8-vkwt4 0/1 Unknown 0 18m <none> 192.168.122.239 nginx8-wkj4q 0/1 Unknown 0 18m <none> 192.168.122.239
Reference bug 1557306