Description of problem: After creating a pod on a worker and then trying to delete the worker without draining it, the pod is successfully moved to another worker and gets stuck in 'Terminating' state on the deleted worker. The deleted worker can't be deleted and is still listed when checking the nodes but it's unreachable. If trying to drain the node with the running pod after the mentioned above, The pod is moved to another worker, And get's stuck in a 'Terminating' state on the current node. Hence, the current node is not deleted properly. Version-Release number of selected component (if applicable): How reproducible: Steps to Reproduce: 1. Create a pod on worker. 2. Delete the worker without draining. cmd: $ oc adm drain $ oc annotate machine <worker-CONSUMER-name> machine.openshift.io/cluster-api-delete-machine=yes -n openshift-machine-api $ oc delete bmh <worker-name> -n openshift-machine-api 3. The pod status is stuck on terminating, So we're unable to delete the worker node. When trying delete the worker node, the status became NotReady, SchedulingDisabled. Actual results: $ oc get nodes: Status of deleted worker node: NotReady, SchedulingDisabled. $ oc get machine -o wide -n openshift-machine-api: The node is still listed. Expected results: Node is successfully deleted and is not listed anywhere( $ oc get nodes $ oc get bmh -n openshift-machine-api $ oc get machine -o wide -n openshift-machine-api ). Additional info:
Can you provide information on the crio and kubelet versions in use?
Also, we need the kubelet and crio logs as well if you have them.
Created attachment 1682043 [details] worker description
I can not reproduce the above steps, in step 2: $ oc delete bmh ip-10-0-168-12.us-east-2.compute.internal -n openshift-machine-api Error from server (NotFound): baremetalhosts.metal3.io "ip-10-0-168-12.us-east-2.compute.internal" not found It seems that baremetalhosts don't exist!
Setting target release to current development version (4.5) for investigation. Where fixes (if any) are required/requested for prior versions, cloned BZs will be created when appropriate.
The issue here is that the cluster-api-provider-baremetal revendored github.com/openshift/cluster-api. In 4.4 and newer, the relevant machine-api code has moved to github.com/openshift/machine-api-operator and should be vendored instead. github.com/openshift/cluster-api is no longer in development.
*** Bug 1837505 has been marked as a duplicate of this bug. ***
*** Bug 1840577 has been marked as a duplicate of this bug. ***
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2020:2409