Description of problem: If a node is unhealthy and being replaced (either by a MachineHealthCheck, or by an operator manually deleting the Machine object), the machine-controller will attempt to drain the machine. However, if the node is unhealthy in such a way that a drain cannot succeed (e.g. node is completely offline or is stopped), the drain will fail. If the drain fails, the whole loop is requeued, and the machine is never properly replaced. Version-Release number of selected component (if applicable): 4.3.18 How reproducible: Consistently Steps to Reproduce: 1. Provision a cluster (e.g. an IPI AWS cluster) 2. Stop one of the machines from the cloud provider (e.g. Stop the AWS instance directly from the console) 3. When the node goes unready, delete the machine object (this can also be accomplished automatically with a machine Actual results: Drain fails and machine is never deleted Expected results: If drain fails, the machine-controller would move on and delete the machine anyways Additional info:
This is fixed in all current versions of 4.4. It's unlikely to get backported to 4.3.
I think bug 1733474 is the one that landed the fix for deleting unreachable nodes. It was cloned back to 4.3.z as bug 1803762, which was closed WONTFIX with "we won't be backporting". Marking this one as a dup of bug 1803762. *** This bug has been marked as a duplicate of bug 1803762 ***