Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

Bug 1805874

Summary:	Node in MachineSet failing EC2 health check goes NotReady and doesn't automatically recover
Product:	OpenShift Container Platform	Reporter:	Naveen Malik <nmalik>
Component:	Cloud Compute	Assignee:	Alberto <agarcial>
Status:	CLOSED DUPLICATE	QA Contact:	Jianwei Hou <jhou>
Severity:	unspecified	Docs Contact:
Priority:	unspecified
Version:	4.3.0
Target Milestone:	---
Target Release:	---
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2020-02-24 08:54:52 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description Naveen Malik 2020-02-21 16:51:31 UTC

Description of problem:
On an OSD 4.3.0 cluster had a infra node go NotReady.  The underlying problem is a failed health check on the EC2 node.  Ultimately I had to force stop the EC2 instance to get things to recover.

I tried oc delete machine on the offending instance.  This ended up creating a new Machine and Node but the old Machine and Node didn't terminate.  This means workloads did not move.

I had to go into EC2 and force stop / terminate the instance, at which point the cluster was able to delete the Machine and Node.

Version-Release number of selected component (if applicable):
4.3.0

How reproducible:
Unknown

Steps to Reproduce:
1.
2.
3.

Actual results:
Node stuck in NotReady required manual intervention.

Expected results:
Node in a MachineSet with underlying EC2 status check failures is automatically replaced by the platform.


Additional info:

Comment 2 Alberto 2020-02-24 08:54:52 UTC

> Node in a MachineSet with underlying EC2 status check failures is automatically replaced by the platform.

For automatic node recovery a machine health check resource is needed https://docs.openshift.com/container-platform/4.3/machine_management/deploying-machine-health-checks.html

In any case either deleting the machine automatically or manually draining get stuck because the node to be deleted is unreachable and stateful pods can't signal deletion appropriately https://kubernetes.io/docs/tasks/run-application/force-delete-stateful-set-pod/.
You can force a machine to skip draining by setting the "machine.openshift.io/exclude-node-draining" annotation on it.

This a dup of https://bugzilla.redhat.com/show_bug.cgi?id=1803762 which already has a PR up.

*** This bug has been marked as a duplicate of bug 1803762 ***