Bug 2009474

+++ This bug was initially created as a clone of Bug #1992841 +++

Description of problem:

After adding a Windows node created through MachineSet and contains label 'machine.openshift.io/os-id:Windows', if the node is deleted using `oc delete node xxx` command and the operator pod is restarted, the Windowsmachine_controller throws a reconciliation error:
"could not get node associated with machine <machine-name>"

Version-Release number of selected component (if applicable):
4.9.0-0.nightly-2021-08-02-145924

How reproducible:
Always.

Steps to Reproduce:
1, Install WMCO operator
2, Create a MachineSet using the Windows label.
3, After Windows instance is fully configured as a node, run command `oc delete node xxx`
4. Restart the operator.

logs:

2021-08-11T19:38:41.402Z	ERROR	controller-runtime.manager.controller.machine	Reconciler error	{"reconciler group": "machine.openshift.io", "reconciler kind": "Machine", "name": "mankulka-04-5r472-windows-worker-us-east-2a-dtw2n", "namespace": "openshift-machine-api", "error": "could not get node associated with machine mankulka-04-5r472-windows-worker-us-east-2a-dtw2n: Node \"ip-10-0-154-73.us-east-2.compute.internal\" not found", "errorVerbose": "Node \"ip-10-0-154-73.us-east-2.compute.internal\" not found\ncould not get node associated with machine mankulka-04-5r472-windows-worker-us-east-2a-dtw2n\ngithub.com/openshift/windows-machine-config-operator/controllers.(*WindowsMachineReconciler).Reconcile\n\t/build/windows-machine-config-operator/controllers/windowsmachine_controller.go:251\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/build/windows-machine-config-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:298\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/build/windows-machine-config-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:253\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/build/windows-machine-config-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:214\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1371"}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
	/build/windows-machine-config-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:253
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
	/build/windows-machine-config-operator/vendor/sigs.k8s.io/controller-runtime/pkg/internal/controller/controller.go:214

Actual results:
Operator throws reconciliation error on trying to find a matching node.

Expected results:
Operator should run without error.

Additional info:

windowsmachine_controller is running into this issue because it was assuming the nodeRef to be nil, if a corresponding node is not currently present in the cluster. However the nodeRef is never updated to nil after a node is deleted by the nodelink_controller. The reasoning being, they do not expect a new node to be reverse mapped to an existing machine rather it is expected the machine would be deleted and re-created to configure a new node. Given this, proposing few ways to solve this issue:
1. WMCO does not requeue, if no node is present(not found) corresponding to a machine.
2. If a node is not found, WMCO deletes the machine(assuming the deletion will lead to recreation and configuration of new node)

--- Additional comment from mohashai on 2021-09-29 20:21:43 UTC ---

After discussing with the team, the decision was made to not react to Windows node deletion events.

This approach was chosen rather than deleting & re-creating the unassociated Machine as optimizing machine management is not in scope of WMCO responsibilities. In addition, not reacting is in line with the
current behavior with Linux Machines (MCO), standardizing OpenShift functionality across OSs. Also, the Machine cannot be reconfigured to create a new Node object since the machine-api's nodelink_controller will not update any Machine's Node reference, neither after deleting a Node nor after reconfiguring a Machine.

A fix has been tested and the PR is under review here https://github.com/openshift/windows-machine-config-operator/pull/675.