Description of problem: If an instance is removed from the cloud-provider (either via a user-deletion, or cloud-provider event of some kind), and the machine-object is reconciled again for some reason, the machine-controller may determine the instance no longer 'exists' and attempt to create the instance. This is undocumented behavior and should not be relied upon for workflows. This operation might interfere with current or future components, such as the node-health-checker. We should track state to ensure the Create() function is called once for any machine-object. For a machine-object that has it's backing instance removed, that should be handled by node-health-checker or similar out-of-band controller.
Added to 4.1 issue tracker comments: https://github.com/openshift/openshift-docs/issues/12487 If we ship a fix for this prior to shipping 4.1 GA, may want to ensure it gets tracked properly there as well.
This is by design the expected behaviour for any kubenetes controller and we shouldn't deviate from it - it reconciles existing state with desired state. If something deletes an instance out of band the machine api will notice only once the controller resync period is expired. How the machine health checking or any other component interacts by consuming the API is orthogonal.
Keeping this open and bumping to 4.3 as we plan to make machines objects "fire and forget" in terms of cloud instance creation
(In reply to Michael Gugino from comment #0) Why is this not expected behavior? If it's not expected behavior then why is it not part of the docs directly? (In reply to Alberto from comment #2) > This is by design the expected behaviour for any kubenetes controller and we > shouldn't deviate from it - it reconciles existing state with desired state. If this is true then we should close as NOT A BUG.
(In reply to Eric Rich from comment #4) > (In reply to Michael Gugino from comment #0) > > Why is this not expected behavior? If it's not expected behavior then why is > it not part of the docs directly? > @Eric This behavior is mostly an artifact from upstream. There's a multitude of reasons why it doesn't fit well for us, and upstream is (I believe) also switching to the 'create once' model. We only recently decided which behavior we actually want. > (In reply to Alberto from comment #2) > > This is by design the expected behaviour for any kubenetes controller and we > > shouldn't deviate from it - it reconciles existing state with desired state. > > If this is true then we should close as NOT A BUG. This statement is outdated. This bug might be redundant if we're tracking feature work elsewhere, but on the other hand, this might be useful to others if they consider it a bug to have the rational here until we cover in docs and code.
Since we introduced machine phases this should be fixed now. If a cloud instance is deleted out of band the backing machine should enter a failed phase. It must be deleted. https://github.com/openshift/cluster-api/pull/75
Verified in 4.3.0-0.nightly-2019-11-13-233341. If the instance is deleted, it's backing machine has 'Failed' phase. The machine must be deleted.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2020:0062