Bug 1951713

Description of problem:

OCP 4.5.16 upgraded to 4.6.21 without issues, although, after the upgrade there were events in the web console stating the master nodes couldn't be updated.

Annotation `machine.openshift.io/instance-state` shows ERROR for all the master nodes.

It was noticed that, initially, master nodes were installed using an image in glance named `rhcos-lab` which doesn't exist anymore. The rest of the nodes were configured to use another RHCOS image which exist in glance and those are ok.



In machine-api-controllers POD logs there's the following message:

~~~
2021-04-15T19:07:15.604191855Z E0415 19:07:15.603326       1 actuator.go:550] Machine error ocpprod-xxxxx-master-0: no image with the name rhcos-lab could be found
2021-04-15T19:07:15.604191855Z E0415 19:07:15.603362       1 controller.go:280] ocpprod-xxxxx-master-0: error updating machine: no image with the name rhcos-lab could be found
2021-04-15T19:07:19.441795240Z E0415 19:07:19.441446       1 actuator.go:550] Machine error ocpprod-xxxxx-master-1: no image with the name rhcos-lab could be found
2021-04-15T19:07:19.441795240Z E0415 19:07:19.441481       1 controller.go:280] ocpprod-xxxxx-master-1: error updating machine: no image with the name rhcos-lab could be found
2021-04-15T19:07:23.577725170Z E0415 19:07:23.577676       1 actuator.go:550] Machine error ocpprod-xxxxx-master-2: no image with the name rhcos-lab could be found
2021-04-15T19:07:23.577937392Z E0415 19:07:23.577906       1 controller.go:280] ocpprod-xxxxx-master-2: error updating machine: no image with the name rhcos-lab could be found
~~~


When comparing master machine objects with the others

worker-xxx-xxxxx machine definition:
-------------------------------------
~~~
spec:
  metadata: {}
  providerSpec:
    value:
      apiVersion: openstackproviderconfig.openshift.io/v1alpha1
      cloudName: openstack
      cloudsSecret:
        name: openstack-cloud-credentials
        namespace: openshift-machine-api
      flavor: openshift
      image: rhcos-4.5                    <<<<<<<<<<<<<<<<<< Here
      kind: OpenstackProviderSpec
~~~


Master machine definition:
---------------------------
~~~
spec:
  metadata: {}
  providerSpec:
    value:
      apiVersion: openstackproviderconfig.openshift.io/v1alpha1
      cloudName: openstack
      cloudsSecret:
        name: openstack-cloud-credentials
        namespace: openshift-machine-api
      flavor: openshift
      image: rhcos-lab                    <<<<<<<<<<<<<<<<<< Here
      kind: OpenstackProviderSpec
~~~





A test was done by modifying the image name, in the machine resource, to the same name as the rest of the nodes which triggered the deletion process for master-2 instance and it remained in `Not Ready` and `phase: Failed`

~~~
  errorMessage: Can't find created instance.
  lastUpdated: '2021-04-19T13:08:59Z'
  nodeRef:
    kind: Node
    name: ocpprod-xxxxx-master-2
    uid: e375026b-3bb0-4b33-98d4-0745cb9644f0
  phase: Failed
~~~


VM was gone from OpenStack 






Version-Release number of selected component (if applicable):

- OpenShift 4.6.21 (upgraded from 4.5.16)
- OpenStack 16.1
- Install method: IPI

How reproducible:

Not sure

Steps to Reproduce:
1. Install IPI OCP on OSP
2. Remove the glance image used for master nodes
3. Change image in machine object to point to an existing image in glance

Actual results:

- VM instance disappeared in OSP.
- Master node remains in NotReady status because it cannot be reached.

Expected results:

- Master VM instance to be created in OSP and join the cluster again with corresponding RHCOS image.

Additional info: