Description of problem: Installing a fresh cluster with TP features enabled leads to a failure due to an issue pulling the MAPO image on the machine-api-controller pod: $ oc describe pods -n openshift-machine-api machine-api-controllers-5dbf96f4b6-srj6m | grep Failed.*kubelet Warning Failed 13m kubelet Failed to pull image "quay.io/openshift/origin-machine-api-provider-openstack": rpc error: code = Unknown desc = reading manifest latest in quay.io/openshift/origin-machine-api-provider-openstack: unauthorized: access to the requested resource is not authorized Warning Failed 13m kubelet Error: ErrImagePull kubelet logs on master-0 shows: Jan 14 09:47:15 ostest-c6gvz-master-0 crio[1690]: time="2022-01-14 09:47:15.663118513Z" level=info msg="Checking image status: quay.io/openshift/origin-machine-api-provider-openstack:latest" id=afa74b61-6921-4057-863f-04d7a689ed65 name=/runtime.v1.ImageService/ImageStatus Jan 14 09:47:15 ostest-c6gvz-master-0 crio[1690]: time="2022-01-14 09:47:15.663882084Z" level=info msg="Image quay.io/openshift/origin-machine-api-provider-openstack:latest not found" id=afa74b61-6921-4057-863f-04d7a689ed65 name=/runtime.v1.ImageService/ImageStatus Jan 14 09:47:15 ostest-c6gvz-master-0 hyperkube[1719]: E0114 09:47:15.664174 1719 pod_workers.go:918] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"machine-controller\" with ImagePullBackOff: \"Back-off pulling image \\\"quay.io/openshift/origin-machine-api-provider-openstack\\\"\"" pod="openshift-machine-api/machine-api-controllers-5dbf96f4b6-srj6m" podUID=cf6ea5e5-1824-4d29-b2d5-a0e121f3eb65 The container images inside the pod machine-api-controllers shows a wrong image reference for machine-controller container: $ oc get pods -n openshift-machine-api machine-api-controllers-5dbf96f4b6-srj6m -o json | jq '.spec.containers[] | "\(.name) \(.image)"' "machineset-controller quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7d275f4a735083d2edce1decf5e2da30233f04216bd472a6c084df0c067aa18f" "machine-controller quay.io/openshift/origin-machine-api-provider-openstack" <--------- "nodelink-controller quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7d275f4a735083d2edce1decf5e2da30233f04216bd472a6c084df0c067aa18f" "machine-healthcheck-controller quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7d275f4a735083d2edce1decf5e2da30233f04216bd472a6c084df0c067aa18f" "kube-rbac-proxy-machineset-mtrc quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:fd82cc4fa9b603e8604d31c403eaea42f376a0085c336d09d66ba2a80523a33f" "kube-rbac-proxy-machine-mtrc quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:fd82cc4fa9b603e8604d31c403eaea42f376a0085c336d09d66ba2a80523a33f" "kube-rbac-proxy-mhc-mtrc quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:fd82cc4fa9b603e8604d31c403eaea42f376a0085c336d09d66ba2a80523a33f" Version-Release number of selected component (if applicable): 4.10.0-0.nightly-2022-01-13-061145 How reproducible: Always Steps to Reproduce: 1. Install enabling the TP features on the manifests Actual results: Installation fails. No workers are deployed. Expected results: Installation succeeds. Additional info:
Verified on 4.10.0-0.nightly-2022-01-20-082726 Installation successful: $ oc get clusterversion NAME VERSION AVAILABLE PROGRESSING SINCE STATUS version 4.10.0-0.nightly-2022-01-20-082726 True False 115m Cluster version is 4.10.0-0.nightly-2022-01-20-082726 $ oc get featureGate/cluster -o yaml apiVersion: config.openshift.io/v1 kind: FeatureGate metadata: annotations: include.release.openshift.io/self-managed-high-availability: "true" include.release.openshift.io/single-node-developer: "true" release.openshift.io/create-only: "true" creationTimestamp: "2022-01-21T08:32:25Z" generation: 1 name: cluster resourceVersion: "1363" uid: ff9f6ec2-1aa1-4f10-8ed9-03997551dd76 spec: featureSet: TechPreviewNoUpgrade $ oc get pods -n openshift-machine-api -l api=clusterapi -o json | jq '.items[].spec.containers[] | "\(.name) \(.image)"' "machineset-controller quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e1822b43818b817842399b05253a42be3b783939f65e6a8c2bab2a65f9a8e7be" "machine-controller quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f601df9d9e58fc6b2cca2ac67b547780f128a62df73bca5d6842e1f1d76d1cdc" "nodelink-controller quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e1822b43818b817842399b05253a42be3b783939f65e6a8c2bab2a65f9a8e7be" "machine-healthcheck-controller quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e1822b43818b817842399b05253a42be3b783939f65e6a8c2bab2a65f9a8e7be" "kube-rbac-proxy-machineset-mtrc quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:fd82cc4fa9b603e8604d31c403eaea42f376a0085c336d09d66ba2a80523a33f" "kube-rbac-proxy-machine-mtrc quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:fd82cc4fa9b603e8604d31c403eaea42f376a0085c336d09d66ba2a80523a33f" "kube-rbac-proxy-mhc-mtrc quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:fd82cc4fa9b603e8604d31c403eaea42f376a0085c336d09d66ba2a80523a33f"
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Moderate: OpenShift Container Platform 4.10.3 security update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2022:0056