Bug 2040620

Summary:	[MAPO] Error pulling MAPO image on installation
Product:	OpenShift Container Platform	Reporter:	rlobillo
Component:	Cloud Compute	Assignee:	Martin André <m.andre>
Cloud Compute sub component:	OpenStack Provider	QA Contact:	rlobillo
Status:	CLOSED ERRATA	Docs Contact:
Severity:	high
Priority:	high	CC:	aos-bugs, m.andre, mfedosin, mifiedle, pprinett
Version:	4.10	Keywords:	TestBlocker, Triaged
Target Milestone:	---
Target Release:	4.10.0
Hardware:	Unspecified
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	No Doc Update
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2022-03-10 16:39:37 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description rlobillo 2022-01-14 09:59:57 UTC

Description of problem:

Installing a fresh cluster with TP features enabled leads to a failure due to an issue pulling the MAPO image on the machine-api-controller pod:

$ oc describe pods -n openshift-machine-api machine-api-controllers-5dbf96f4b6-srj6m | grep Failed.*kubelet
  Warning  Failed            13m                   kubelet            Failed to pull image "quay.io/openshift/origin-machine-api-provider-openstack": rpc error: code = Unknown desc = reading manifest latest in quay.io/openshift/origin-machine-api-provider-openstack: unauthorized: access to the requested resource is not authorized
  Warning  Failed            13m                   kubelet            Error: ErrImagePull


kubelet logs on master-0 shows:

Jan 14 09:47:15 ostest-c6gvz-master-0 crio[1690]: time="2022-01-14 09:47:15.663118513Z" level=info msg="Checking image status: quay.io/openshift/origin-machine-api-provider-openstack:latest" id=afa74b61-6921-4057-863f-04d7a689ed65 name=/runtime.v1.ImageService/ImageStatus
Jan 14 09:47:15 ostest-c6gvz-master-0 crio[1690]: time="2022-01-14 09:47:15.663882084Z" level=info msg="Image quay.io/openshift/origin-machine-api-provider-openstack:latest not found" id=afa74b61-6921-4057-863f-04d7a689ed65 name=/runtime.v1.ImageService/ImageStatus
Jan 14 09:47:15 ostest-c6gvz-master-0 hyperkube[1719]: E0114 09:47:15.664174    1719 pod_workers.go:918] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"machine-controller\" with ImagePullBackOff: \"Back-off pulling image \\\"quay.io/openshift/origin-machine-api-provider-openstack\\\"\"" pod="openshift-machine-api/machine-api-controllers-5dbf96f4b6-srj6m" podUID=cf6ea5e5-1824-4d29-b2d5-a0e121f3eb65

The container images inside the pod machine-api-controllers shows a wrong image reference for machine-controller container:

$ oc get pods -n openshift-machine-api machine-api-controllers-5dbf96f4b6-srj6m -o json | jq '.spec.containers[] | "\(.name) \(.image)"'
"machineset-controller quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7d275f4a735083d2edce1decf5e2da30233f04216bd472a6c084df0c067aa18f"
"machine-controller quay.io/openshift/origin-machine-api-provider-openstack" <---------
"nodelink-controller quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7d275f4a735083d2edce1decf5e2da30233f04216bd472a6c084df0c067aa18f"
"machine-healthcheck-controller quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7d275f4a735083d2edce1decf5e2da30233f04216bd472a6c084df0c067aa18f"
"kube-rbac-proxy-machineset-mtrc quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:fd82cc4fa9b603e8604d31c403eaea42f376a0085c336d09d66ba2a80523a33f"
"kube-rbac-proxy-machine-mtrc quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:fd82cc4fa9b603e8604d31c403eaea42f376a0085c336d09d66ba2a80523a33f"
"kube-rbac-proxy-mhc-mtrc quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:fd82cc4fa9b603e8604d31c403eaea42f376a0085c336d09d66ba2a80523a33f"



Version-Release number of selected component (if applicable):
4.10.0-0.nightly-2022-01-13-061145

How reproducible: Always


Steps to Reproduce:
1. Install enabling the TP features on the manifests

Actual results: Installation fails. No workers are deployed.

Expected results: Installation succeeds.


Additional info:

Comment 6 rlobillo 2022-01-21 11:00:01 UTC

Verified on 4.10.0-0.nightly-2022-01-20-082726

Installation successful:

$ oc get clusterversion
NAME      VERSION                              AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.10.0-0.nightly-2022-01-20-082726   True        False         115m    Cluster version is 4.10.0-0.nightly-2022-01-20-082726


$ oc get featureGate/cluster -o yaml
apiVersion: config.openshift.io/v1
kind: FeatureGate
metadata:
  annotations:
    include.release.openshift.io/self-managed-high-availability: "true"
    include.release.openshift.io/single-node-developer: "true"
    release.openshift.io/create-only: "true"
  creationTimestamp: "2022-01-21T08:32:25Z"
  generation: 1
  name: cluster
  resourceVersion: "1363"
  uid: ff9f6ec2-1aa1-4f10-8ed9-03997551dd76
spec:
  featureSet: TechPreviewNoUpgrade


$ oc get pods -n openshift-machine-api -l api=clusterapi -o json | jq '.items[].spec.containers[] | "\(.name) \(.image)"'
"machineset-controller quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e1822b43818b817842399b05253a42be3b783939f65e6a8c2bab2a65f9a8e7be"
"machine-controller quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f601df9d9e58fc6b2cca2ac67b547780f128a62df73bca5d6842e1f1d76d1cdc"
"nodelink-controller quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e1822b43818b817842399b05253a42be3b783939f65e6a8c2bab2a65f9a8e7be"
"machine-healthcheck-controller quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:e1822b43818b817842399b05253a42be3b783939f65e6a8c2bab2a65f9a8e7be"
"kube-rbac-proxy-machineset-mtrc quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:fd82cc4fa9b603e8604d31c403eaea42f376a0085c336d09d66ba2a80523a33f"
"kube-rbac-proxy-machine-mtrc quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:fd82cc4fa9b603e8604d31c403eaea42f376a0085c336d09d66ba2a80523a33f"
"kube-rbac-proxy-mhc-mtrc quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:fd82cc4fa9b603e8604d31c403eaea42f376a0085c336d09d66ba2a80523a33f"

Comment 10 errata-xmlrpc 2022-03-10 16:39:37 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Moderate: OpenShift Container Platform 4.10.3 security update), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2022:0056