Description of problem: console-mce pods show ImagePullError after upgrading to ocp 4.10 Version-Release number of selected component (if applicable): ACM 2.5.0-DOWNSTREAM-2022-03-17-03-36-41 (Final S4) How reproducible: Steps to Reproduce: 1. Install ACM/MCH/MCE on ocp 4.9.23 successfully 2. Upgrade to ocp 4.10.5 3. MCE console is not available due to 'imagePullError' Actual results: Expected results: Additional info: oc get mce -n multicluster-engine -oyaml apiVersion: v1 items: - apiVersion: multicluster.openshift.io/v1 kind: MultiClusterEngine metadata: annotations: imageRepository: brew.registry.redhat.io/rh-osbs/rhacm2 creationTimestamp: "2022-03-18T14:20:38Z" finalizers: - finalizer.multicluster.openshift.io generation: 3 labels: installer.name: multiclusterhub installer.namespace: ocm name: multiclusterengine resourceVersion: "219903084" uid: 3610c1ca-1d25-4175-89f4-84b55c80ac95 spec: availabilityConfig: High imagePullSecret: multiclusterhub-operator-pull-secret overrides: components: - enabled: true name: assisted-service - enabled: true name: cluster-lifecycle - enabled: true name: cluster-manager - enabled: true name: discovery - enabled: true name: hive - enabled: true name: server-foundation - enabled: false name: managed-service-account - enabled: false name: hypershift-preview - enabled: true name: console-mce targetNamespace: multicluster-engine tolerations: - effect: NoSchedule key: node-role.kubernetes.io/infra operator: Exists status: components: - kind: Component lastTransitionTime: "2022-03-21T20:50:58Z" message: No resources present name: managedservice reason: ComponentDisabled status: "True" type: NotPresent - kind: Component lastTransitionTime: "2022-03-21T20:50:58Z" message: No resources present name: hypershift-addon-manager reason: ComponentDisabled status: "True" type: NotPresent - kind: Component lastTransitionTime: "2022-03-21T20:50:58Z" message: No resources present name: hypershift-deployment-controller reason: ComponentDisabled status: "True" type: NotPresent - kind: Deployment lastTransitionTime: "2022-03-21T20:45:12Z" message: Deployment does not have minimum availability. name: console-mce-console reason: MinimumReplicasUnavailable status: "False" type: Available conditions: - lastTransitionTime: "2022-03-21T20:28:58Z" lastUpdateTime: "2022-03-21T20:28:58Z" reason: ComponentsUnavailable status: "False" type: Available phase: Progressing kind: List metadata: resourceVersion: "" selfLink: "" oc get deploy -n multicluster-engine NAME READY UP-TO-DATE AVAILABLE AGE agentinstalladmission 2/2 2 2 7h7m assisted-service 1/1 1 1 7h7m cluster-curator-controller 2/2 2 2 3d6h cluster-manager 3/3 3 3 3d6h clusterclaims-controller 2/2 2 2 3d6h clusterlifecycle-state-metrics-v2 1/1 1 1 3d6h console-mce-console 1/2 2 1 8m51s discovery-operator 1/1 1 1 3d6h hive-operator 1/1 1 1 3d6h infrastructure-operator 1/1 1 1 3d6h managedcluster-import-controller-v2 2/2 2 2 3d6h multicluster-engine-operator 2/2 2 2 3d6h ocm-controller 2/2 2 2 3d6h ocm-proxyserver 2/2 2 2 3d6h ocm-webhook 2/2 2 2 3d6h provider-credential-controller 1/1 1 1 3d6h oc get deploy -n multicluster-engine console-mce-console -oyaml apiVersion: apps/v1 kind: Deployment metadata: annotations: deployment.kubernetes.io/revision: "3" creationTimestamp: "2022-03-21T20:45:12Z" generation: 3 labels: backplaneconfig.name: multiclusterengine name: console-mce-console namespace: multicluster-engine ownerReferences: - apiVersion: multicluster.openshift.io/v1 blockOwnerDeletion: true controller: true kind: MultiClusterEngine name: multiclusterengine uid: 3610c1ca-1d25-4175-89f4-84b55c80ac95 resourceVersion: "219899462" uid: fe89da20-20ee-41d0-a354-832ecc7bc632 spec: progressDeadlineSeconds: 600 replicas: 2 revisionHistoryLimit: 10 selector: matchLabels: app: console-mce strategy: rollingUpdate: maxSurge: 25% maxUnavailable: 25% type: RollingUpdate template: metadata: creationTimestamp: null labels: app: console-mce ocm-antiaffinity-selector: console-mce spec: affinity: nodeAffinity: requiredDuringSchedulingIgnoredDuringExecution: nodeSelectorTerms: - matchExpressions: - key: kubernetes.io/arch operator: In values: - amd64 - ppc64le - s390x - arm64 podAntiAffinity: preferredDuringSchedulingIgnoredDuringExecution: - podAffinityTerm: labelSelector: matchExpressions: - key: ocm-antiaffinity-selector operator: In values: - console-mce - key: component operator: In values: - console-mce topologyKey: topology.kubernetes.io/zone weight: 70 - podAffinityTerm: labelSelector: matchExpressions: - key: ocm-antiaffinity-selector operator: In values: - console-mce - key: component operator: In values: - console-mce topologyKey: kubernetes.io/hostname weight: 35 containers: - env: - name: PORT value: "3000" - name: CLUSTER_API_URL value: https://kubernetes.default.svc:443 image: brew.registry.redhat.io/rh-osbs/rhacm2/console-mce:2.5.0-PR1362-a93fd6412cd9956a0a21a5e2d3ebc8c4fd8c3966 imagePullPolicy: IfNotPresent livenessProbe: failureThreshold: 1 httpGet: path: /livenessProbe port: 3000 scheme: HTTPS initialDelaySeconds: 10 periodSeconds: 10 successThreshold: 1 timeoutSeconds: 1 name: console ports: - containerPort: 3000 protocol: TCP readinessProbe: failureThreshold: 1 httpGet: path: /readinessProbe port: 3000 scheme: HTTPS periodSeconds: 10 successThreshold: 1 timeoutSeconds: 1 resources: requests: cpu: 3m memory: 40Mi securityContext: allowPrivilegeEscalation: false capabilities: drop: - ALL privileged: false readOnlyRootFilesystem: true terminationMessagePath: /dev/termination-log terminationMessagePolicy: File volumeMounts: - mountPath: /app/certs name: console-mce-console-certs - mountPath: /app/config name: console-mce-console-config dnsPolicy: ClusterFirst imagePullSecrets: - name: multiclusterhub-operator-pull-secret restartPolicy: Always schedulerName: default-scheduler securityContext: runAsNonRoot: true serviceAccount: console-mce serviceAccountName: console-mce terminationGracePeriodSeconds: 30 tolerations: - effect: NoSchedule key: node-role.kubernetes.io/infra operator: Exists volumes: - name: console-mce-console-certs secret: defaultMode: 420 secretName: console-mce-console-certs - configMap: defaultMode: 420 name: console-config name: console-mce-console-config status: availableReplicas: 1 conditions: - lastTransitionTime: "2022-03-21T20:45:12Z" lastUpdateTime: "2022-03-21T20:45:12Z" message: Deployment does not have minimum availability. reason: MinimumReplicasUnavailable status: "False" type: Available - lastTransitionTime: "2022-03-21T20:45:12Z" lastUpdateTime: "2022-03-21T20:46:10Z" message: ReplicaSet "console-mce-console-bb897c7f5" is progressing. reason: ReplicaSetUpdated status: "True" type: Progressing observedGeneration: 3 readyReplicas: 1 replicas: 3 unavailableReplicas: 2 updatedReplicas: 2 oc get po -n multicluster-engine | grep -v 'Running\|Completed' NAME READY STATUS RESTARTS AGE console-mce-console-bb897c7f5-9lllp 0/1 ImagePullBackOff 0 10m console-mce-console-bb897c7f5-kqt8t 0/1 ImagePullBackOff 0 10m oc get event -n multicluster-engine | grep console-mce-console-bb897c7f5-9lllp 13m Normal Scheduled pod/console-mce-console-bb897c7f5-9lllp Successfully assigned multicluster-engine/console-mce-console-bb897c7f5-9lllp to fog04.cluster.internal 13m Normal AddedInterface pod/console-mce-console-bb897c7f5-9lllp Add eth0 [10.131.0.32/23] from ovn-kubernetes 12m Normal Pulling pod/console-mce-console-bb897c7f5-9lllp Pulling image "brew.registry.redhat.io/rh-osbs/rhacm2/console-mce:2.5.0-PR1362-a93fd6412cd9956a0a21a5e2d3ebc8c4fd8c3966" 12m Warning Failed pod/console-mce-console-bb897c7f5-9lllp Failed to pull image "brew.registry.redhat.io/rh-osbs/rhacm2/console-mce:2.5.0-PR1362-a93fd6412cd9956a0a21a5e2d3ebc8c4fd8c3966": rpc error: code = Unknown desc = reading manifest 2.5.0-PR1362-a93fd6412cd9956a0a21a5e2d3ebc8c4fd8c3966 in brew.registry.redhat.io/rh-osbs/rhacm2/console-mce: received unexpected HTTP status: 503 Service Unavailable 12m Warning Failed pod/console-mce-console-bb897c7f5-9lllp Error: ErrImagePull 3m23s Normal BackOff pod/console-mce-console-bb897c7f5-9lllp Back-off pulling image "brew.registry.redhat.io/rh-osbs/rhacm2/console-mce:2.5.0-PR1362-a93fd6412cd9956a0a21a5e2d3ebc8c4fd8c3966" 11m Warning Failed pod/console-mce-console-bb897c7f5-9lllp Error: ImagePullBackOff 13m Normal SuccessfulCreate replicaset/console-mce-console-bb897c7f5 Created pod: console-mce-console-bb897c7f5-9lllp
This looks to be the environment that I had patched to test the bare metal provisioning issue (the image points to my PR, which won't exist on brew.registry.redhat.io). Please let me know if you want me to restore the original image.
Please help to restore the original image.
G2Bsync 1075242858 comment KevinFCormier Tue, 22 Mar 2022 14:19:39 UTC G2Bsync Restored.
G2Bsync 1095660073 comment thuyn-581 Mon, 11 Apr 2022 22:40:12 UTC G2BSync Retest OK.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Important: Red Hat Advanced Cluster Management 2.5 security updates, images, and bug fixes), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2022:4956
The Red Hat OpenShift Cluster Manager application for OpenShift Container Platform allows you to deploy OpenShift clusters to either on-premise or cloud environments. OpenShift Container Platform 4.10 is supported on Red Hat Enterprise Linux (RHEL) 8.4 through 8.7, as well as on Red Hat Enterprise Linux CoreOS (RHCOS) 4.10. You must use RHCOS machines for the control plane, and you can use either RHCOS or RHEL for compute machines. https://www.kantimemedicare.us/
It's seem like environment. It resolved in a recent advisory to fix this bug and it has been closed with a resolution of ERRATA. https://www.myfedloan.us/