Bug 2066464

Summary:	console-mce-console pods throw ImagePullError after upgrading to ocp 4.10
Product:	Red Hat Advanced Cluster Management for Kubernetes	Reporter:	Thuy Nguyen <thnguyen>
Component:	Console	Assignee:	Kevin Cormier <kcormier>
Status:	CLOSED ERRATA	QA Contact:	Xiang Yin <xiyin>
Severity:	medium	Docs Contact:	Christopher Dawson <cdawson>
Priority:	unspecified
Version:	rhacm-2.5	CC:	devinrodgersr94316, dho, dhuynh, hershellasagna, huichen
Target Milestone:	---	Flags:	bot-tracker-sync: rhacm-2.5+
Target Release:	rhacm-2.5
Hardware:	x86_64
OS:	Unspecified
Whiteboard:
Fixed In Version:		Doc Type:	If docs needed, set a value
Doc Text:		Story Points:	---
Clone Of:		Environment:
Last Closed:	2022-06-09 02:09:56 UTC	Type:	Bug
Regression:	---	Mount Type:	---
Documentation:	---	CRM:
Verified Versions:		Category:	---
oVirt Team:	---	RHEL 7.3 requirements from Atomic Host:
Cloudforms Team:	---	Target Upstream Version:
Embargoed:

Description Thuy Nguyen 2022-03-21 21:00:22 UTC

Description of problem: console-mce pods show ImagePullError after upgrading to ocp 4.10


Version-Release number of selected component (if applicable):
ACM 2.5.0-DOWNSTREAM-2022-03-17-03-36-41 (Final S4)

How reproducible:


Steps to Reproduce:
1. Install ACM/MCH/MCE on ocp 4.9.23 successfully
2. Upgrade to ocp 4.10.5
3. MCE console is not available due to 'imagePullError'

Actual results:


Expected results:


Additional info:

oc get mce -n multicluster-engine -oyaml
apiVersion: v1
items:
- apiVersion: multicluster.openshift.io/v1
  kind: MultiClusterEngine
  metadata:
    annotations:
      imageRepository: brew.registry.redhat.io/rh-osbs/rhacm2
    creationTimestamp: "2022-03-18T14:20:38Z"
    finalizers:
    - finalizer.multicluster.openshift.io
    generation: 3
    labels:
      installer.name: multiclusterhub
      installer.namespace: ocm
    name: multiclusterengine
    resourceVersion: "219903084"
    uid: 3610c1ca-1d25-4175-89f4-84b55c80ac95
  spec:
    availabilityConfig: High
    imagePullSecret: multiclusterhub-operator-pull-secret
    overrides:
      components:
      - enabled: true
        name: assisted-service
      - enabled: true
        name: cluster-lifecycle
      - enabled: true
        name: cluster-manager
      - enabled: true
        name: discovery
      - enabled: true
        name: hive
      - enabled: true
        name: server-foundation
      - enabled: false
        name: managed-service-account
      - enabled: false
        name: hypershift-preview
      - enabled: true
        name: console-mce
    targetNamespace: multicluster-engine
    tolerations:
    - effect: NoSchedule
      key: node-role.kubernetes.io/infra
      operator: Exists
  status:
    components:
    - kind: Component
      lastTransitionTime: "2022-03-21T20:50:58Z"
      message: No resources present
      name: managedservice
      reason: ComponentDisabled
      status: "True"
      type: NotPresent
    - kind: Component
      lastTransitionTime: "2022-03-21T20:50:58Z"
      message: No resources present
      name: hypershift-addon-manager
      reason: ComponentDisabled
      status: "True"
      type: NotPresent
    - kind: Component
      lastTransitionTime: "2022-03-21T20:50:58Z"
      message: No resources present
      name: hypershift-deployment-controller
      reason: ComponentDisabled
      status: "True"
      type: NotPresent
    - kind: Deployment
      lastTransitionTime: "2022-03-21T20:45:12Z"
      message: Deployment does not have minimum availability.
      name: console-mce-console
      reason: MinimumReplicasUnavailable
      status: "False"
      type: Available
    conditions:
    - lastTransitionTime: "2022-03-21T20:28:58Z"
      lastUpdateTime: "2022-03-21T20:28:58Z"
      reason: ComponentsUnavailable
      status: "False"
      type: Available
    phase: Progressing
kind: List
metadata:
  resourceVersion: ""
  selfLink: ""


oc get deploy -n multicluster-engine
NAME                                  READY   UP-TO-DATE   AVAILABLE   AGE
agentinstalladmission                 2/2     2            2           7h7m
assisted-service                      1/1     1            1           7h7m
cluster-curator-controller            2/2     2            2           3d6h
cluster-manager                       3/3     3            3           3d6h
clusterclaims-controller              2/2     2            2           3d6h
clusterlifecycle-state-metrics-v2     1/1     1            1           3d6h
console-mce-console                   1/2     2            1           8m51s
discovery-operator                    1/1     1            1           3d6h
hive-operator                         1/1     1            1           3d6h
infrastructure-operator               1/1     1            1           3d6h
managedcluster-import-controller-v2   2/2     2            2           3d6h
multicluster-engine-operator          2/2     2            2           3d6h
ocm-controller                        2/2     2            2           3d6h
ocm-proxyserver                       2/2     2            2           3d6h
ocm-webhook                           2/2     2            2           3d6h
provider-credential-controller        1/1     1            1           3d6h


oc get deploy -n multicluster-engine console-mce-console -oyaml
apiVersion: apps/v1
kind: Deployment
metadata:
  annotations:
    deployment.kubernetes.io/revision: "3"
  creationTimestamp: "2022-03-21T20:45:12Z"
  generation: 3
  labels:
    backplaneconfig.name: multiclusterengine
  name: console-mce-console
  namespace: multicluster-engine
  ownerReferences:
  - apiVersion: multicluster.openshift.io/v1
    blockOwnerDeletion: true
    controller: true
    kind: MultiClusterEngine
    name: multiclusterengine
    uid: 3610c1ca-1d25-4175-89f4-84b55c80ac95
  resourceVersion: "219899462"
  uid: fe89da20-20ee-41d0-a354-832ecc7bc632
spec:
  progressDeadlineSeconds: 600
  replicas: 2
  revisionHistoryLimit: 10
  selector:
    matchLabels:
      app: console-mce
  strategy:
    rollingUpdate:
      maxSurge: 25%
      maxUnavailable: 25%
    type: RollingUpdate
  template:
    metadata:
      creationTimestamp: null
      labels:
        app: console-mce
        ocm-antiaffinity-selector: console-mce
    spec:
      affinity:
        nodeAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:
            - matchExpressions:
              - key: kubernetes.io/arch
                operator: In
                values:
                - amd64
                - ppc64le
                - s390x
                - arm64
        podAntiAffinity:
          preferredDuringSchedulingIgnoredDuringExecution:
          - podAffinityTerm:
              labelSelector:
                matchExpressions:
                - key: ocm-antiaffinity-selector
                  operator: In
                  values:
                  - console-mce
                - key: component
                  operator: In
                  values:
                  - console-mce
              topologyKey: topology.kubernetes.io/zone
            weight: 70
          - podAffinityTerm:
              labelSelector:
                matchExpressions:
                - key: ocm-antiaffinity-selector
                  operator: In
                  values:
                  - console-mce
                - key: component
                  operator: In
                  values:
                  - console-mce
              topologyKey: kubernetes.io/hostname
            weight: 35
      containers:
      - env:
        - name: PORT
          value: "3000"
        - name: CLUSTER_API_URL
          value: https://kubernetes.default.svc:443
        image: brew.registry.redhat.io/rh-osbs/rhacm2/console-mce:2.5.0-PR1362-a93fd6412cd9956a0a21a5e2d3ebc8c4fd8c3966
        imagePullPolicy: IfNotPresent
        livenessProbe:
          failureThreshold: 1
          httpGet:
            path: /livenessProbe
            port: 3000
            scheme: HTTPS
          initialDelaySeconds: 10
          periodSeconds: 10
          successThreshold: 1
          timeoutSeconds: 1
        name: console
        ports:
        - containerPort: 3000
          protocol: TCP
        readinessProbe:
          failureThreshold: 1
          httpGet:
            path: /readinessProbe
            port: 3000
            scheme: HTTPS
          periodSeconds: 10
          successThreshold: 1
          timeoutSeconds: 1
        resources:
          requests:
            cpu: 3m
            memory: 40Mi
        securityContext:
          allowPrivilegeEscalation: false
          capabilities:
            drop:
            - ALL
          privileged: false
          readOnlyRootFilesystem: true
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
        volumeMounts:
        - mountPath: /app/certs
          name: console-mce-console-certs
        - mountPath: /app/config
          name: console-mce-console-config
      dnsPolicy: ClusterFirst
      imagePullSecrets:
      - name: multiclusterhub-operator-pull-secret
      restartPolicy: Always
      schedulerName: default-scheduler
      securityContext:
        runAsNonRoot: true
      serviceAccount: console-mce
      serviceAccountName: console-mce
      terminationGracePeriodSeconds: 30
      tolerations:
      - effect: NoSchedule
        key: node-role.kubernetes.io/infra
        operator: Exists
      volumes:
      - name: console-mce-console-certs
        secret:
          defaultMode: 420
          secretName: console-mce-console-certs
      - configMap:
          defaultMode: 420
          name: console-config
        name: console-mce-console-config
status:
  availableReplicas: 1
  conditions:
  - lastTransitionTime: "2022-03-21T20:45:12Z"
    lastUpdateTime: "2022-03-21T20:45:12Z"
    message: Deployment does not have minimum availability.
    reason: MinimumReplicasUnavailable
    status: "False"
    type: Available
  - lastTransitionTime: "2022-03-21T20:45:12Z"
    lastUpdateTime: "2022-03-21T20:46:10Z"
    message: ReplicaSet "console-mce-console-bb897c7f5" is progressing.
    reason: ReplicaSetUpdated
    status: "True"
    type: Progressing
  observedGeneration: 3
  readyReplicas: 1
  replicas: 3
  unavailableReplicas: 2
  updatedReplicas: 2


oc get po -n multicluster-engine | grep -v 'Running\|Completed'
NAME                                                   READY   STATUS             RESTARTS      AGE
console-mce-console-bb897c7f5-9lllp                    0/1     ImagePullBackOff   0             10m
console-mce-console-bb897c7f5-kqt8t                    0/1     ImagePullBackOff   0             10m


oc get event -n multicluster-engine | grep console-mce-console-bb897c7f5-9lllp
13m         Normal    Scheduled                         pod/console-mce-console-bb897c7f5-9lllp                                   Successfully assigned multicluster-engine/console-mce-console-bb897c7f5-9lllp to fog04.cluster.internal
13m         Normal    AddedInterface                    pod/console-mce-console-bb897c7f5-9lllp                                   Add eth0 [10.131.0.32/23] from ovn-kubernetes
12m         Normal    Pulling                           pod/console-mce-console-bb897c7f5-9lllp                                   Pulling image "brew.registry.redhat.io/rh-osbs/rhacm2/console-mce:2.5.0-PR1362-a93fd6412cd9956a0a21a5e2d3ebc8c4fd8c3966"
12m         Warning   Failed                            pod/console-mce-console-bb897c7f5-9lllp                                   Failed to pull image "brew.registry.redhat.io/rh-osbs/rhacm2/console-mce:2.5.0-PR1362-a93fd6412cd9956a0a21a5e2d3ebc8c4fd8c3966": rpc error: code = Unknown desc = reading manifest 2.5.0-PR1362-a93fd6412cd9956a0a21a5e2d3ebc8c4fd8c3966 in brew.registry.redhat.io/rh-osbs/rhacm2/console-mce: received unexpected HTTP status: 503 Service Unavailable
12m         Warning   Failed                            pod/console-mce-console-bb897c7f5-9lllp                                   Error: ErrImagePull
3m23s       Normal    BackOff                           pod/console-mce-console-bb897c7f5-9lllp                                   Back-off pulling image "brew.registry.redhat.io/rh-osbs/rhacm2/console-mce:2.5.0-PR1362-a93fd6412cd9956a0a21a5e2d3ebc8c4fd8c3966"
11m         Warning   Failed                            pod/console-mce-console-bb897c7f5-9lllp                                   Error: ImagePullBackOff
13m         Normal    SuccessfulCreate                  replicaset/console-mce-console-bb897c7f5                                  Created pod: console-mce-console-bb897c7f5-9lllp

Comment 1 Kevin Cormier 2022-03-21 21:05:54 UTC

This looks to be the environment that I had patched to test the bare metal provisioning issue (the image points to my PR, which won't exist on brew.registry.redhat.io). Please let me know if you want me to restore the original image.

Comment 2 Thuy Nguyen 2022-03-21 21:28:45 UTC

Please help to restore the original image.

Comment 3 bot-tracker-sync 2022-03-22 16:43:31 UTC

G2Bsync 1075242858 comment 
 KevinFCormier Tue, 22 Mar 2022 14:19:39 UTC 
 G2Bsync Restored.

Comment 4 bot-tracker-sync 2022-04-12 00:20:50 UTC

G2Bsync 1095660073 comment 
 thuyn-581 Mon, 11 Apr 2022 22:40:12 UTC 
 G2BSync 
Retest OK.

Comment 8 errata-xmlrpc 2022-06-09 02:09:56 UTC

Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (Important: Red Hat Advanced Cluster Management 2.5 security updates, images, and bug fixes), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHSA-2022:4956

Comment 9 Fields44 2023-01-09 08:50:46 UTC Comment hidden (spam)

The Red Hat OpenShift Cluster Manager application for OpenShift Container Platform allows you to deploy OpenShift clusters to either on-premise or cloud environments.

OpenShift Container Platform 4.10 is supported on Red Hat Enterprise Linux (RHEL) 8.4 through 8.7, as well as on Red Hat Enterprise Linux CoreOS (RHCOS) 4.10.

You must use RHCOS machines for the control plane, and you can use either RHCOS or RHEL for compute machines.
https://www.kantimemedicare.us/

Comment 10 Michelle 2023-06-22 08:29:00 UTC Comment hidden (spam)

It's seem like environment. It resolved in a recent advisory to fix this bug and it has been closed with a resolution of ERRATA. https://www.myfedloan.us/