Bug 2071088

Summary: PODs in ImagePullBackOff with deployer-2.0.0-8 and ODF catsrc image 4.10.0-218 for ODF MS(serviceaccount misses correct ImagePullSecretRef)
Product: [Red Hat Storage] Red Hat OpenShift Data Foundation Reporter: Neha Berry <nberry>
Component: odf-managed-serviceAssignee: Ohad <omitrani>
Status: CLOSED CURRENTRELEASE QA Contact: Neha Berry <nberry>
Severity: urgent Docs Contact:
Priority: unspecified    
Version: 4.10CC: aeyal, branto, madam, muagarwa, ocs-bugs, odf-bz-bot, owasserm, sabose, sheggodu
Target Milestone: ---   
Target Release: ODF 4.10.0   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2022-04-21 09:12:54 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Neha Berry 2022-04-01 18:59:04 UTC
Description of problem:
=====================================
Provider QE addon installaion from OCM UI: Operator pods in ImagePullBackOff in deployer version 2.0.0-8 and catalogsource version 4.10.0-218 which was created for MS ODF

The issue is with the image  quay.io/rhceph-dev/odf4-odf-compose-operator-bundle-index:4.10.0-218  
ServiceAccounts are missing imagePullSecretRef

oc get serviceaccount ocs-operator -o yaml
apiVersion: v1
imagePullSecrets:
- name: ocs-operator-dockercfg-flk6c
kind: ServiceAccount
metadata:
  creationTimestamp: "2022-04-01T18:28:04Z"
  labels:
    operators.coreos.com/ocs-operator.openshift-storage: ""
  name: ocs-operator
  namespace: openshift-storage
  ownerReferences:
  - apiVersion: operators.coreos.com/v1alpha1
    blockOwnerDeletion: false
    controller: false
    kind: ClusterServiceVersion
    name: ocs-operator.v4.10.0
    uid: 9b0184f1-aa13-4263-ab46-51a7661b1c83
  resourceVersion: "353540"
  uid: 6b47a6af-2485-473b-87d5-d512da09887f
secrets:
- name: ocs-operator-token-974ck
- name: ocs-operator-dockercfg-flk6c

Version-Release number of selected component (if applicable):
================================================================
oc describe catsrc redhat-operators |grep -i image
        f:image:
  Image:         quay.io/rhceph-dev/odf4-odf-compose-operator-bundle-index:4.10.0-218

oc get csv -n openshift-storage -o json ocs-operator.v4.10.0 | jq '.metadata.labels["full_version"]'
"4.10.0-217"

oc describe csv ocs-osd-deployer.v2.0.0 |grep -i image                                              
    Mediatype:   image/svg+xml
                Image:  quay.io/openshift/origin-kube-rbac-proxy:4.10.0
                Image:             quay.io/osd-addons/ocs-osd-deployer:2.0.0-8
                Image:             quay.io/osd-addons/ocs-osd-deployer:2.0.0-8


OCP = 4.9.24


How reproducible:
=======================
Always

Steps to Reproduce:
=======================
1. Create a ROSA provider cluster
2. Install provider-QE add-on from OCM UI providing the onbaording pubkey and size as 4TB/20TB
3. Check the status of CSV and pods in openshift-storage namespace

Actual results:
==============
PODs in ImagePullBackOff


Expected results:
=======================
Service account imagePullSecretsRef should point to the correct secret.
imagePullSecrets:
- name: addon-pullsecret
- name: ocs-operator-dockercfg-gsz45


Additional info:
==================
--------------
========CSV ======
NAME                                      DISPLAY                       VERSION           REPLACES                                  PHASE
mcg-operator.v4.10.0                      NooBaa Operator               4.10.0                                                      Failed
ocs-operator.v4.10.0                      OpenShift Container Storage   4.10.0                                                      Failed
ocs-osd-deployer.v2.0.0                   OCS OSD Deployer              2.0.0                                                       Installing
odf-operator.v4.10.0                      OpenShift Data Foundation     4.10.0                                                      Failed
ose-prometheus-operator.4.8.0             Prometheus Operator           4.8.0                                                       Succeeded
route-monitor-operator.v0.1.408-c2256a2   Route Monitor Operator        0.1.408-c2256a2   route-monitor-operator.v0.1.406-54ff884   Succeeded
--------------
=======PODS ======
NAME                                                              READY   STATUS              RESTARTS   AGE     IP            NODE                           NOMINATED NODE   READINESS GATES
4aabeb718dda0121db1bd9db302cb7e120f7eca652baeb490b9724--1-tc8c4   0/1     Completed           0          13m     10.128.3.43   ip-10-0-206-203.ec2.internal   <none>           <none>
547c24e226a293c362220fdd33b338c36e1388c9a5ada27a96ff1c--1-6mdkg   0/1     Completed           0          13m     10.128.3.39   ip-10-0-206-203.ec2.internal   <none>           <none>
e0f50bcd4923a044437e6ca88e00f62afbfd091465975e0b3b0b0d--1-g7cpq   0/1     Completed           0          13m     10.128.3.42   ip-10-0-206-203.ec2.internal   <none>           <none>
noobaa-operator-77f7bc65c-5h86w                                   0/1     ImagePullBackOff    0          13m     10.129.2.28   ip-10-0-168-85.ec2.internal    <none>           <none>
ocs-metrics-exporter-58f8489f67-4v56f                             0/1     ImagePullBackOff    0          13m     10.129.2.30   ip-10-0-168-85.ec2.internal    <none>           <none>
ocs-operator-c7db7cc9b-9zdm9                                      0/1     ImagePullBackOff    0          13m     10.131.0.37   ip-10-0-133-183.ec2.internal   <none>           <none>
ocs-osd-controller-manager-585c999dcc-nl9hv                       2/3     Running             0          8m23s   10.128.3.46   ip-10-0-206-203.ec2.internal   <none>           <none>
odf-console-674b4b894-k247q                                       0/1     ContainerCreating   0          13m     <none>        ip-10-0-206-203.ec2.internal   <none>           <none>
odf-operator-controller-manager-9bf866dbc-jvpxp                   0/2     ImagePullBackOff    0          13m     10.128.3.44   ip-10-0-206-203.ec2.internal   <none>           <none>
prometheus-operator-5dc6c569-hwcs4                                1/1     Running             0          13m     10.129.2.27   ip-10-0-168-85.ec2.internal    <none>           <none>
redhat-operators-2f2dk                                            1/1     Running             0          14m     10.129.2.26   ip-10-0-168-85.ec2.internal    <none>           <none>
rook-ceph-operator-994b5d895-8fxmd                                0/1     ImagePullBackOff    0          13m     10.129.2.29   ip-10-0-168-85.ec2.internal    <none>           <none>