Bug 2065102

Summary: Add-on installation failed with error installCheck Failed - unauthorized access to the requested resource(pods -ImagePullBackOff status)
Product: [Red Hat Storage] Red Hat OpenShift Data Foundation Reporter: suchita <sgatfane>
Component: odf-managed-serviceAssignee: Akshay Khanna <akkhanna>
Status: CLOSED CURRENTRELEASE QA Contact: Neha Berry <nberry>
Severity: high Docs Contact:
Priority: unspecified    
Version: 4.10CC: aeyal, akkhanna, ebenahar, nberry, ocs-bugs, odf-bz-bot, rperiyas, sabose
Target Milestone: ---   
Target Release: ---   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2022-04-21 09:12:53 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description suchita 2022-03-17 11:23:37 UTC 
Description of problem:
Add-on installation failed. pods are in ImagePullBackOff status. 

 Warning  Failed          13s (x2 over 26s)  kubelet            Failed to pull image "quay.io/rhceph-dev/odf4-mcg-rhel8-operator@sha256:092cbae08e6b7c4471c716a4e47894865a6b06c1df64dffeda8f401dba684101": rpc error: code = Unknown desc = reading manifest sha256:092cbae08e6b7c4471c716a4e47894865a6b06c1df64dffeda8f401dba684101 in quay.io/rhceph-dev/odf4-mcg-rhel8-operator: unauthorized: access to the requested resource is not authorized


Version-Release number of selected component (if applicable):

mcg-operator.v4.10.0 
ocs-operator.v4.10.0  (4.10.0-197)
ocs-osd-deployer.v2.0.0
odf-operator.v4.10.0   
ose-prometheus-operator.4.8.0
route-monitor-operator.v0.1.406-54ff884

How reproducible:
4/4

Steps to Reproduce:
1. Install Rosa cluster
2. Create openshift-storage namespace and Create catalog source in openshift-storage namespace (yaml shared in additional information)
3. initiate provider add-on installation

Actual results:
Add-on installation failed with error install-check Failed
and 
pods are in ImagePullBackOff Status
and 



Expected results:
Deployment should succeed. 
all CSV should be in the succeeded state and all pods  come up and running 




Additional info:
followed the section : Install deployer without global-pullSecret
https://docs.google.com/document/d/1CqkNxoAO8frzOX9RhDF70aMGQ6JoEHj4wMcPZssaza4/edit#bookmark=id.tu7j8ebr76i

==== openshift-marketplace====
NAME                            DISPLAY                                                           TYPE   PUBLISHER            AGE
addon-ocs-provider-qe-catalog   Red Hat OpenShift Data Foundation Managed Service (provider-qe)   grpc   OSD Red Hat Addons   99m
certified-operators             Certified Operators                                               grpc   Red Hat              128m
community-operators             Community Operators                                               grpc   Red Hat              128m
redhat-marketplace              Red Hat Marketplace                                               grpc   Red Hat              128m
redhat-operators                Red Hat Operators                                                 grpc   Red Hat              128m
============waiting for next iteration ===============
Thu Mar 17 10:21:57 AM UTC 2022
--------------
========CSV ======
NAME                                      DISPLAY                       VERSION           REPLACES                                  PHASE
mcg-operator.v4.10.0                      NooBaa Operator               4.10.0                                                      Failed
ocs-operator.v4.10.0                      OpenShift Container Storage   4.10.0                                                      Failed
ocs-osd-deployer.v2.0.0                   OCS OSD Deployer              2.0.0                                                       Failed
odf-operator.v4.10.0                      OpenShift Data Foundation     4.10.0                                                      Failed
ose-prometheus-operator.4.8.0             Prometheus Operator           4.8.0                                                       Succeeded
route-monitor-operator.v0.1.406-54ff884   Route Monitor Operator        0.1.406-54ff884   route-monitor-operator.v0.1.404-e29b74b   Succeeded
--------------
=======PODS ======
NAME                                                              READY   STATUS             RESTARTS   AGE    IP            NODE                                         NOMINATED NODE   READINESS GATES
70dced7c0d3ac47187f71a0e9c7fe76110fbf3f3b9d00cd463c70a--1-fdj5x   0/1     Completed          0          99m    10.131.0.11   ip-10-0-157-209.us-east-2.compute.internal   <none>           <none>
8f8921fe86dbf507190d5f468306e24ad6dc498bf087400ad60d07--1-6nwcx   0/1     Completed          0          99m    10.131.0.14   ip-10-0-157-209.us-east-2.compute.internal   <none>           <none>
bd70b2e99d54a7122624e6d4b19adf74de9396bac492401ee13864--1-q6lsn   0/1     Completed          0          99m    10.131.0.10   ip-10-0-157-209.us-east-2.compute.internal   <none>           <none>
noobaa-operator-b67844956-cj8w7                                   0/1     ImagePullBackOff   0          16m    10.128.2.34   ip-10-0-188-230.us-east-2.compute.internal   <none>           <none>
ocs-metrics-exporter-5597c58685-lq69l                             1/1     Running            0          98m    10.131.0.18   ip-10-0-157-209.us-east-2.compute.internal   <none>           <none>
ocs-operator-7b6d5f999b-2nrnf                                     1/1     Running            0          98m    10.131.0.16   ip-10-0-157-209.us-east-2.compute.internal   <none>           <none>
ocs-osd-controller-manager-555b8667fb-vjtw9                       2/3     Running            0          92m    10.129.2.35   ip-10-0-198-253.us-east-2.compute.internal   <none>           <none>
ocs-osd-controller-manager-749c5c8f5d-cb49p                       2/3     Running            0          98m    10.131.0.19   ip-10-0-157-209.us-east-2.compute.internal   <none>           <none>
odf-console-77b6ddffb8-m55rh                                      0/1     ErrImagePull       0          16m    10.128.2.35   ip-10-0-188-230.us-east-2.compute.internal   <none>           <none>
odf-operator-controller-manager-9f8898b5-n9h68                    2/2     Running            0          86m    10.131.0.35   ip-10-0-157-209.us-east-2.compute.internal   <none>           <none>
prometheus-operator-5dc6c569-cqf4m                                1/1     Running            0          98m    10.131.0.20   ip-10-0-157-209.us-east-2.compute.internal   <none>           <none>
redhat-operators-zks26                                            1/1     Running            0          103m   10.129.2.30   ip-10-0-198-253.us-east-2.compute.internal   <none>           <none>
rook-ceph-operator-c9d47548c-zs5rl                                0/1     ImagePullBackOff   0          15m    10.128.2.37   ip-10-0-188-230.us-east-2.compute.internal   <none>           <none>
==========================

Workaround :  
Create your quay credentials. To get the password for the quay account click on Account Settings
Then click on Generate Encrypted Password. Use the displayed encrypted password

oc registry login --registry="quay.io" --auth-basic="<quay_username>:<quay_password>" --to=temp1.json

Set your credentials to addon-pull secret

oc set data secret/addon-pullsecret -n openshift-storage --from-file=.dockerconfigjson=temp1.json

keep trying creating pull secret and restart the pods which are in ImagePullBackOff status. 

with reconciliation timimg pod will come to running status
Comment 1 Akshay Khanna 2022-03-17 12:49:40 UTC 
Raised an MR to fix this - https://gitlab.cee.redhat.com/service/managed-tenants/-/merge_requests/1960
Comment 3 Akshay Khanna 2022-03-21 11:54:47 UTC 
This issue is fixed after a new MR is merged, I've verified it too.

Here is the link: https://gitlab.cee.redhat.com/service/managed-tenants/-/merge_requests/1971