Description of problem: Add-on installation failed. pods are in ImagePullBackOff status. Warning Failed 13s (x2 over 26s) kubelet Failed to pull image "quay.io/rhceph-dev/odf4-mcg-rhel8-operator@sha256:092cbae08e6b7c4471c716a4e47894865a6b06c1df64dffeda8f401dba684101": rpc error: code = Unknown desc = reading manifest sha256:092cbae08e6b7c4471c716a4e47894865a6b06c1df64dffeda8f401dba684101 in quay.io/rhceph-dev/odf4-mcg-rhel8-operator: unauthorized: access to the requested resource is not authorized Version-Release number of selected component (if applicable): mcg-operator.v4.10.0 ocs-operator.v4.10.0 (4.10.0-197) ocs-osd-deployer.v2.0.0 odf-operator.v4.10.0 ose-prometheus-operator.4.8.0 route-monitor-operator.v0.1.406-54ff884 How reproducible: 4/4 Steps to Reproduce: 1. Install Rosa cluster 2. Create openshift-storage namespace and Create catalog source in openshift-storage namespace (yaml shared in additional information) 3. initiate provider add-on installation Actual results: Add-on installation failed with error install-check Failed and pods are in ImagePullBackOff Status and Expected results: Deployment should succeed. all CSV should be in the succeeded state and all pods come up and running Additional info: followed the section : Install deployer without global-pullSecret https://docs.google.com/document/d/1CqkNxoAO8frzOX9RhDF70aMGQ6JoEHj4wMcPZssaza4/edit#bookmark=id.tu7j8ebr76i ==== openshift-marketplace==== NAME DISPLAY TYPE PUBLISHER AGE addon-ocs-provider-qe-catalog Red Hat OpenShift Data Foundation Managed Service (provider-qe) grpc OSD Red Hat Addons 99m certified-operators Certified Operators grpc Red Hat 128m community-operators Community Operators grpc Red Hat 128m redhat-marketplace Red Hat Marketplace grpc Red Hat 128m redhat-operators Red Hat Operators grpc Red Hat 128m ============waiting for next iteration =============== Thu Mar 17 10:21:57 AM UTC 2022 -------------- ========CSV ====== NAME DISPLAY VERSION REPLACES PHASE mcg-operator.v4.10.0 NooBaa Operator 4.10.0 Failed ocs-operator.v4.10.0 OpenShift Container Storage 4.10.0 Failed ocs-osd-deployer.v2.0.0 OCS OSD Deployer 2.0.0 Failed odf-operator.v4.10.0 OpenShift Data Foundation 4.10.0 Failed ose-prometheus-operator.4.8.0 Prometheus Operator 4.8.0 Succeeded route-monitor-operator.v0.1.406-54ff884 Route Monitor Operator 0.1.406-54ff884 route-monitor-operator.v0.1.404-e29b74b Succeeded -------------- =======PODS ====== NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES 70dced7c0d3ac47187f71a0e9c7fe76110fbf3f3b9d00cd463c70a--1-fdj5x 0/1 Completed 0 99m 10.131.0.11 ip-10-0-157-209.us-east-2.compute.internal <none> <none> 8f8921fe86dbf507190d5f468306e24ad6dc498bf087400ad60d07--1-6nwcx 0/1 Completed 0 99m 10.131.0.14 ip-10-0-157-209.us-east-2.compute.internal <none> <none> bd70b2e99d54a7122624e6d4b19adf74de9396bac492401ee13864--1-q6lsn 0/1 Completed 0 99m 10.131.0.10 ip-10-0-157-209.us-east-2.compute.internal <none> <none> noobaa-operator-b67844956-cj8w7 0/1 ImagePullBackOff 0 16m 10.128.2.34 ip-10-0-188-230.us-east-2.compute.internal <none> <none> ocs-metrics-exporter-5597c58685-lq69l 1/1 Running 0 98m 10.131.0.18 ip-10-0-157-209.us-east-2.compute.internal <none> <none> ocs-operator-7b6d5f999b-2nrnf 1/1 Running 0 98m 10.131.0.16 ip-10-0-157-209.us-east-2.compute.internal <none> <none> ocs-osd-controller-manager-555b8667fb-vjtw9 2/3 Running 0 92m 10.129.2.35 ip-10-0-198-253.us-east-2.compute.internal <none> <none> ocs-osd-controller-manager-749c5c8f5d-cb49p 2/3 Running 0 98m 10.131.0.19 ip-10-0-157-209.us-east-2.compute.internal <none> <none> odf-console-77b6ddffb8-m55rh 0/1 ErrImagePull 0 16m 10.128.2.35 ip-10-0-188-230.us-east-2.compute.internal <none> <none> odf-operator-controller-manager-9f8898b5-n9h68 2/2 Running 0 86m 10.131.0.35 ip-10-0-157-209.us-east-2.compute.internal <none> <none> prometheus-operator-5dc6c569-cqf4m 1/1 Running 0 98m 10.131.0.20 ip-10-0-157-209.us-east-2.compute.internal <none> <none> redhat-operators-zks26 1/1 Running 0 103m 10.129.2.30 ip-10-0-198-253.us-east-2.compute.internal <none> <none> rook-ceph-operator-c9d47548c-zs5rl 0/1 ImagePullBackOff 0 15m 10.128.2.37 ip-10-0-188-230.us-east-2.compute.internal <none> <none> ========================== Workaround : Create your quay credentials. To get the password for the quay account click on Account Settings Then click on Generate Encrypted Password. Use the displayed encrypted password oc registry login --registry="quay.io" --auth-basic="<quay_username>:<quay_password>" --to=temp1.json Set your credentials to addon-pull secret oc set data secret/addon-pullsecret -n openshift-storage --from-file=.dockerconfigjson=temp1.json keep trying creating pull secret and restart the pods which are in ImagePullBackOff status. with reconciliation timimg pod will come to running status
Raised an MR to fix this - https://gitlab.cee.redhat.com/service/managed-tenants/-/merge_requests/1960
This issue is fixed after a new MR is merged, I've verified it too. Here is the link: https://gitlab.cee.redhat.com/service/managed-tenants/-/merge_requests/1971