Description of problem: When using the private index image, the catalogSource pod works well. But, the bundle pulling failed. I guess the root cause is that the CatalogSource secret wasn't added to this bundle pod. [root@preserve-olm-env data]# oc get job NAME COMPLETIONS DURATION AGE d3383223ae0bc9c573b86c2d918fe9f6f9988f5771d0a503da2943d2fa37991 0/1 12m 12m [root@preserve-olm-env data]# oc get pods [root@preserve-olm-env data]# oc get pods NAME READY STATUS RESTARTS AGE d3383223ae0bc9c573b86c2d918fe9f6f9988f5771d0a503da2943d2faxzbsk 0/1 Init:ImagePullBackOff 0 12m Version-Release number of selected component (if applicable): [root@preserve-olm-env data]# oc version Client Version: 4.7.0-0.nightly-2020-12-16-224526 Server Version: 4.7.0-0.nightly-2020-12-20-031835 Kubernetes Version: v1.20.0+87544c5 [root@preserve-olm-env data]# oc -n openshift-operator-lifecycle-manager exec catalog-operator-7c9b666d58-zvmsd -- olm --version OLM version: 0.17.0 git commit: ffb66b0f4150bb82cbc6a1e7571901fe7724f91e How reproducible: always Steps to Reproduce: 1. Install OCP 4.7, and login it as a cluster-admin rolee. 2. Create a private image on Quay.io and login it, for example, quay.io/jiazha/upstream-opm-builder [root@preserve-olm-env data]# docker login quay.io Authenticating with existing credentials... WARNING! Your password will be stored unencrypted in /root/.docker/config.json. Configure a credential helper to remove this warning. See https://docs.docker.com/engine/reference/commandline/login/#credentials-store Login Succeeded 3. Replace the etcd operator image with the private image. And, build the private bundle and index image. [root@preserve-olm-env etcd]# opm alpha bundle build -c clusterwide-alpha -e clusterwide-alpha -d ./0.9.4-clusterwide/ -p etcd -o -t quay.io/jiazha/upstream-opm-builder:etcdbundle ... [root@preserve-olm-env etcd]# docker push quay.io/jiazha/upstream-opm-builder:etcdbundle The push refers to repository [quay.io/jiazha/upstream-opm-builder] ... [root@preserve-olm-env etcd]# opm index add -b quay.io/jiazha/upstream-opm-builder:etcdbundle -c docker -t quay.io/jiazha/upstream-opm-builder:etcdindex INFO[0000] building the index bundles="[quay.io/jiazha/upstream-opm-builder:etcdbundle]" ... [root@preserve-olm-env etcd]# docker push quay.io/jiazha/upstream-opm-builder:etcdindex ... 4, Create a secret that contains the auth of this private repo, like below: [root@preserve-olm-env data]# oc create secret generic secret-cs --from-file=.dockerconfigjson=/root/.docker/config.json --type=kubernetes.io/dockerconfigjson secret/secret-cs created 5. Create a CatalogSource to consume this private image, and set the `spec.secrets` fields to consume this secret. [root@preserve-olm-env data]# cat cs-private.yaml apiVersion: operators.coreos.com/v1alpha1 kind: CatalogSource metadata: name: cs-private namespace: openshift-marketplace spec: displayName: Jian Test publisher: Jian sourceType: grpc secrets: - "secret-cs" image: quay.io/jiazha/upstream-opm-builder:etcdindex updateStrategy: registryPoll: interval: 10m [root@preserve-olm-env data]# oc create -f cs-private.yaml catalogsource.operators.coreos.com/etcd-test created [root@preserve-olm-env data]# oc get catalogsource NAME DISPLAY TYPE PUBLISHER AGE cs-private Jian Test grpc Jian 24m [root@preserve-olm-env data]# oc get pods NAME READY STATUS RESTARTS AGE cs-private-c2qrf 1/1 Running 0 26m [root@preserve-olm-env data]# oc get packagemanifest|grep etcd etcd Jian Test 25m etcd Community Operators 8h 6, Subscribe to the etcd operator from this CatalogSource. [root@preserve-olm-env data]# cat sub-etcd-cluster.yaml apiVersion: operators.coreos.com/v1alpha1 kind: Subscription metadata: name: etcd namespace: openshift-operators spec: channel: clusterwide-alpha installPlanApproval: Automatic name: etcd source: cs-private sourceNamespace: openshift-marketplace startingCSV: etcdoperator.v0.9.4-clusterwide [root@preserve-olm-env data]# oc get sub -A NAMESPACE NAME PACKAGE SOURCE CHANNEL e2e-test-compliance-ktj2cslj-7vvvl compliance-operator compliance-operator compliance-operator 4.6 openshift-operators etcd etcd cs-private clusterwide-alpha [root@preserve-olm-env data]# oc get ip -n openshift-operators NAME CSV APPROVAL APPROVED install-8xgk4 etcdoperator.v0.9.4-clusterwide Automatic true [root@preserve-olm-env data]# oc get csv -n openshift-operators No resources found in openshift-operators namespace. Actual results: The bundle pod failed to pull the private bundle image: [root@preserve-olm-env data]# oc get job NAME COMPLETIONS DURATION AGE d3383223ae0bc9c573b86c2d918fe9f6f9988f5771d0a503da2943d2fa37991 0/1 29s 29s [root@preserve-olm-env data]# oc get pods NAME READY STATUS RESTARTS AGE d3383223ae0bc9c573b86c2d918fe9f6f9988f5771d0a503da2943d2faxzbsk 0/1 Init:ImagePullBackOff 0 12m Normal BackOff <invalid> (x5 over <invalid>) kubelet Back-off pulling image "quay.io/jiazha/upstream-opm-builder:etcdbundle" Warning Failed <invalid> (x5 over <invalid>) kubelet Error: ImagePullBackOff Normal Pulling <invalid> (x4 over <invalid>) kubelet Pulling image "quay.io/jiazha/upstream-opm-builder:etcdbundle" Warning Failed <invalid> (x4 over <invalid>) kubelet Error: ErrImagePull Expected results: The bundle pods can use the CatalogSource secret to pull the private bundle image successfully. Additional info: [root@preserve-olm-env data]# oc get pods d3383223ae0bc9c573b86c2d918fe9f6f9988f5771d0a503da2943d2faxzbsk -o yaml apiVersion: v1 kind: Pod metadata: annotations: k8s.v1.cni.cncf.io/network-status: |- [{ "name": "", "interface": "eth0", "ips": [ "10.128.2.191" ], "default": true, "dns": {} }] k8s.v1.cni.cncf.io/networks-status: |- [{ "name": "", "interface": "eth0", "ips": [ "10.128.2.191" ], "default": true, "dns": {} }] openshift.io/scc: restricted creationTimestamp: "2020-12-22T09:01:35Z" generateName: d3383223ae0bc9c573b86c2d918fe9f6f9988f5771d0a503da2943d2fa37991- labels: controller-uid: a8dd76a8-ce65-467e-b70e-9bf971d3f899 job-name: d3383223ae0bc9c573b86c2d918fe9f6f9988f5771d0a503da2943d2fa37991 managedFields: - apiVersion: v1 fieldsType: FieldsV1 fieldsV1: f:metadata: f:generateName: {} f:labels: .: {} f:controller-uid: {} f:job-name: {} f:ownerReferences: .: {} k:{"uid":"a8dd76a8-ce65-467e-b70e-9bf971d3f899"}: .: {} f:apiVersion: {} f:blockOwnerDeletion: {} f:controller: {} f:kind: {} f:name: {} f:uid: {} f:spec: f:containers: k:{"name":"extract"}: .: {} f:command: {} f:env: .: {} k:{"name":"CONTAINER_IMAGE"}: .: {} f:name: {} f:value: {} f:image: {} f:imagePullPolicy: {} f:name: {} f:resources: {} f:terminationMessagePath: {} f:terminationMessagePolicy: {} f:volumeMounts: .: {} k:{"mountPath":"/bundle"}: .: {} f:mountPath: {} f:name: {} f:dnsPolicy: {} f:enableServiceLinks: {} f:initContainers: .: {} k:{"name":"pull"}: .: {} f:command: {} f:image: {} f:imagePullPolicy: {} f:name: {} f:resources: {} f:terminationMessagePath: {} f:terminationMessagePolicy: {} f:volumeMounts: .: {} k:{"mountPath":"/bundle"}: .: {} f:mountPath: {} f:name: {} k:{"mountPath":"/util"}: .: {} f:mountPath: {} f:name: {} k:{"name":"util"}: .: {} f:command: {} f:image: {} f:imagePullPolicy: {} f:name: {} f:resources: {} f:terminationMessagePath: {} f:terminationMessagePolicy: {} f:volumeMounts: .: {} k:{"mountPath":"/util"}: .: {} f:mountPath: {} f:name: {} f:restartPolicy: {} f:schedulerName: {} f:securityContext: .: {} f:fsGroup: {} f:seLinuxOptions: f:level: {} f:terminationGracePeriodSeconds: {} f:volumes: .: {} k:{"name":"bundle"}: .: {} f:emptyDir: {} f:name: {} k:{"name":"util"}: .: {} f:emptyDir: {} f:name: {} manager: kube-controller-manager operation: Update time: "2020-12-22T09:01:35Z" - apiVersion: v1 fieldsType: FieldsV1 fieldsV1: f:metadata: f:annotations: f:k8s.v1.cni.cncf.io/network-status: {} f:k8s.v1.cni.cncf.io/networks-status: {} manager: multus operation: Update time: "2020-12-22T09:01:37Z" - apiVersion: v1 fieldsType: FieldsV1 fieldsV1: f:status: f:conditions: k:{"type":"ContainersReady"}: .: {} f:lastProbeTime: {} f:lastTransitionTime: {} f:message: {} f:reason: {} f:status: {} f:type: {} k:{"type":"Initialized"}: .: {} f:lastProbeTime: {} f:lastTransitionTime: {} f:message: {} f:reason: {} f:status: {} f:type: {} k:{"type":"Ready"}: .: {} f:lastProbeTime: {} f:lastTransitionTime: {} f:message: {} f:reason: {} f:status: {} f:type: {} f:containerStatuses: {} f:hostIP: {} f:initContainerStatuses: {} f:podIP: {} f:podIPs: .: {} k:{"ip":"10.128.2.191"}: .: {} f:ip: {} f:startTime: {} manager: kubelet operation: Update time: "2020-12-22T09:01:40Z" name: d3383223ae0bc9c573b86c2d918fe9f6f9988f5771d0a503da2943d2faxzbsk namespace: openshift-marketplace ownerReferences: - apiVersion: batch/v1 blockOwnerDeletion: true controller: true kind: Job name: d3383223ae0bc9c573b86c2d918fe9f6f9988f5771d0a503da2943d2fa37991 uid: a8dd76a8-ce65-467e-b70e-9bf971d3f899 resourceVersion: "205999" uid: dc3cc527-8cac-470f-8c84-8765b5be2afd spec: containers: - command: - opm - alpha - bundle - extract - -m - /bundle/ - -n - openshift-marketplace - -c - d3383223ae0bc9c573b86c2d918fe9f6f9988f5771d0a503da2943d2fa37991 env: - name: CONTAINER_IMAGE value: quay.io/jiazha/upstream-opm-builder:etcdbundle image: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:262b08f1fc93d6409bc612a11687821371e10fbf8301f3f94db2234d619ccc87 imagePullPolicy: IfNotPresent name: extract resources: {} securityContext: capabilities: drop: - KILL - MKNOD - SETGID - SETUID runAsUser: 1000270000 terminationMessagePath: /dev/termination-log terminationMessagePolicy: File volumeMounts: - mountPath: /bundle name: bundle - mountPath: /var/run/secrets/kubernetes.io/serviceaccount name: default-token-dzp95 readOnly: true dnsPolicy: ClusterFirst enableServiceLinks: true imagePullSecrets: - name: default-dockercfg-98ccr initContainers: - command: - /bin/cp - -Rv - /bin/cpb - /util/cpb image: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:82a6e2add54339cdc10687b4c5d827ec8106cc56422e7039c131e7727755b3c9 imagePullPolicy: IfNotPresent name: util resources: {} securityContext: capabilities: drop: - KILL - MKNOD - SETGID - SETUID runAsUser: 1000270000 terminationMessagePath: /dev/termination-log terminationMessagePolicy: File volumeMounts: - mountPath: /util name: util - mountPath: /var/run/secrets/kubernetes.io/serviceaccount name: default-token-dzp95 readOnly: true - command: - /util/cpb - /bundle image: quay.io/jiazha/upstream-opm-builder:etcdbundle imagePullPolicy: Always name: pull resources: {} securityContext: capabilities: drop: - KILL - MKNOD - SETGID - SETUID runAsUser: 1000270000 terminationMessagePath: /dev/termination-log terminationMessagePolicy: File volumeMounts: - mountPath: /bundle name: bundle - mountPath: /util name: util - mountPath: /var/run/secrets/kubernetes.io/serviceaccount name: default-token-dzp95 readOnly: true nodeName: ip-10-0-158-39.us-east-2.compute.internal preemptionPolicy: PreemptLowerPriority priority: 0 restartPolicy: OnFailure schedulerName: default-scheduler securityContext: fsGroup: 1000270000 seLinuxOptions: level: s0:c16,c15 serviceAccount: default serviceAccountName: default terminationGracePeriodSeconds: 30 tolerations: - effect: NoExecute key: node.kubernetes.io/not-ready operator: Exists tolerationSeconds: 300 - effect: NoExecute key: node.kubernetes.io/unreachable operator: Exists tolerationSeconds: 300 volumes: - emptyDir: {} name: bundle - emptyDir: {} name: util - name: default-token-dzp95 secret: defaultMode: 420 secretName: default-token-dzp95 status: conditions: - lastProbeTime: null lastTransitionTime: "2020-12-22T09:01:35Z" message: 'containers with incomplete status: [pull]' reason: ContainersNotInitialized status: "False" type: Initialized - lastProbeTime: null lastTransitionTime: "2020-12-22T09:01:35Z" message: 'containers with unready status: [extract]' reason: ContainersNotReady status: "False" type: Ready - lastProbeTime: null lastTransitionTime: "2020-12-22T09:01:35Z" message: 'containers with unready status: [extract]' reason: ContainersNotReady status: "False" type: ContainersReady - lastProbeTime: null lastTransitionTime: "2020-12-22T09:01:35Z" status: "True" type: PodScheduled containerStatuses: - image: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:262b08f1fc93d6409bc612a11687821371e10fbf8301f3f94db2234d619ccc87 imageID: "" lastState: {} name: extract ready: false restartCount: 0 started: false state: waiting: reason: PodInitializing hostIP: 10.0.158.39 initContainerStatuses: - containerID: cri-o://6f019c2053a89674fc932149542316ce7dd6a2bc7a2d77d863a185453ed2994f image: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:82a6e2add54339cdc10687b4c5d827ec8106cc56422e7039c131e7727755b3c9 imageID: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:82a6e2add54339cdc10687b4c5d827ec8106cc56422e7039c131e7727755b3c9 lastState: {} name: util ready: true restartCount: 0 state: terminated: containerID: cri-o://6f019c2053a89674fc932149542316ce7dd6a2bc7a2d77d863a185453ed2994f exitCode: 0 finishedAt: "2020-12-22T09:01:40Z" reason: Completed startedAt: "2020-12-22T09:01:39Z" - image: quay.io/jiazha/upstream-opm-builder:etcdbundle imageID: "" lastState: {} name: pull ready: false restartCount: 0 state: waiting: message: Back-off pulling image "quay.io/jiazha/upstream-opm-builder:etcdbundle" reason: ImagePullBackOff phase: Pending podIP: 10.128.2.191 podIPs: - ip: 10.128.2.191 qosClass: BestEffort startTime: "2020-12-22T09:01:35Z"
Cluster version is 4.7.0-0.nightly-2021-01-18-000316 [root@preserve-olm-env data]# oc -n openshift-operator-lifecycle-manager exec catalog-operator-6d9d94fdb8-wk2vh -- olm --version OLM version: 0.17.0 git commit: cab348020d3dafccfb7eef5ef4e05f7fe402b544 1, Create a pull secret called "secret-cs" in "openshift-marketplace" namespace. [root@preserve-olm-env data]# oc project Using project "openshift-marketplace" on server "https://api.xxia18shared.qe.devcluster.openshift.com:6443". [root@preserve-olm-env data]# oc extract secret/pull-secret -n openshift-config --confirm .dockerconfigjson [root@preserve-olm-env data]# oc create secret generic secret-cs --from-file=.dockerconfigjson=/root/.docker/config.json --type=kubernetes.io/dockerconfigjson secret/secret-cs created 2, Create a CatalogSource CR to consume a private index image that provides etcdoperator. [root@preserve-olm-env data]# oc create -f cs-private.yaml catalogsource.operators.coreos.com/cs-private created [root@preserve-olm-env data]# cat cs-private.yaml apiVersion: operators.coreos.com/v1alpha1 kind: CatalogSource metadata: name: cs-private namespace: openshift-marketplace spec: displayName: OLM Test publisher: Jian sourceType: grpc secrets: - "secret-cs" image: quay.io/jiazha/upstream-opm-builder:etcdindex updateStrategy: registryPoll: interval: 10m [root@preserve-olm-env data]# oc get packagemanifest|grep etcd etcd Community Operators 3h26m etcd OLM Test 3m5s 3, Subscribe to this etcdoperator. [root@preserve-olm-env data]# cat sub-etcd-cluster.yaml apiVersion: operators.coreos.com/v1alpha1 kind: Subscription metadata: name: etcd-private namespace: openshift-operators spec: channel: clusterwide-alpha installPlanApproval: Automatic name: etcd source: cs-private sourceNamespace: openshift-marketplace startingCSV: etcdoperator.v0.9.4-clusterwide 4, The job that unpacks the bundle images works well. [root@preserve-olm-env data]# oc get job NAME COMPLETIONS DURATION AGE ... d3383223ae0bc9c573b86c2d918fe9f6f9988f5771d0a503da2943d2fa37991 1/1 8s 34m [root@preserve-olm-env data]# oc get pods NAME READY STATUS RESTARTS AGE ... d3383223ae0bc9c573b86c2d918fe9f6f9988f5771d0a503da2943d2fajxtbz 0/1 Completed 0 34m ... [root@preserve-olm-env data]# oc get job d3383223ae0bc9c573b86c2d918fe9f6f9988f5771d0a503da2943d2fa37991 -o yaml|grep imagePullSecrets -A1 f:imagePullSecrets: .: {} -- imagePullSecrets: - name: secret-cs 5, Check if this operator can be installed. [root@preserve-olm-env data]# oc get sub -n openshift-operators NAME PACKAGE SOURCE CHANNEL etcd-private etcd cs-private clusterwide-alpha [root@preserve-olm-env data]# oc get ip -n openshift-operators NAME CSV APPROVAL APPROVED install-lm8d4 etcdoperator.v0.9.4-clusterwide Automatic true [root@preserve-olm-env data]# oc get csv -n openshift-operators NAME DISPLAY VERSION REPLACES PHASE etcdoperator.v0.9.4-clusterwide etcd 0.9.4-clusterwide Installing ... [root@preserve-olm-env data]# oc get job d3383223ae0bc9c573b86c2d918fe9f6f9988f5771d0a503da2943d2fa37991 -o yaml|grep image: f:image: {} f:image: {} f:image: {} image: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:56493169715122404a9007e9c087ba36a5851f0cbccebd82c2c0a162ef80fdef image: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:452e9c8f942f706eed65c0b726b863a2dc796dc029f9736a1596a2fc23e8b29f image: quay.io/jiazha/upstream-opm-builder:etcdbundle [root@preserve-olm-env data]# oc get csv -n openshift-operators NAME DISPLAY VERSION REPLACES PHASE etcdoperator.v0.9.4-clusterwide etcd 0.9.4-clusterwide Failed [root@preserve-olm-env data]# oc get pods -n openshift-operators NAME READY STATUS RESTARTS AGE etcd-operator-65f8576977-n5txr 0/3 ImagePullBackOff 0 19m [root@preserve-olm-env data]# oc get deployment -n openshift-operators NAME READY UP-TO-DATE AVAILABLE AGE etcd-operator 0/1 1 0 20m [root@preserve-olm-env data]# oc get deployment -n openshift-operators etcd-operator -o yaml|grep imagePullSecrets ... This private operator failed to install since we didn't inject the pull-secret to its deployment in the configmap. [root@preserve-olm-env data]# oc get cm d3383223ae0bc9c573b86c2d918fe9f6f9988f5771d0a503da2943d2fa37991 -o yaml|grep imagePullSecrets [root@preserve-olm-env data]# I tried to add the above pull secret(cs-secret) auth to the pull-secret of "openshift-config". But, it still failed to pull this private image. [root@preserve-olm-env data]# cat .dockerconfigjson | jq --compact-output '.auths["quay.io/jiazha"] |= . + {"auth":"xxx"}' > new_dockerconfigjson [root@preserve-olm-env data]# oc set data secret/pull-secret -n openshift-config --from-file=.dockerconfigjson=new_dockerconfigjson secret/pull-secret data updated [root@preserve-olm-env data]# oc get pods NAME READY STATUS RESTARTS AGE etcd-operator-65f8576977-2xlsb 0/3 ImagePullBackOff 0 9s Normal BackOff <invalid> (x2 over <invalid>) kubelet Back-off pulling image "quay.io/jiazha/upstream-opm-builder@sha256:2cce9f8e95c9b4ce19b9ffbb95298b9ba3c4960dc962030d8bce655f3811adb0" Warning Failed <invalid> (x2 over <invalid>) kubelet Error: ImagePullBackOff I change the Status to ASSIGNED since I think we should have an official solution to install the operator using the private image.
Hi Anik, Thanks for your information! I understand, the problem here is that I add this pull-secret to the pull-secret of "openshift-config", but it still doesn't work. Anyway, I think this problem is not related to this bug, verify it first.
Updates: adding this pull-secret to the pull-secret of "openshift-config", it works after a few minutes. [root@preserve-olm-env data]# oc project Using project "openshift-operators" on server "https://api.hongli-aw47.qe.devcluster.openshift.com:6443". [root@preserve-olm-env data]# oc get csv NAME DISPLAY VERSION REPLACES PHASE etcdoperator.v0.9.4-clusterwide etcd 0.9.4-clusterwide Installing [root@preserve-olm-env data]# oc get pods NAME READY STATUS RESTARTS AGE etcd-operator-65f8576977-pzbf7 0/3 ImagePullBackOff 0 22s [root@preserve-olm-env data]# oc get pods NAME READY STATUS RESTARTS AGE etcd-operator-65f8576977-8mh4h 3/3 Running 0 154m That's a workaround if users want to install the operator that using the private image. [root@preserve-olm-env data]# cat .dockerconfigjson | jq --compact-output '.auths["quay.io/jiazha"] |= . + {"auth":"xxx"}' > new_dockerconfigjson [root@preserve-olm-env data]# oc set data secret/pull-secret -n openshift-config --from-file=.dockerconfigjson=new_dockerconfigjson secret/pull-secret data updated
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Moderate: OpenShift Container Platform 4.7.0 security, bug fix, and enhancement update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2020:5633