Bug 2156538
| Summary: | failed to pull image ose-csi-external-snapshotter and ose-csi-external-resizer | ||||||
|---|---|---|---|---|---|---|---|
| Product: | [Red Hat Storage] Red Hat OpenShift Data Foundation | Reporter: | Bin Hu <bihu> | ||||
| Component: | build | Assignee: | Boris Ranto <branto> | ||||
| Status: | CLOSED NOTABUG | QA Contact: | Petr Balogh <pbalogh> | ||||
| Severity: | urgent | Docs Contact: | |||||
| Priority: | urgent | ||||||
| Version: | 4.10 | CC: | asadawar, assingh, bkunal, branto, dkhandel, ebattat, hnallurv, jaliang, kelwhite, madam, mdeore, muagarwa, ocs-bugs, odf-bz-bot, rar, tmuthami | ||||
| Target Milestone: | --- | Keywords: | Reopened | ||||
| Target Release: | --- | Flags: | assingh:
needinfo?
(dkhandel) |
||||
| Hardware: | x86_64 | ||||||
| OS: | Linux | ||||||
| Whiteboard: | |||||||
| Fixed In Version: | Doc Type: | If docs needed, set a value | |||||
| Doc Text: | Story Points: | --- | |||||
| Clone Of: | Environment: | ||||||
| Last Closed: | 2023-07-12 13:40:36 UTC | Type: | Bug | ||||
| Regression: | --- | Mount Type: | --- | ||||
| Documentation: | --- | CRM: | |||||
| Verified Versions: | Category: | --- | |||||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |||||
| Cloudforms Team: | --- | Target Upstream Version: | |||||
| Embargoed: | |||||||
| Attachments: |
|
||||||
This issue also happens in my OCP 4.10 and 4.9 created by RHPDS. The same issue is happened to me on OCP4.12.0-rc6/ODF4.11.4: Normal Scheduled 9m default-scheduler Successfully assigned openshift-storage/csi-cephfsplugin-provisioner-79c6c9b997-dfqdh to master-2 Normal AddedInterface 9m multus Add eth0 [10.130.0.48/23] from ovn-kubernetes Normal Pulling 9m kubelet Pulling image "registry.redhat.io/openshift4/ose-csi-external-attacher@sha256:96c3a693fc6427102c817f8a7e6934f562fdb177ce847c78126cee95b0c17ce2" Normal Pulled 8m54s kubelet Successfully pulled image "registry.redhat.io/openshift4/ose-csi-external-attacher@sha256:96c3a693fc6427102c817f8a7e6934f562fdb177ce847c78126cee95b0c17ce2" in 5.493074712s Normal Created 8m54s kubelet Created container csi-attacher Normal Started 8m54s kubelet Started container csi-attacher Normal Pulled 8m53s kubelet Container image "registry.redhat.io/openshift4/ose-csi-external-provisioner@sha256:c9f239db31e224f6e35e544d6941c0ffc1e80d4846a8036ccbaee39ee42f1846" already present on machine Normal Pulling 8m52s kubelet Pulling image "registry.redhat.io/odf4/cephcsi-rhel8@sha256:7fab87df9e7bfddcf4b49855fb69a6c8ddbdfe7f6fe92c5418c2fd7cc94d29f2" Normal Created 8m52s kubelet Created container csi-provisioner Normal Started 8m52s kubelet Started container csi-provisioner Normal Created 8m34s kubelet Created container csi-cephfsplugin Normal Pulled 8m34s kubelet Successfully pulled image "registry.redhat.io/odf4/cephcsi-rhel8@sha256:7fab87df9e7bfddcf4b49855fb69a6c8ddbdfe7f6fe92c5418c2fd7cc94d29f2" in 18.462252795s Normal Started 8m34s kubelet Started container csi-cephfsplugin Warning Failed 8m33s (x2 over 8m53s) kubelet Failed to pull image "registry.redhat.io/openshift4/ose-csi-external-snapshotter@sha256:19208354f2830edd7a91e52a9cf02686eab41442e469a06ffd9df19e023fdf61": rpc error: code = Unknown desc = reading manifest sha256:19208354f2830edd7a91e52a9cf02686eab41442e469a06ffd9df19e023fdf61 in registry.redhat.io/openshift4/ose-csi-external-snapshotter: manifest unknown: manifest unknown Normal Pulling 8m33s (x2 over 8m53s) kubelet Pulling image "registry.redhat.io/openshift4/ose-csi-external-resizer@sha256:e5081f598b82998beecc276019fbed4efadea89fe34001b258f50c5d7aea3b18" Normal Pulling 8m33s (x2 over 8m54s) kubelet Pulling image "registry.redhat.io/openshift4/ose-csi-external-snapshotter@sha256:19208354f2830edd7a91e52a9cf02686eab41442e469a06ffd9df19e023fdf61" Warning Failed 8m33s (x2 over 8m53s) kubelet Error: ErrImagePull Warning Failed 8m32s (x2 over 8m53s) kubelet Error: ErrImagePull Warning Failed 8m32s (x2 over 8m53s) kubelet Failed to pull image "registry.redhat.io/openshift4/ose-csi-external-resizer@sha256:e5081f598b82998beecc276019fbed4efadea89fe34001b258f50c5d7aea3b18": rpc error: code = Unknown desc = reading manifest sha256:e5081f598b82998beecc276019fbed4efadea89fe34001b258f50c5d7aea3b18 in registry.redhat.io/openshift4/ose-csi-external-resizer: manifest unknown: manifest unknown Warning Failed 8m32s kubelet Error: ImagePullBackOff Normal BackOff 3m58s (x19 over 8m32s) kubelet Back-off pulling image "registry.redhat.io/openshift4/ose-csi-external-snapshotter@sha256:19208354f2830edd7a91e52a9cf02686eab41442e469a06ffd9df19e023fdf61" Hello, Reopening the bug as we hit the same issue while working with one of the customer escalations. They are using ocs-operator.v4.10.9 and the below 2 images were throwing manifest unknown error. ~~~ # podman pull registry.redhat.io/openshift4/ose-csi-external-resizer@sha256:e5081f598b82998beecc276019fbed4efadea89fe34001b258f50c5d7aea3b18 Trying to pull registry.redhat.io/openshift4/ose-csi-external-resizer@sha256:e5081f598b82998beecc276019fbed4efadea89fe34001b258f50c5d7aea3b18... manifest unknown: manifest unknown Error: error pulling image "registry.redhat.io/openshift4/ose-csi-external-resizer@sha256:e5081f598b82998beecc276019fbed4efadea89fe34001b258f50c5d7aea3b18": unable to pull registry.redhat.io/openshift4/ose-csi-external-resizer@sha256:e5081f598b82998beecc276019fbed4efadea89fe34001b258f50c5d7aea3b18: unable to pull image: Error initializing source docker://registry.redhat.io/openshift4/ose-csi-external-resizer@sha256:e5081f598b82998beecc276019fbed4efadea89fe34001b258f50c5d7aea3b18: Error reading manifest sha256:e5081f598b82998beecc276019fbed4efadea89fe34001b258f50c5d7aea3b18 in registry.redhat.io/openshift4/ose-csi-external-resizer: manifest unknown: manifest unknown # podman pull registry.redhat.io/openshift4/ose-csi-external-provisioner@sha256:c9f239db31e224f6e35e544d6941c0ffc1e80d4846a8036ccbaee39ee42f1846 Trying to pull registry.redhat.io/openshift4/ose-csi-external-provisioner@sha256:c9f239db31e224f6e35e544d6941c0ffc1e80d4846a8036ccbaee39ee42f1846... Getting image source signatures Copying blob c1f322f72a56 skipped: already exists Copying blob f2c1a2360be4 skipped: already exists Copying blob 7cde10acfa2e skipped: already exists Copying blob 1df162fae087 skipped: already exists Copying config e9ceafcb43 done Writing manifest to image destination Storing signatures e9ceafcb43f661c54ee1151fe79542fb609a08afd713648a63562b834b221d76 # podman pull registry.redhat.io/openshift4/ose-csi-external-snapshotter@sha256:19208354f2830edd7a91e52a9cf02686eab41442e469a06ffd9df19e023fdf61 Trying to pull registry.redhat.io/openshift4/ose-csi-external-snapshotter@sha256:19208354f2830edd7a91e52a9cf02686eab41442e469a06ffd9df19e023fdf61... manifest unknown: manifest unknown Error: error pulling image "registry.redhat.io/openshift4/ose-csi-external-snapshotter@sha256:19208354f2830edd7a91e52a9cf02686eab41442e469a06ffd9df19e023fdf61": unable to pull registry.redhat.io/openshift4/ose-csi-external-snapshotter@sha256:19208354f2830edd7a91e52a9cf02686eab41442e469a06ffd9df19e023fdf61: unable to pull image: Error initializing source docker://registry.redhat.io/openshift4/ose-csi-external-snapshotter@sha256:19208354f2830edd7a91e52a9cf02686eab41442e469a06ffd9df19e023fdf61: Error reading manifest sha256:19208354f2830edd7a91e52a9cf02686eab41442e469a06ffd9df19e023fdf61 in registry.redhat.io/openshift4/ose-csi-external-snapshotter: manifest unknown: manifest unknown ~~~ As a workaround, we had to edit the operator CSV and manually replace the images with the latest (working) version. This is not a product bug but a migration bug/issue. It should be resolved now AFAIK. Basically, the images were being migrated to quay and there were issues during that process. The processs should be complete, now. I'm closing this, feel free to re-open if you still hit the issues (an EXD JIRA ticket would be better for that though). |
Created attachment 1934568 [details] image not exist on registry Description of problem (please be detailed as possible and provide log snippests): ODF 4.10.9 failed to download images ose-csi-external-snapshotter, ose-csi-external-resizer Version of all relevant components (if applicable): ODF 4.10.9 Does this issue impact your ability to continue to work with the product (please explain in detail what is the user impact)? I am blocked because ODF external mode cannot deploy successful Is there any workaround available to the best of your knowledge? No Rate from 1 - 5 the complexity of the scenario you performed that caused this bug (1 - very simple, 5 - very complex)? 1 Can this issue reproducible? Yes Can this issue reproduce from the UI? Yes If this is a regression, please provide more details to justify this: Yes Steps to Reproduce: 1. create a storageSystem, external mode, upload JSON file to deploy ODF external mode, that's all 2. 3. Actual results: images can be downloaded successful Expected results: Failed to pull image "registry.redhat.io/openshift4/ose-csi-external-snapshotter@sha256:19208354f2830edd7a91e52a9cf02686eab41442e469a06ffd9df19e023fdf61": rpc error: code = Unknown desc = reading manifest sha256:19208354f2830edd7a91e52a9cf02686eab41442e469a06ffd9df19e023fdf61 in registry.redhat.io/openshift4/ose-csi-external-snapshotter: manifest unknown: manifest unknown Failed to pull image "registry.redhat.io/openshift4/ose-csi-external-resizer@sha256:e5081f598b82998beecc276019fbed4efadea89fe34001b258f50c5d7aea3b18": rpc error: code = Unknown desc = reading manifest sha256:e5081f598b82998beecc276019fbed4efadea89fe34001b258f50c5d7aea3b18 in registry.redhat.io/openshift4/ose-csi-external-resizer: manifest unknown: manifest unknown The images does not exist: [root@bastion-rhel8 ~]# podman pull registry.redhat.io/openshift4/ose-csi-external-snapshotter@sha254f2830edd7a91e52a9cf02686eab41442e469a06ffd9df19e023fdf61 Trying to pull registry.redhat.io/openshift4/ose-csi-external-snapshotter@sha256:19208354f2830edd7a02686eab41442e469a06ffd9df19e023fdf61... Error: initializing source docker://registry.redhat.io/openshift4/ose-csi-external-snapshotter@sha254f2830edd7a91e52a9cf02686eab41442e469a06ffd9df19e023fdf61: reading manifest sha256:19208354f2830ed9cf02686eab41442e469a06ffd9df19e023fdf61 in registry.redhat.io/openshift4/ose-csi-external-snapshotfest unknown: manifest unknown [root@bastion-rhel8 ~]# [root@bastion-rhel8 ~]# [root@bastion-rhel8 ~]# [root@bastion-rhel8 ~]# podman pull registry.redhat.io/openshift4/ose-csi-external-resizer@sha256:e82998beecc276019fbed4efadea89fe34001b258f50c5d7aea3b18 Trying to pull registry.redhat.io/openshift4/ose-csi-external-resizer@sha256:e5081f598b82998beecc27efadea89fe34001b258f50c5d7aea3b18... Error: initializing source docker://registry.redhat.io/openshift4/ose-csi-external-resizer@sha256:e82998beecc276019fbed4efadea89fe34001b258f50c5d7aea3b18: reading manifest sha256:e5081f598b82998beeced4efadea89fe34001b258f50c5d7aea3b18 in registry.redhat.io/openshift4/ose-csi-external-resizer: mannown: manifest unknown # the image does not exist [root@bastion-rhel8 ~]# skopeo inspect docker://registry.redhat.io/openshift4/ose-csi-external-snapshotter@sha256:19208354f2830edd7a91e52a9cf02686eab41442e469a06ffd9df19e023fdf61 FATA[0000] Error parsing image name "docker://registry.redhat.io/openshift4/ose-csi-external-snapshotter@sha256:19208354f2830edd7a91e52a9cf02686eab41442e469a06ffd9df19e023fdf61": reading manifest sha256:19208354f2830edd7a91e52a9cf02686eab41442e469a06ffd9df19e023fdf61 in registry.redhat.io/openshift4/ose-csi-external-snapshotter: manifest unknown: manifest unknown [root@bastion-rhel8 ~]# skopeo inspect docker://registry.redhat.io/openshift4/ose-csi-external-resizer@sha256:e5081f598b82998beecc276019fbed4efadea89fe34001b258f50c5d7aea3b18 FATA[0000] Error parsing image name "docker://registry.redhat.io/openshift4/ose-csi-external-resizer@sha256:e5081f598b82998beecc276019fbed4efadea89fe34001b258f50c5d7aea3b18": reading manifest sha256:e5081f598b82998beecc276019fbed4efadea89fe34001b258f50c5d7aea3b18 in registry.redhat.io/openshift4/ose-csi-external-resizer: manifest unknown: manifest unknown # The CSV use these 2 images [root@bastion-rhel8 ~]# oc get csv ocs-operator.v4.10.9 -n openshift-storage -oyaml |grep ROOK_CSI_RESIZER_IMAGE -A5 - name: ROOK_CSI_RESIZER_IMAGE value: registry.redhat.io/openshift4/ose-csi-external-resizer@sha256:e5081f598b82998beecc276019fbed4efadea89fe34001b258f50c5d7aea3b18 - name: ROOK_CSI_PROVISIONER_IMAGE value: registry.redhat.io/openshift4/ose-csi-external-provisioner@sha256:c9f239db31e224f6e35e544d6941c0ffc1e80d4846a8036ccbaee39ee42f1846 - name: ROOK_CSI_SNAPSHOTTER_IMAGE value: registry.redhat.io/openshift4/ose-csi-external-snapshotter@sha256:19208354f2830edd7a91e52a9cf02686eab41442e469a06ffd9df19e023fdf61 Additional info: uploading ODF must-gather data