Description of problem: 4.3 channel is publishing the image url as registry.redhat.io/openshift4/ose-elasticsearch-operator@sha256:5e50822dc262ddcec0ce65fd1dee14f61398047b95b5dfc438a3043b54d2f93d I would expect registry.redhat.io/openshift4/ose-elasticsearch-operator@sha256:5e50822dc262ddcec0ce65fd1dee14f61398047b95b5dfc438a3043b54d2f93d to be the sh256 Digest of the ose-elasticsearch-operator image, but it is not. sha256:5e50822dc262ddcec0ce65fd1dee14f61398047b95b5dfc438a3043b54d2f93d is the RepoDigest of the image, not the Digest of the image. Version-Release number of selected component (if applicable): 4.3 How reproducible: Manually pulling the image referenced in the 4.3 update channel registry.redhat.io/openshift4/ose-elasticsearch-operator@sha256:5e50822dc262ddcec0ce65fd1dee14f61398047b95b5dfc438a3043b54d2f93d $ podman images | grep ose-elasticsearch # just to show that i don't have any existing images $ podman pull registry.redhat.io/openshift4/ose-elasticsearch-operator@sha256:5e50822dc262ddcec0ce65fd1dee14f61398047b95b5dfc438a3043b54d2f93d Trying to pull registry.redhat.io/openshift4/ose-elasticsearch-operator@sha256:5e50822dc262ddcec0ce65fd1dee14f61398047b95b5dfc438a3043b54d2f93d... Getting image source signatures Copying blob 4fbc3bafa3d4 skipped: already exists Copying blob 8d4203411daf skipped: already exists Copying blob e232ae6317c2 skipped: already exists Copying blob 34971b2d1eb9 skipped: already exists Copying blob c53747ad4b97 done Copying config b045933af4 done Writing manifest to image destination Storing signatures b045933af4f7a0883f1a0366c2377a03b94a77bba24e7c15b4b4171dbfdb5638 $ podman inspect b045933af4f7a0883f1a0366c2377a03b94a77bba24e7c15b4b4171dbfdb5638 | jq '.[0]|.Digest,.RepoDigests' "sha256:4b790902039f3ea801643011f807d7fdc20c1bf014c923351f5eebba46b3d327" [ "registry.redhat.io/openshift4/ose-elasticsearch-operator@sha256:4b790902039f3ea801643011f807d7fdc20c1bf014c923351f5eebba46b3d327", "registry.redhat.io/openshift4/ose-elasticsearch-operator@sha256:5e50822dc262ddcec0ce65fd1dee14f61398047b95b5dfc438a3043b54d2f93d" ] $ podman rmi b045933af4f7a0883f1a0366c2377a03b94a77bba24e7c15b4b4171dbfdb5638 Untagged: registry.redhat.io/openshift4/ose-elasticsearch-operator@sha256:5e50822dc262ddcec0ce65fd1dee14f61398047b95b5dfc438a3043b54d2f93d Deleted: b045933af4f7a0883f1a0366c2377a03b94a77bba24e7c15b4b4171dbfdb5638 $ When I use jq '.[0]|.Digest,.RepoDigests' to extract the Digest and RepoDigest fields for the image you can see that sha256:5e50822dc262ddcec0ce65fd1dee14f61398047b95b5dfc438a3043b54d2f93d is the RepoDigest and not the Digest for the image Actual results: When I use jq '.[0]|.Digest,.RepoDigests' to extract the Digest and RepoDigest fields for the image you can see that sha256:5e50822dc262ddcec0ce65fd1dee14f61398047b95b5dfc438a3043b54d2f93d is the RepoDigest and not the Digest for the image Expected results: published in the catalogue as registry.redhat.io/openshift4/ose-elasticsearch-operator@sha256:4b790902039f3ea801643011f807d7fdc20c1bf014c923351f5eebba46b3d327 Additional info:
Operator Registry start to RepoDigest as image url from 4.3 to support multiple-artchuture. Could you check if your podman support multiple-artchuture. I can pull the image_url attached above using Docker.
I am not sure why you are asking me if, I can pull the image using podman? If you look at the example quoted, i can clearly pull the image using podman. However the issue is. once I mirror this image to a docker registry, i am not able to pull this image from the mirror'd docker registry Going to try explaining the issue one more time. In the 4.2 stable channel the image published for ose-elasticsearch-operator is registry.redhat.io/openshift4/ose-elasticsearch-operator@sha256:d94eebefb44bed0bcaab45fd3e59554c7d44caf37a4c57e9cd340a00dd2d490d I attempt to mirror this image using the `oc image mirror` command to my local docker registry. and then pull the image using podman. Everything works as expected. (notsure if the below formatting works or not) ``` export MIRROR_REGISTRY=helper.ocp4.example.com:5000 export NAMESPACE=olmtest-take2 [sushil@rhltop ~]$ oc image mirror registry.redhat.io/openshift4/ose-elasticsearch-operator@sha256:d94eebefb44bed0bcaab45fd3e59554c7d44caf37a4c57e9cd340a00dd2d490d ${MIRROR_REGISTRY}/${NAMESPACE}/ose-elasticsearch-operator helper.ocp4.example.com:5000/ olmtest-take2/ose-elasticsearch-operator blobs: registry.redhat.io/openshift4/ose-elasticsearch-operator sha256:4fbc3bafa3d4400bb97a733c1fe12f2f99bf38b9d5b913d5034f29798739654d 1.585KiB registry.redhat.io/openshift4/ose-elasticsearch-operator sha256:3d5c8986188005c86ab63feaa2c62a6dadccf0709f8e8596e9320d89d35748bd 5.176KiB registry.redhat.io/openshift4/ose-elasticsearch-operator sha256:4a21444dc61515c74392d648347471d1ebc8cd8ee35bb734dcfc9978bec41837 3.336MiB registry.redhat.io/openshift4/ose-elasticsearch-operator sha256:eb8b56a4e1384d8bf72a65f3953d4d3e9c2980191ec75ed41751f1b560e247b8 7.844MiB registry.redhat.io/openshift4/ose-elasticsearch-operator sha256:92b84f826d5dd4245465279ad2178fc1a7010209a721ccc753ab6fae91d55a3c 16.62MiB registry.redhat.io/openshift4/ose-elasticsearch-operator sha256:34971b2d1eb98e410c2802e6bb3a7f499f9c5bff1b870ed329089ecdb21cf856 72.71MiB manifests: sha256:d94eebefb44bed0bcaab45fd3e59554c7d44caf37a4c57e9cd340a00dd2d490d stats: shared=0 unique=6 size=100.5MiB ratio=1.00 phase 0: helper.ocp4.example.com:5000 olmtest-take2/ose-elasticsearch-operator blobs=6 mounts=0 manifests=1 shared=0 info: Planning completed in 7.33s uploading: helper.ocp4.example.com:5000/olmtest-take2/ose-elasticsearch-operator sha256:4a21444dc61515c74392d648347471d1ebc8cd8ee35bb734dcfc9978bec41837 3.336MiB uploading: helper.ocp4.example.com:5000/olmtest-take2/ose-elasticsearch-operator sha256:92b84f826d5dd4245465279ad2178fc1a7010209a721ccc753ab6fae91d55a3c 16.62MiB uploading: helper.ocp4.example.com:5000/olmtest-take2/ose-elasticsearch-operator sha256:34971b2d1eb98e410c2802e6bb3a7f499f9c5bff1b870ed329089ecdb21cf856 72.71MiB uploading: helper.ocp4.example.com:5000/olmtest-take2/ose-elasticsearch-operator sha256:eb8b56a4e1384d8bf72a65f3953d4d3e9c2980191ec75ed41751f1b560e247b8 7.844MiB sha256:d94eebefb44bed0bcaab45fd3e59554c7d44caf37a4c57e9cd340a00dd2d490d helper.ocp4.example.com:5000/olmtest-take2/ose-elasticsearch-operator info: Mirroring completed in 7.51s (14.02MB/s) [sushil@rhltop ~]$ [sushil@rhltop ~]$ [sushil@rhltop ~]$ [sushil@rhltop ~]$ [sushil@rhltop ~]$ podman pull ${MIRROR_REGISTRY}/${NAMESPACE}/ose-elasticsearch-operator@sha256:d94eebefb44bed0bcaab45fd3e59554c7d44caf37a4c57e9cd340a00dd2d490d Trying to pull helper.ocp4.example.com:5000/olmtest-take2/ose-elasticsearch-operator@sha256:d94eebefb44bed0bcaab45fd3e59554c7d44caf37a4c57e9cd340a00dd2d490d... Getting image source signatures Copying blob 34971b2d1eb9 skipped: already exists Copying blob 4fbc3bafa3d4 skipped: already exists Copying blob 92b84f826d5d done Copying blob 4a21444dc615 done Copying blob eb8b56a4e138 done Copying config 3d5c898618 done Writing manifest to image destination Storing signatures 3d5c8986188005c86ab63feaa2c62a6dadccf0709f8e8596e9320d89d35748bd [sushil@rhltop ~]$ [sushil@rhltop ~]$ ``` Now trying to do the same for the 4.3 stable channel the image published for ose-elasticsearch-operator registry.redhat.io/openshift4/ose-elasticsearch-operator@sha256:5e50822dc262ddcec0ce65fd1dee14f61398047b95b5dfc438a3043b54d2f93d ``` export MIRROR_REGISTRY=helper.ocp4.example.com:5000 export NAMESPACE=olmtest-take3 [sushil@rhltop ~]$ oc image mirror registry.redhat.io/openshift4/ose-elasticsearch-operator@sha256:5e50822dc262ddcec0ce65fd1dee14f61398047b95b5dfc438a3043b54d2f93d ${MIRROR_REGISTRY}/${NAMESPACE}/ose-elasticsearch-operator helper.ocp4.example.com:5000/ olmtest-take3/ose-elasticsearch-operator blobs: registry.redhat.io/openshift4/ose-elasticsearch-operator sha256:4fbc3bafa3d4400bb97a733c1fe12f2f99bf38b9d5b913d5034f29798739654d 1.585KiB registry.redhat.io/openshift4/ose-elasticsearch-operator sha256:b045933af4f7a0883f1a0366c2377a03b94a77bba24e7c15b4b4171dbfdb5638 5.171KiB registry.redhat.io/openshift4/ose-elasticsearch-operator sha256:8d4203411daf0c5f64ba4820c5cfca11c859871429f7ca8b02eebf810962ea7d 3.336MiB registry.redhat.io/openshift4/ose-elasticsearch-operator sha256:e232ae6317c2547a0c0941579360e4600a08bdd3398fc22ad2160e187256e94f 7.842MiB registry.redhat.io/openshift4/ose-elasticsearch-operator sha256:c53747ad4b973dea6b6c85e215eadb4a7a1a16d530c87e9db3f31a252edc8f99 18.51MiB registry.redhat.io/openshift4/ose-elasticsearch-operator sha256:34971b2d1eb98e410c2802e6bb3a7f499f9c5bff1b870ed329089ecdb21cf856 72.71MiB manifests: sha256:4b790902039f3ea801643011f807d7fdc20c1bf014c923351f5eebba46b3d327 stats: shared=0 unique=6 size=102.4MiB ratio=1.00 phase 0: helper.ocp4.example.com:5000 olmtest-take3/ose-elasticsearch-operator blobs=6 mounts=0 manifests=1 shared=0 info: Planning completed in 7.29s uploading: helper.ocp4.example.com:5000/olmtest-take3/ose-elasticsearch-operator sha256:c53747ad4b973dea6b6c85e215eadb4a7a1a16d530c87e9db3f31a252edc8f99 18.51MiB uploading: helper.ocp4.example.com:5000/olmtest-take3/ose-elasticsearch-operator sha256:8d4203411daf0c5f64ba4820c5cfca11c859871429f7ca8b02eebf810962ea7d 3.336MiB uploading: helper.ocp4.example.com:5000/olmtest-take3/ose-elasticsearch-operator sha256:e232ae6317c2547a0c0941579360e4600a08bdd3398fc22ad2160e187256e94f 7.842MiB uploading: helper.ocp4.example.com:5000/olmtest-take3/ose-elasticsearch-operator sha256:34971b2d1eb98e410c2802e6bb3a7f499f9c5bff1b870ed329089ecdb21cf856 72.71MiB sha256:4b790902039f3ea801643011f807d7fdc20c1bf014c923351f5eebba46b3d327 helper.ocp4.example.com:5000/olmtest-take3/ose-elasticsearch-operator info: Mirroring completed in 7s (15.32MB/s) [sushil@rhltop ~]$ [sushil@rhltop ~]$ [sushil@rhltop ~]$ podman pull ${MIRROR_REGISTRY}/${NAMESPACE}/ose-elasticsearch-operator@sha256:5e50822dc262ddcec0ce65fd1dee14f61398047b95b5dfc438a3043b54d2f93d Trying to pull helper.ocp4.example.com:5000/olmtest-take3/ose-elasticsearch-operator@sha256:5e50822dc262ddcec0ce65fd1dee14f61398047b95b5dfc438a3043b54d2f93d... manifest unknown: manifest unknown Error: error pulling image "helper.ocp4.example.com:5000/olmtest-take3/ose-elasticsearch-operator@sha256:5e50822dc262ddcec0ce65fd1dee14f61398047b95b5dfc438a3043b54d2f93d": unable to pull helper.ocp4.example.com:5000/olmtest-take3/ose-elasticsearch-operator@sha256:5e50822dc262ddcec0ce65fd1dee14f61398047b95b5dfc438a3043b54d2f93d: unable to pull image: Error initializing source docker://helper.ocp4.example.com:5000/olmtest-take3/ose-elasticsearch-operator@sha256:5e50822dc262ddcec0ce65fd1dee14f61398047b95b5dfc438a3043b54d2f93d: Error reading manifest sha256:5e50822dc262ddcec0ce65fd1dee14f61398047b95b5dfc438a3043b54d2f93d in helper.ocp4.example.com:5000/olmtest-take3/ose-elasticsearch-operator: manifest unknown: manifest unknown [sushil@rhltop ~]$ ``` As a result I am not able to seutp a mirror'd repo and use it for setting up disconnected Operators. Because once mirrored the image reference for ose-elasticsearch-operator doesn't work. First time updating a BZ, so not sure sure if the above formating works. Dumping the above output into a gist as well, hopeing it will make it easier to read. https://gist.github.com/sushilsuresh/e1b3ab9ba9be6dfc232611eb8145f41f
From 4.3, all the 3rd level operators are changed to use manifest lists in the CSV files. Looks like your MIRROR_REGISTRY doesn't support manifest lists. And I guess the manifest of the ose-elasticsearch-operator in your MIRROR_REGISTRY is `4b790902039f3ea801643011f807d7fdc20c1bf014c923351f5eebba46b3d327`, not `5e50822dc262ddcec0ce65fd1dee14f61398047b95b5dfc438a3043b54d2f93d`. Maybe you could try to mirror the images to a registry that supports v2 schema2.
The version of my docker image registry is docker.io/library/registry:2.6.2 If I were to check https://docs.docker.com/registry/compatibility/ As far as I understand, Docker Registry v2.3 and above supports and defaults to schema2 format. What am I missing? Thank you. -- Regards Sushil Suresh.
I have tried replicating the same process using the latest docker image. Below is the command I use to start the docker image registry podman run -td --name {{ registry_container_name }} -p 5000:5000 -v {{ registry_base_path }}/data:/var/lib/registry:z -v {{ registry_base_path }}/certs:/certs:z -v {{ registry_base_path }}/auth:/auth:z -e REGISTRY_AUTH=htpasswd -e REGISTRY_AUTH_HTPASSWD_REALM=Registry -e REGISTRY_AUTH_HTPASSWD_PATH=/auth/htpasswd -e REGISTRY_HTTP_TLS_CERTIFICATE=/certs/domain.crt -e REGISTRY_HTTP_TLS_KEY=/certs/domain.key docker.io/library/registry:latest I get the exact same error as before. My understanding is that the latest docker registry images remove support for v2 schema1 So this clearly is not an issue with the docker registry not supporting v2 schema2. Either the oc image mirror command is messing things up. OR The original publied image manifest is wrong. Happy to be proved wrong. I am fairly new at this. I just want to know what the root cause is and a being able to mirror the ose-elasticsearch-operator image in a disconnectd environment and have it working. in case you are wondering, my laptop is running Fedora 31 and below are my podman and oc command versions. [sushil@rhltop ocp4-upi-helpernode]$ podman version Version: 1.8.0 RemoteAPI Version: 1 Go Version: go1.13.6 OS/Arch: linux/amd64 [sushil@rhltop ocp4-upi-helpernode]$ oc version Client Version: 4.3.1 I would encourage you to setup a local docker image registry and try mirroring the below image and pulling the same from the mirror's registry. The suspect image in question is registry.redhat.io/openshift4/ose-elasticsearch-operator@sha256:5e50822dc262ddcec0ce65fd1dee14f61398047b95b5dfc438a3043b54d2f93d Thanks again. -- Regards Sushil Suresh.
i'm told you need to add "--filter-by-os=‘.*’" to your oc image mirror command to make this work, can you try that?
Nope, doesn't work either. Exact same error - manifest unknown: manifest unknown see Gist - https://gist.github.com/sushilsuresh/268ff30de35dd2825f1bf758b991c7e8 For your benifit, going to paste the console output below as welll ----------------------------------------------------------------------------------- [sushil@rhltop ~]$ [sushil@rhltop ~]$ export MIRROR_REGISTRY=helper.ocp4.example.com:5000 [sushil@rhltop ~]$ export NAMESPACE=olmtest-take1 [sushil@rhltop ~]$ [sushil@rhltop ~]$ [sushil@rhltop ~]$ oc image mirror --filter-by-os=.* registry.redhat.io/openshift4/ose-elasticsearch-operator@sha256:5e50822dc262ddcec0ce65fd1dee14f61398047b95b5dfc438a3043b54d2f93d ${MIRROR_REGISTRY}/${NAMESPACE}/ose-elasticsearch-operator helper.ocp4.example.com:5000/ olmtest-take1/ose-elasticsearch-operator blobs: registry.redhat.io/openshift4/ose-elasticsearch-operator sha256:4fbc3bafa3d4400bb97a733c1fe12f2f99bf38b9d5b913d5034f29798739654d 1.585KiB registry.redhat.io/openshift4/ose-elasticsearch-operator sha256:b045933af4f7a0883f1a0366c2377a03b94a77bba24e7c15b4b4171dbfdb5638 5.171KiB registry.redhat.io/openshift4/ose-elasticsearch-operator sha256:8d4203411daf0c5f64ba4820c5cfca11c859871429f7ca8b02eebf810962ea7d 3.336MiB registry.redhat.io/openshift4/ose-elasticsearch-operator sha256:e232ae6317c2547a0c0941579360e4600a08bdd3398fc22ad2160e187256e94f 7.842MiB registry.redhat.io/openshift4/ose-elasticsearch-operator sha256:c53747ad4b973dea6b6c85e215eadb4a7a1a16d530c87e9db3f31a252edc8f99 18.51MiB registry.redhat.io/openshift4/ose-elasticsearch-operator sha256:34971b2d1eb98e410c2802e6bb3a7f499f9c5bff1b870ed329089ecdb21cf856 72.71MiB manifests: sha256:4b790902039f3ea801643011f807d7fdc20c1bf014c923351f5eebba46b3d327 stats: shared=0 unique=6 size=102.4MiB ratio=1.00 phase 0: helper.ocp4.example.com:5000 olmtest-take1/ose-elasticsearch-operator blobs=6 mounts=0 manifests=1 shared=0 info: Planning completed in 3.32s uploading: helper.ocp4.example.com:5000/olmtest-take1/ose-elasticsearch-operator sha256:e232ae6317c2547a0c0941579360e4600a08bdd3398fc22ad2160e187256e94f 7.842MiB uploading: helper.ocp4.example.com:5000/olmtest-take1/ose-elasticsearch-operator sha256:34971b2d1eb98e410c2802e6bb3a7f499f9c5bff1b870ed329089ecdb21cf856 72.71MiB uploading: helper.ocp4.example.com:5000/olmtest-take1/ose-elasticsearch-operator sha256:8d4203411daf0c5f64ba4820c5cfca11c859871429f7ca8b02eebf810962ea7d 3.336MiB uploading: helper.ocp4.example.com:5000/olmtest-take1/ose-elasticsearch-operator sha256:c53747ad4b973dea6b6c85e215eadb4a7a1a16d530c87e9db3f31a252edc8f99 18.51MiB sha256:4b790902039f3ea801643011f807d7fdc20c1bf014c923351f5eebba46b3d327 helper.ocp4.example.com:5000/olmtest-take1/ose-elasticsearch-operator info: Mirroring completed in 21.07s (5.096MB/s) [sushil@rhltop ~]$ [sushil@rhltop ~]$ [sushil@rhltop ~]$ [sushil@rhltop ~]$ podman pull ${MIRROR_REGISTRY}/${NAMESPACE}/ose-elasticsearch-operator@sha256:5e50822dc262ddcec0ce65fd1dee14f61398047b95b5dfc438a3043b54d2f93d Trying to pull helper.ocp4.example.com:5000/olmtest-take1/ose-elasticsearch-operator@sha256:5e50822dc262ddcec0ce65fd1dee14f61398047b95b5dfc438a3043b54d2f93d... manifest unknown: manifest unknown Error: error pulling image "helper.ocp4.example.com:5000/olmtest-take1/ose-elasticsearch-operator@sha256:5e50822dc262ddcec0ce65fd1dee14f61398047b95b5dfc438a3043b54d2f93d": unable to pull helper.ocp4.example.com:5000/olmtest-take1/ose-elasticsearch-operator@sha256:5e50822dc262ddcec0ce65fd1dee14f61398047b95b5dfc438a3043b54d2f93d: unable to pull image: Error initializing source docker://helper.ocp4.example.com:5000/olmtest-take1/ose-elasticsearch-operator@sha256:5e50822dc262ddcec0ce65fd1dee14f61398047b95b5dfc438a3043b54d2f93d: Error reading manifest sha256:5e50822dc262ddcec0ce65fd1dee14f61398047b95b5dfc438a3043b54d2f93d in helper.ocp4.example.com:5000/olmtest-take1/ose-elasticsearch-operator: manifest unknown: manifest unknown [sushil@rhltop ~]$ ----------------------------------------------------------------------------------- What next ? Thank you. -- Regards Sushil Suresh.
Thanks to Shawn Hurley for digging into this a bit, there appears to be a bug(well, intentional decision but one that needs to be revisited) in oc image mirror in which if the manifestlist being mirrored has only a single manifest in it, oc image mirror only mirrors the single manifest instead of the manifestlist: https://github.com/openshift/oc/blob/680ad4847f7de413f5fa28ed31bd59fd4ba4af10/pkg/cli/image/manifest/manifest.go#L382-L388 So that's causing the behavior seen here even when --filter-by-os=.* is being passed. The workaround in the short term is to use "skopeo copy --all" to mirror the content instead, until this bug is resolved such that oc image mirror can do the right thing (possibly by introducing a new flag that allows you to choose whether you want the existing behavior or the full manifestlist mirroring, in order to preserve backward compatibility). Note that there is also a related bug in "oc adm catalog mirror" which prevents it from doing the right thing in these scenarios, i'll update this bug when i can track that down.
Finally i have it working. Yay !!! So as suggested I attempted to use skopeo copy --all To mirror the image and, I do not have an issue pulling down the image. So would you please mind referencing the bugs for both the oc image mirror and od adm catalog mirror commands. Thanks again. for helping me get to the bottom of this. Appreciate the help. Gist for readabiliy https://gist.github.com/sushilsuresh/c792fa207ecf94192a7005aeb2d222bb Pasting the output below as well. --------------------------------------------------------------------------------------- [sushil@rhltop ocp4-ansible-roles]$ [sushil@rhltop ocp4-ansible-roles]$ export MIRROR_REGISTRY=helper.ocp4.example.com:5000 [sushil@rhltop ocp4-ansible-roles]$ export NAMESPACE=olmtest-take1 6] [sushil@rhltop ocp4-ansible-roles]$ 6] [sushil@rhltop ocp4-ansible-roles]$ [sushil@rhltop ocp4-ansible-roles]$ skopeo copy --all docker://registry.redhat.io/openshift4/ose-elasticsearch-operator@sha256:5e50822dc262ddcec0ce65fd1dee14f61398047b95b5dfc438a3043b54d2f93d docker://${MIRROR_REGISTRY}/${NAMESPACE}/ose-e lasticsearch-operator Getting image list signatures 6] Copying 1 of 1 images in list Copying image sha256:4b790902039f3ea801643011f807d7fdc20c1bf014c923351f5eebba46b3d327 (1/1) Getting image source signatures Copying blob e232ae6317c2 done Copying blob c53747ad4b97 done Copying blob 34971b2d1eb9 done Copying blob 8d4203411daf done Copying blob 4fbc3bafa3d4 done Copying config b045933af4 done Writing manifest to image destination Storing signatures Writing manifest list to image destination Storing list signatures [sushil@rhltop ocp4-ansible-roles]$ [sushil@rhltop ocp4-ansible-roles]$ [sushil@rhltop ocp4-ansible-roles]$ ------------------------------------------ [root@rhltop ~]# [root@rhltop ~]# export MIRROR_REGISTRY=helper.ocp4.example.com:5000 [root@rhltop ~]# export NAMESPACE=olmtest-take1 [root@rhltop ~]# [root@rhltop ~]# [root@rhltop ~]# podman pull ${MIRROR_REGISTRY}/${NAMESPACE}/ose-elasticsearch-operator@sha256:5e50822dc262ddcec0ce65fd1dee14f61398047b95b5dfc438a3043b54d2f93d Trying to pull helper.ocp4.example.com:5000/olmtest-take1/ose-elasticsearch-operator@sha256:5e50822dc262ddcec0ce65fd1dee14f61398047b95b5dfc438a3043b54d2f93d... Getting image source signatures Copying blob 4fbc3bafa3d4 done Copying blob c53747ad4b97 done Copying blob 34971b2d1eb9 done Copying blob e232ae6317c2 done Copying blob 8d4203411daf done Copying config b045933af4 done Writing manifest to image destination Storing signatures b045933af4f7a0883f1a0366c2377a03b94a77bba24e7c15b4b4171dbfdb5638 [root@rhltop ~]# ---------------------------------------------------------------------------------------
This is the 4.3 bugs for the `oc adm catalog mirror` issue with mirroring manifest lists. https://bugzilla.redhat.com/show_bug.cgi?id=1797026
and this bug itself is the one for the oc image mirror behavior (that it doesn't copy the manifestlist if the manifestlist only contains a single manifest). I retitled it accordingly.
Sweet Thank you sir. Will track this. -- Regards Sushil Suresh.
(In reply to Ben Parees from comment #8) > Thanks to Shawn Hurley for digging into this a bit, there appears to be a > bug(well, intentional decision but one that needs to be revisited) in oc > image mirror in which if the manifestlist being mirrored has only a single > manifest in it, oc image mirror only mirrors the single manifest instead of > the manifestlist: > > https://github.com/openshift/oc/blob/ > 680ad4847f7de413f5fa28ed31bd59fd4ba4af10/pkg/cli/image/manifest/manifest. > go#L382-L388 > > So that's causing the behavior seen here even when --filter-by-os=.* is > being passed. > > The workaround in the short term is to use "skopeo copy --all" to mirror the > content instead, until this bug is resolved such that oc image mirror can do > the right thing (possibly by introducing a new flag that allows you to > choose whether you want the existing behavior or the full manifestlist > mirroring, in order to preserve backward compatibility). > > > Note that there is also a related bug in "oc adm catalog mirror" which > prevents it from doing the right thing in these scenarios, i'll update this > bug when i can track that down. @Ben Actually '--all' option is not there in 'skopeo-0.1.37-3.el7.x86_64' and this is the latest version in RHEL 7.7 , is there any other workaround which we can use in RHEL 7.7 which can only update to 'skopeo-0.1.37-3.el7.x86_64' or maybe if any repo present which has the latest official versin of skopeo which is 0.1.40.
Spoke too soon, I'm moving this to OLM team to decide about how they are planning to handle this.
*** Bug 1797203 has been marked as a duplicate of this bug. ***
I have encountered the same error where OC cli change digest of the mirrored image. The skopeo workaround helped. However, it'll be better if we can use OC cli. Any ETA on a fix?
We also hit that with a customer in a fully air-gapped scenario (Openshift 4.4.7, OC 4.4.7) during the attempt to mirror the OLM catalog to an internal registry. The Skopeo workaround worked, but we also observed an interesting fact: Focus on the last lines of the snippet below. 2 of 3 manifests are copied but the first is repeated and the one is missing. The missing hash is always the one that is referred to as the digest. Maybe that's a clue about what is going on. Also, only images with 3 manifests are affected, most other images only have 2, which works flawlessly. Please fix this issue, air-gapped installations are hard enough and bugs like this do raise concerns about the quality and stability of Openshift as an enterprise application ecosystem. --- oc image mirror -a pull-secret.txt --keep-manifest-list=true registry.redhat.io/openshift4/ose-elasticsearch-operator@sha256:882fff755fb513b7af659e5b47643bffe9d2ce9d492e208bbc4aa9b390570310=mirror.lab.example.com:5000/openshift4/ose-elasticsearch-operator mirror.lab.example.com:5000/ openshift4/ose-elasticsearch-operator blobs: registry.redhat.io/openshift4/ose-elasticsearch-operator sha256:82a8f4ea76cb6f833c5f179b3e6eda9f2267ed8ac7d1bf652f88ac3e9cc453d1 1.561KiB registry.redhat.io/openshift4/ose-elasticsearch-operator sha256:5caacc3f346e3cae733746c1d97e5b259333610acb9977953c8193646d61b65c 5.521KiB registry.redhat.io/openshift4/ose-elasticsearch-operator sha256:5808abd2c28ac0118bd858443443074e64a1819defecfaffd8b64f34e492fcf0 3.339MiB registry.redhat.io/openshift4/ose-elasticsearch-operator sha256:9f932d1a696e1cb4b35e1ac5e48b96a55a9b30abdbd11d312fd6e52ec5ead728 7.858MiB registry.redhat.io/openshift4/ose-elasticsearch-operator sha256:68b2a20998ac18623266c87e0ad6297e00a080f5b54c05de5c49738901f81fb1 17.48MiB registry.redhat.io/openshift4/ose-elasticsearch-operator sha256:a3ac36470b00df382448e79f7a749aa6833e4ac9cc90e3391f778820db9fa407 72.74MiB manifests: sha256:79238e45bffc48b49c69ece3e28a205c25e93e717665341cf7f1a8d6fe4e9424 sha256:882fff755fb513b7af659e5b47643bffe9d2ce9d492e208bbc4aa9b390570310 sha256:f142729ac4b87b219cf0871f91d736de96bf5c885e7ebe2c7caf876693e2614f stats: shared=0 unique=6 size=101.4MiB ratio=1.00 phase 0: mirror.lab.example.com:5000 openshift4/ose-elasticsearch-operator blobs=6 mounts=0 manifests=3 shared=0 info: Planning completed in 1.89s sha256:79238e45bffc48b49c69ece3e28a205c25e93e717665341cf7f1a8d6fe4e9424 mirror.lab.example.com:5000/openshift4/ose-elasticsearch-operator sha256:f142729ac4b87b219cf0871f91d736de96bf5c885e7ebe2c7caf876693e2614f mirror.lab.example.com:5000/openshift4/ose-elasticsearch-operator sha256:79238e45bffc48b49c69ece3e28a205c25e93e717665341cf7f1a8d6fe4e9424 mirror.lab.example.com:5000/openshift4/ose-elasticsearch-operator info: Mirroring completed in 430ms (0B/s)
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2020:2409
The 4.4 fix
Changing a CLOSED ERRATA bug back to NEW is not great, and will make the shipped 4.5.1 errata very confusing. Can we move this bug back to CLOSED ERRATA and create new bugs for any remaining issues?
Selim, I think the issue your command has is that you are using `--filter-by-os='linux/amd64`, to get the mirror to work you will need to use `--filter-by-os=.*` oc adm catalog mirror registry.home:7000/mirror/redhat-operators:test mirror.registry --filter-by-os='.*' --to-manifests=/var/lib/import/olm/redhat-operators/ --path=/:/var/lib/import/olm/redhat-operators/image Operator manifests reference images that are built with manifestlists, and in the background cri-o will pull the underlying image with the correct architecture. Marking this back as CLOSED ERRATA. If there is indeed an issue after this, feel free to open another bug and we can triage from there.