Description of problem: Version-Release number of selected component (if applicable): How reproducible: Steps to Reproduce: 1. 2. 3. Actual results: Expected results: Additional info:
Description: After mirroring an image from registry.redhat.io into a local podman registry, this image can not be pulled 1. oc image mirror registry.redhat.io/openshift4/ose-local-storage-operator@sha256:7bf8f73cb99ae708679348a9375a672e616404c81210997ef95a536432cfde7c=cnvqe-02.lab.eng.tlv2.redhat.com:5000/openshift4/ose-local-storage-operator -a ../pull-secret.txt ----> cnvqe-02.lab.eng.tlv2.redhat.com:5000/ openshift4/ose-local-storage-operator blobs: registry.redhat.io/openshift4/ose-local-storage-operator sha256:bb0da44cdbced801240e74437a617d4fe0e39c29cf3bbabb7f6a96d2620cfeaa 1.689KiB registry.redhat.io/openshift4/ose-local-storage-operator sha256:64d17e55f311d519bca4902d407c333cf8538910f6e0902f7ee48094a4d632f4 5.358KiB registry.redhat.io/openshift4/ose-local-storage-operator sha256:ec4ff9475976a7b289648fec6e13c415797b384d450c78c65949dee98a589b86 3.341MiB registry.redhat.io/openshift4/ose-local-storage-operator sha256:5d09098707ef3c5e92bf1c2d72fe6640a3e660e4d795f28e883ea2447990e9d9 9.158MiB registry.redhat.io/openshift4/ose-local-storage-operator sha256:8201a0884cd07adc5a2a0f6a3d69d1707a302e799012efefed86c5e9de03fc5b 16.16MiB registry.redhat.io/openshift4/ose-local-storage-operator sha256:a03401a44180b6581a149376d6fd2d5bd85d938445fd5b5ad270e14ddde4937c 72.71MiB manifests: sha256:c00b5b3f8d446a0c76b7a638ba144f173298d62ffcfdf3adc29e159e5c674f82 stats: shared=0 unique=6 size=101.4MiB ratio=1.00 phase 0: cnvqe-02.lab.eng.tlv2.redhat.com:5000 openshift4/ose-local-storage-operator blobs=6 mounts=0 manifests=1 shared=0 info: Planning completed in 2.98s sha256:c00b5b3f8d446a0c76b7a638ba144f173298d62ffcfdf3adc29e159e5c674f82 cnvqe-02.lab.eng.tlv2.redhat.com:5000/openshift4/ose-local-storage-operator info: Mirroring completed in 400ms (0B/s) 2. docker pull cnvqe-02.lab.eng.tlv2.redhat.com:5000/openshift4/ose-local-storage-operator@sha256:7bf8f73cb99ae708679348a9375a672e616404c81210997ef95a536432cfde7c Trying to pull cnvqe-02.lab.eng.tlv2.redhat.com:5000/openshift4/ose-local-storage-operator@sha256:7bf8f73cb99ae708679348a9375a672e616404c81210997ef95a536432cfde7c... manifest unknown: manifest unknown Error: error pulling image "cnvqe-02.lab.eng.tlv2.redhat.com:5000/openshift4/ose-local-storage-operator@sha256:7bf8f73cb99ae708679348a9375a672e616404c81210997ef95a536432cfde7c": unable to pull cnvqe-02.lab.eng.tlv2.redhat.com:5000/openshift4/ose-local-storage-operator@sha256:7bf8f73cb99ae708679348a9375a672e616404c81210997ef95a536432cfde7c: unable to pull image: Error initializing source docker://cnvqe-02.lab.eng.tlv2.redhat.com:5000/openshift4/ose-local-storage-operator@sha256:7bf8f73cb99ae708679348a9375a672e616404c81210997ef95a536432cfde7c: Error reading manifest sha256:7bf8f73cb99ae708679348a9375a672e616404c81210997ef95a536432cfde7c in cnvqe-02.lab.eng.tlv2.redhat.com:5000/openshift4/ose-local-storage-operator: manifest unknown: manifest unknown
I find if you tag the mirrored image like so: oc image mirror registry.redhat.io/openshift4/ose-local-storage-operator@sha256:7bf8f73cb99ae708679348a9375a672e616404c81210997ef95a536432cfde7c=yourregistry:5000/openshift4/ose-local-storage-operator:anything -a ../pull-secret.txt then you can podman|docker pull yourregistry:5000/openshift4/ose-local-storage-operator:anything and a podman|docker inspect will show "RepoDigests": [ "yourregistry:5000/openshift4/ose-local-storage-operator@sha256:c00b5b3f8d446a0c76b7a638ba144f173298d62ffcfdf3adc29e159e5c674f82" ], Now I will investigate whether there is a bug or if it is not intended to mirror images from their digest like so: 'oc image mirror registry/repository/name@sha256:digest' Looking at the help menu, oc image mirror examples only show mirroring either from or to a tag.
(In reply to Sally from comment #3) > I find if you tag the mirrored image like so: > > oc image mirror > registry.redhat.io/openshift4/ose-local-storage-operator@sha256: > 7bf8f73cb99ae708679348a9375a672e616404c81210997ef95a536432cfde7c=yourregistry > :5000/openshift4/ose-local-storage-operator:anything -a ../pull-secret.txt > > then you can podman|docker pull > yourregistry:5000/openshift4/ose-local-storage-operator:anything > > and a podman|docker inspect will show > "RepoDigests": [ > > "yourregistry:5000/openshift4/ose-local-storage-operator@sha256: > c00b5b3f8d446a0c76b7a638ba144f173298d62ffcfdf3adc29e159e5c674f82" > ], > > Now I will investigate whether there is a bug or if it is not intended to > mirror images from their digest like so: > 'oc image mirror registry/repository/name@sha256:digest' > > Looking at the help menu, oc image mirror examples only show mirroring > either from or to a tag. image mirror with some tag will not help in this situation, since the pull is done with digest notation. tag is not supported with ImageContentSourcePolicy, used in disconnected clusters, where operators images are cloned into local registry, and later pulled.
The image digest is modified during the mirror, see https://bugzilla.redhat.com/show_bug.cgi?id=1859452#c1 the output of the `oc image mirror`: sha256:c00b5b3f8d446a0c76b7a638ba144f173298d62ffcfdf3adc29e159e5c674f82 cnvqe-02.lab.eng.tlv2.redhat.com:5000/openshift4/ose-local-storage-operator info: Mirroring completed in 400ms (0B/s) So, you can pull by digest (please confirm) with: docker pull cnvqe-02.lab.eng.tlv2.redhat.com:5000/openshift4/ose-local-storage-operator@sha256:c00b5b3f8d446a0c76b7a638ba144f173298d62ffcfdf3adc29e159e5c674f82 The original digest (sha256:7bf8f73cb99ae708679348a9375a...) was modified when the image was copied to the local registry. I'm looking into why `oc image mirror` is not preserving the digest, will report back.
(In reply to Sally from comment #5) > The image digest is modified during the mirror, see > https://bugzilla.redhat.com/show_bug.cgi?id=1859452#c1 the output of the > `oc image mirror`: > > sha256:c00b5b3f8d446a0c76b7a638ba144f173298d62ffcfdf3adc29e159e5c674f82 > cnvqe-02.lab.eng.tlv2.redhat.com:5000/openshift4/ose-local-storage-operator > info: Mirroring completed in 400ms (0B/s) > > So, you can pull by digest (please confirm) with: > docker pull > cnvqe-02.lab.eng.tlv2.redhat.com:5000/openshift4/ose-local-storage- > operator@sha256: > c00b5b3f8d446a0c76b7a638ba144f173298d62ffcfdf3adc29e159e5c674f82 > > The original digest (sha256:7bf8f73cb99ae708679348a9375a...) was modified > when the image was copied to the local registry. > > I'm looking into why `oc image mirror` is not preserving the digest, will > report back. I can't pull with the new digest, image is pulled by OLM (with original digest), and it's redirected to local-registry by imageContentSourcePolicy (in a disconnected/restricted cluster)
When you run an inspect on this image, you'll see the digest that will be preserved when you mirror this image. If it's a multi-arch image like this one is, you'll see the digest that is for your arch (in my case I see "x86_64"): $ podman inspect registry.redhat.io/openshift4/ose-local-storage-operator@sha256:7bf8f73cb99ae708679348a9375a672e616404c81210997ef95a536432cfde7c [ { "Id": "64d17e55f311d519bca4902d407c333cf8538910f6e0902f7ee48094a4d632f4", "Digest": "sha256:c00b5b3f8d446a0c76b7a638ba144f173298d62ffcfdf3adc29e159e5c674f82", "RepoTags": null, "RepoDigests": [ "registry.redhat.io/openshift4/ose-local-storage-operator@sha256:7bf8f73cb99ae708679348a9375a672e616404c81210997ef95a536432cfde7c", "registry.redhat.io/openshift4/ose-local-storage-operator@sha256:c00b5b3f8d446a0c76b7a638ba144f173298d62ffcfdf3adc29e159e5c674f82" ], See the "Digest": "sha256:c00b5b3f8d446a0c76b7a638ba144f173298d62ffcfdf3adc29e159e5c674f82" - that is the resulting digest when you copy or mirror this image, and you will be able to pull the image by that digest.
You can see the digest is preserved when you mirror the single arch image ("architecture": "x86_64" here): $ oc image mirror registry.redhat.io/openshift4/ose-local-storage-operator@sha256:c00b5b3f8d446a0c76b7a638ba144f173298d62ffcfdf3adc29e159e5c674f82=localhost:5000/openshift4/ose-local-storage-operator -a ~/installer/pull-secret-local localhost:5000/ openshift4/ose-local-storage-operator manifests: sha256:c00b5b3f8d446a0c76b7a638ba144f173298d62ffcfdf3adc29e159e5c674f82 stats: shared=0 unique=0 size=0B phase 0: localhost:5000 openshift4/ose-local-storage-operator blobs=0 mounts=0 manifests=1 shared=0 info: Planning completed in 1.63s sha256:c00b5b3f8d446a0c76b7a638ba144f173298d62ffcfdf3adc29e159e5c674f82 localhost:5000/openshift4/ose-local-storage-operator info: Mirroring completed in 10ms (0B/s)
Also, with multi-arch images, mirroring with the 'filter-by-os=/*' flag results in mirroring all manifests, and then all will be pull-able w/ podman or docker: $ oc image mirror registry.redhat.io/openshift4/ose-local-storage-operator@sha256:7bf8f73cb99ae708679348a9375a672e616404c81210997ef95a536432cfde7c=localhost:5000/openshift4/ose-local-storage-operator --filter-by-os=/* I0729 23:15:11.082924 334895 manifest.go:348] manifestDigest: sha256:7bf8f73cb99ae708679348a9375a672e616404c81210997ef95a536432cfde7c I0729 23:15:11.083103 334895 manifest.go:358] FILTERED: [{{application/vnd.docker.distribution.manifest.v2+json 1371 sha256:c00b5b3f8d446a0c76b7a638ba144f173298d62ffcfdf3adc29e159e5c674f82 [] map[] <nil>} {amd64 linux [] []}}] I0729 23:15:11.083167 334895 manifest.go:358] FILTERED: [{{application/vnd.docker.distribution.manifest.v2+json 1371 sha256:c00b5b3f8d446a0c76b7a638ba144f173298d62ffcfdf3adc29e159e5c674f82 [] map[] <nil>} {amd64 linux [] []}} {{application/vnd.docker.distribution.manifest.v2+json 1372 sha256:5adc972084e6a20303629b7566208b83db58c8cef3370b0041cf6a4ae35673e1 [] map[] <nil>} {ppc64le linux [] []}}] I0729 23:15:11.083216 334895 manifest.go:358] FILTERED: [{{application/vnd.docker.distribution.manifest.v2+json 1371 sha256:c00b5b3f8d446a0c76b7a638ba144f173298d62ffcfdf3adc29e159e5c674f82 [] map[] <nil>} {amd64 linux [] []}} {{application/vnd.docker.distribution.manifest.v2+json 1372 sha256:5adc972084e6a20303629b7566208b83db58c8cef3370b0041cf6a4ae35673e1 [] map[] <nil>} {ppc64le linux [] []}} {{application/vnd.docker.distribution.manifest.v2+json 1372 sha256:c1813865a8f8e69fab4dbcbbfc318b3555de690123589726feea67f7595982af [] map[] <nil>} {s390x linux [] []}}] localhost:5000/ openshift4/ose-local-storage-operator manifests: sha256:5adc972084e6a20303629b7566208b83db58c8cef3370b0041cf6a4ae35673e1 sha256:7bf8f73cb99ae708679348a9375a672e616404c81210997ef95a536432cfde7c sha256:c00b5b3f8d446a0c76b7a638ba144f173298d62ffcfdf3adc29e159e5c674f82 sha256:c1813865a8f8e69fab4dbcbbfc318b3555de690123589726feea67f7595982af stats: shared=0 unique=0 size=0B phase 0: localhost:5000 openshift4/ose-local-storage-operator blobs=0 mounts=0 manifests=4 shared=0 info: Planning completed in 4.03s sha256:c1813865a8f8e69fab4dbcbbfc318b3555de690123589726feea67f7595982af localhost:5000/openshift4/ose-local-storage-operator sha256:5adc972084e6a20303629b7566208b83db58c8cef3370b0041cf6a4ae35673e1 localhost:5000/openshift4/ose-local-storage-operator sha256:c00b5b3f8d446a0c76b7a638ba144f173298d62ffcfdf3adc29e159e5c674f82 localhost:5000/openshift4/ose-local-storage-operator sha256:7bf8f73cb99ae708679348a9375a672e616404c81210997ef95a536432cfde7c localhost:5000/openshift4/ose-local-storage-operator info: Mirroring completed in 20ms (0B/s) and then: $ podman pull localhost:5000/openshift4/ose-local-storage-operator@sha256:7bf8f73cb99ae708679348a9375a672e616404c81210997ef95a536432cfde7c succeeds
Hi Dan, this is an issue with mirroring images w/ multiple manifests (multi-arch images) but it is with 'oc image mirror' and the issue is with documentation/improving help menu/flags. Nothing on your end needs fixing AFAICT. Thanks! Asher, there are 2 flags you can pass that will result in mirroring all manifests of a multi-arch image: $ oc image mirror registry.redhat.io/openshift4/ose-local-storage-operator@sha256:7bf8f73cb99ae708679348a9375a672e616404c81210997ef95a536432cfde7c=local-registry/openshift4/ose-local-storage-operator --keep-manifest-list $ oc image mirror registry.redhat.io/openshift4/ose-local-storage-operator@sha256:7bf8f73cb99ae708679348a9375a672e616404c81210997ef95a536432cfde7c=localhost:5000/openshift4/ose-local-storage-operator --filter-by-os=.* --keep-manifest-list and --filter-by-os=.* result in mirroring all manifests - I'll be updating the help menu and documentation to make this more clear.
Setting status to 'Upcoming Sprint' as I'm actively working on this bug.
I've improved the help menu to guide users when using 'oc image mirror' and multi-arch images. The help menu clarifies that to mirror multi-arch image you should use the flags --filter-by-os=.* to mirror all architectures, --filter-by-os=linux/s390x to mirror only a particular os/arch, or nothing to default to mirroring whatever your system os/arch is. That's all that's required to resolve this issue. Closing as not a bug, since the digest is preserved, just that there was the unexpected os/arch digest different than the manifest list digest.