Bug 1859452 - fail to pull image mirrored by oc image mirror
Summary: fail to pull image mirrored by oc image mirror
Keywords:
Status: CLOSED NOTABUG
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: oc
Version: 4.5
Hardware: Unspecified
OS: Unspecified
medium
medium
Target Milestone: ---
: 4.6.0
Assignee: Sally
QA Contact: zhou ying
URL:
Whiteboard: non-multi-arch
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2020-07-22 07:35 UTC by Asher Shoshan
Modified: 2020-08-20 21:38 UTC (History)
6 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2020-08-20 21:38:05 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Github openshift oc pull 519 0 None closed Bug 1859452: Improve help menu when 'oc image mirror' and multi-arch images 2020-12-04 15:53:37 UTC

Description Asher Shoshan 2020-07-22 07:35:02 UTC
Description of problem:


Version-Release number of selected component (if applicable):


How reproducible:


Steps to Reproduce:
1.
2.
3.

Actual results:


Expected results:


Additional info:

Comment 1 Asher Shoshan 2020-07-22 07:40:41 UTC
Description:

After mirroring an image from registry.redhat.io into a local podman registry, this image can not be pulled

1. oc image mirror registry.redhat.io/openshift4/ose-local-storage-operator@sha256:7bf8f73cb99ae708679348a9375a672e616404c81210997ef95a536432cfde7c=cnvqe-02.lab.eng.tlv2.redhat.com:5000/openshift4/ose-local-storage-operator -a ../pull-secret.txt

---->
cnvqe-02.lab.eng.tlv2.redhat.com:5000/
  openshift4/ose-local-storage-operator
    blobs:
      registry.redhat.io/openshift4/ose-local-storage-operator sha256:bb0da44cdbced801240e74437a617d4fe0e39c29cf3bbabb7f6a96d2620cfeaa 1.689KiB
      registry.redhat.io/openshift4/ose-local-storage-operator sha256:64d17e55f311d519bca4902d407c333cf8538910f6e0902f7ee48094a4d632f4 5.358KiB
      registry.redhat.io/openshift4/ose-local-storage-operator sha256:ec4ff9475976a7b289648fec6e13c415797b384d450c78c65949dee98a589b86 3.341MiB
      registry.redhat.io/openshift4/ose-local-storage-operator sha256:5d09098707ef3c5e92bf1c2d72fe6640a3e660e4d795f28e883ea2447990e9d9 9.158MiB
      registry.redhat.io/openshift4/ose-local-storage-operator sha256:8201a0884cd07adc5a2a0f6a3d69d1707a302e799012efefed86c5e9de03fc5b 16.16MiB
      registry.redhat.io/openshift4/ose-local-storage-operator sha256:a03401a44180b6581a149376d6fd2d5bd85d938445fd5b5ad270e14ddde4937c 72.71MiB
    manifests:
      sha256:c00b5b3f8d446a0c76b7a638ba144f173298d62ffcfdf3adc29e159e5c674f82
  stats: shared=0 unique=6 size=101.4MiB ratio=1.00

phase 0:
  cnvqe-02.lab.eng.tlv2.redhat.com:5000 openshift4/ose-local-storage-operator blobs=6 mounts=0 manifests=1 shared=0

info: Planning completed in 2.98s
sha256:c00b5b3f8d446a0c76b7a638ba144f173298d62ffcfdf3adc29e159e5c674f82 cnvqe-02.lab.eng.tlv2.redhat.com:5000/openshift4/ose-local-storage-operator
info: Mirroring completed in 400ms (0B/s)


2. docker pull cnvqe-02.lab.eng.tlv2.redhat.com:5000/openshift4/ose-local-storage-operator@sha256:7bf8f73cb99ae708679348a9375a672e616404c81210997ef95a536432cfde7c
Trying to pull cnvqe-02.lab.eng.tlv2.redhat.com:5000/openshift4/ose-local-storage-operator@sha256:7bf8f73cb99ae708679348a9375a672e616404c81210997ef95a536432cfde7c...
  manifest unknown: manifest unknown
Error: error pulling image "cnvqe-02.lab.eng.tlv2.redhat.com:5000/openshift4/ose-local-storage-operator@sha256:7bf8f73cb99ae708679348a9375a672e616404c81210997ef95a536432cfde7c": unable to pull cnvqe-02.lab.eng.tlv2.redhat.com:5000/openshift4/ose-local-storage-operator@sha256:7bf8f73cb99ae708679348a9375a672e616404c81210997ef95a536432cfde7c: unable to pull image: Error initializing source docker://cnvqe-02.lab.eng.tlv2.redhat.com:5000/openshift4/ose-local-storage-operator@sha256:7bf8f73cb99ae708679348a9375a672e616404c81210997ef95a536432cfde7c: Error reading manifest sha256:7bf8f73cb99ae708679348a9375a672e616404c81210997ef95a536432cfde7c in cnvqe-02.lab.eng.tlv2.redhat.com:5000/openshift4/ose-local-storage-operator: manifest unknown: manifest unknown

Comment 3 Sally 2020-07-27 23:26:31 UTC
I find if you tag the mirrored image like so:

oc image mirror registry.redhat.io/openshift4/ose-local-storage-operator@sha256:7bf8f73cb99ae708679348a9375a672e616404c81210997ef95a536432cfde7c=yourregistry:5000/openshift4/ose-local-storage-operator:anything -a ../pull-secret.txt

then you can podman|docker pull yourregistry:5000/openshift4/ose-local-storage-operator:anything

and a podman|docker inspect will show 
"RepoDigests": [
            "yourregistry:5000/openshift4/ose-local-storage-operator@sha256:c00b5b3f8d446a0c76b7a638ba144f173298d62ffcfdf3adc29e159e5c674f82"
],

Now I will investigate whether there is a bug or if it is not intended to mirror images from their digest like so:
'oc image mirror registry/repository/name@sha256:digest'

Looking at the help menu, oc image mirror examples only show mirroring either from or to a tag.

Comment 4 Asher Shoshan 2020-07-28 07:04:12 UTC
(In reply to Sally from comment #3)
> I find if you tag the mirrored image like so:
> 
> oc image mirror
> registry.redhat.io/openshift4/ose-local-storage-operator@sha256:
> 7bf8f73cb99ae708679348a9375a672e616404c81210997ef95a536432cfde7c=yourregistry
> :5000/openshift4/ose-local-storage-operator:anything -a ../pull-secret.txt
> 
> then you can podman|docker pull
> yourregistry:5000/openshift4/ose-local-storage-operator:anything
> 
> and a podman|docker inspect will show 
> "RepoDigests": [
>            
> "yourregistry:5000/openshift4/ose-local-storage-operator@sha256:
> c00b5b3f8d446a0c76b7a638ba144f173298d62ffcfdf3adc29e159e5c674f82"
> ],
> 
> Now I will investigate whether there is a bug or if it is not intended to
> mirror images from their digest like so:
> 'oc image mirror registry/repository/name@sha256:digest'
> 
> Looking at the help menu, oc image mirror examples only show mirroring
> either from or to a tag.

image mirror with some tag will not help in this situation, since the pull is done with digest notation.
tag is not supported with ImageContentSourcePolicy, used in disconnected clusters, where operators images are cloned into local registry, and later pulled.

Comment 5 Sally 2020-07-29 04:10:50 UTC
The image digest is modified during the mirror, see https://bugzilla.redhat.com/show_bug.cgi?id=1859452#c1  the output of the `oc image mirror`:

sha256:c00b5b3f8d446a0c76b7a638ba144f173298d62ffcfdf3adc29e159e5c674f82 cnvqe-02.lab.eng.tlv2.redhat.com:5000/openshift4/ose-local-storage-operator
info: Mirroring completed in 400ms (0B/s)

So, you can pull by digest (please confirm) with:
docker pull cnvqe-02.lab.eng.tlv2.redhat.com:5000/openshift4/ose-local-storage-operator@sha256:c00b5b3f8d446a0c76b7a638ba144f173298d62ffcfdf3adc29e159e5c674f82  

The original digest (sha256:7bf8f73cb99ae708679348a9375a...) was modified when the image was copied to the local registry.

I'm looking into why `oc image mirror` is not preserving the digest, will report back.

Comment 6 Asher Shoshan 2020-07-29 07:44:31 UTC
(In reply to Sally from comment #5)
> The image digest is modified during the mirror, see
> https://bugzilla.redhat.com/show_bug.cgi?id=1859452#c1  the output of the
> `oc image mirror`:
> 
> sha256:c00b5b3f8d446a0c76b7a638ba144f173298d62ffcfdf3adc29e159e5c674f82
> cnvqe-02.lab.eng.tlv2.redhat.com:5000/openshift4/ose-local-storage-operator
> info: Mirroring completed in 400ms (0B/s)
> 
> So, you can pull by digest (please confirm) with:
> docker pull
> cnvqe-02.lab.eng.tlv2.redhat.com:5000/openshift4/ose-local-storage-
> operator@sha256:
> c00b5b3f8d446a0c76b7a638ba144f173298d62ffcfdf3adc29e159e5c674f82  
> 
> The original digest (sha256:7bf8f73cb99ae708679348a9375a...) was modified
> when the image was copied to the local registry.
> 
> I'm looking into why `oc image mirror` is not preserving the digest, will
> report back.

I can't pull with the new digest, image is pulled by OLM (with original digest), and it's redirected to local-registry by imageContentSourcePolicy (in a disconnected/restricted cluster)

Comment 7 Sally 2020-07-29 19:37:05 UTC
When you run an inspect on this image, you'll see the digest that will be preserved when you mirror this image.  If it's a multi-arch image like this one is, you'll see the digest that is for your arch (in my case I see "x86_64"):

$ podman inspect registry.redhat.io/openshift4/ose-local-storage-operator@sha256:7bf8f73cb99ae708679348a9375a672e616404c81210997ef95a536432cfde7c
[
    {
        "Id": "64d17e55f311d519bca4902d407c333cf8538910f6e0902f7ee48094a4d632f4",
        "Digest": "sha256:c00b5b3f8d446a0c76b7a638ba144f173298d62ffcfdf3adc29e159e5c674f82",
        "RepoTags": null,
        "RepoDigests": [
            "registry.redhat.io/openshift4/ose-local-storage-operator@sha256:7bf8f73cb99ae708679348a9375a672e616404c81210997ef95a536432cfde7c",
            "registry.redhat.io/openshift4/ose-local-storage-operator@sha256:c00b5b3f8d446a0c76b7a638ba144f173298d62ffcfdf3adc29e159e5c674f82"
        ],

See the "Digest": "sha256:c00b5b3f8d446a0c76b7a638ba144f173298d62ffcfdf3adc29e159e5c674f82" - that is the resulting digest when you copy or mirror this image, and you will be able to pull the image by that digest.

Comment 8 Sally 2020-07-29 19:39:16 UTC
You can see the digest is preserved when you mirror the single arch image ("architecture": "x86_64" here):

$ oc image mirror registry.redhat.io/openshift4/ose-local-storage-operator@sha256:c00b5b3f8d446a0c76b7a638ba144f173298d62ffcfdf3adc29e159e5c674f82=localhost:5000/openshift4/ose-local-storage-operator -a ~/installer/pull-secret-local
localhost:5000/
  openshift4/ose-local-storage-operator
    manifests:
      sha256:c00b5b3f8d446a0c76b7a638ba144f173298d62ffcfdf3adc29e159e5c674f82
  stats: shared=0 unique=0 size=0B

phase 0:
  localhost:5000 openshift4/ose-local-storage-operator blobs=0 mounts=0 manifests=1 shared=0

info: Planning completed in 1.63s
sha256:c00b5b3f8d446a0c76b7a638ba144f173298d62ffcfdf3adc29e159e5c674f82 localhost:5000/openshift4/ose-local-storage-operator
info: Mirroring completed in 10ms (0B/s)

Comment 9 Sally 2020-07-30 03:21:23 UTC
Also, with multi-arch images, mirroring with the 'filter-by-os=/*' flag results in mirroring all manifests, and then all will be pull-able w/ podman or docker:

$ oc image mirror registry.redhat.io/openshift4/ose-local-storage-operator@sha256:7bf8f73cb99ae708679348a9375a672e616404c81210997ef95a536432cfde7c=localhost:5000/openshift4/ose-local-storage-operator --filter-by-os=/*
I0729 23:15:11.082924  334895 manifest.go:348] manifestDigest: sha256:7bf8f73cb99ae708679348a9375a672e616404c81210997ef95a536432cfde7c
I0729 23:15:11.083103  334895 manifest.go:358] FILTERED: [{{application/vnd.docker.distribution.manifest.v2+json 1371 sha256:c00b5b3f8d446a0c76b7a638ba144f173298d62ffcfdf3adc29e159e5c674f82 [] map[] <nil>} {amd64 linux  []  []}}]
I0729 23:15:11.083167  334895 manifest.go:358] FILTERED: [{{application/vnd.docker.distribution.manifest.v2+json 1371 sha256:c00b5b3f8d446a0c76b7a638ba144f173298d62ffcfdf3adc29e159e5c674f82 [] map[] <nil>} {amd64 linux  []  []}} {{application/vnd.docker.distribution.manifest.v2+json 1372 sha256:5adc972084e6a20303629b7566208b83db58c8cef3370b0041cf6a4ae35673e1 [] map[] <nil>} {ppc64le linux  []  []}}]
I0729 23:15:11.083216  334895 manifest.go:358] FILTERED: [{{application/vnd.docker.distribution.manifest.v2+json 1371 sha256:c00b5b3f8d446a0c76b7a638ba144f173298d62ffcfdf3adc29e159e5c674f82 [] map[] <nil>} {amd64 linux  []  []}} {{application/vnd.docker.distribution.manifest.v2+json 1372 sha256:5adc972084e6a20303629b7566208b83db58c8cef3370b0041cf6a4ae35673e1 [] map[] <nil>} {ppc64le linux  []  []}} {{application/vnd.docker.distribution.manifest.v2+json 1372 sha256:c1813865a8f8e69fab4dbcbbfc318b3555de690123589726feea67f7595982af [] map[] <nil>} {s390x linux  []  []}}]
localhost:5000/
  openshift4/ose-local-storage-operator
    manifests:
      sha256:5adc972084e6a20303629b7566208b83db58c8cef3370b0041cf6a4ae35673e1
      sha256:7bf8f73cb99ae708679348a9375a672e616404c81210997ef95a536432cfde7c
      sha256:c00b5b3f8d446a0c76b7a638ba144f173298d62ffcfdf3adc29e159e5c674f82
      sha256:c1813865a8f8e69fab4dbcbbfc318b3555de690123589726feea67f7595982af
  stats: shared=0 unique=0 size=0B

phase 0:
  localhost:5000 openshift4/ose-local-storage-operator blobs=0 mounts=0 manifests=4 shared=0

info: Planning completed in 4.03s
sha256:c1813865a8f8e69fab4dbcbbfc318b3555de690123589726feea67f7595982af localhost:5000/openshift4/ose-local-storage-operator
sha256:5adc972084e6a20303629b7566208b83db58c8cef3370b0041cf6a4ae35673e1 localhost:5000/openshift4/ose-local-storage-operator
sha256:c00b5b3f8d446a0c76b7a638ba144f173298d62ffcfdf3adc29e159e5c674f82 localhost:5000/openshift4/ose-local-storage-operator
sha256:7bf8f73cb99ae708679348a9375a672e616404c81210997ef95a536432cfde7c localhost:5000/openshift4/ose-local-storage-operator
info: Mirroring completed in 20ms (0B/s)

and then:
$ podman pull localhost:5000/openshift4/ose-local-storage-operator@sha256:7bf8f73cb99ae708679348a9375a672e616404c81210997ef95a536432cfde7c  succeeds

Comment 11 Sally 2020-07-30 15:28:56 UTC
Hi Dan, this is an issue with mirroring images w/ multiple manifests (multi-arch images) but it is with 'oc image mirror' and the issue is with documentation/improving help menu/flags.  Nothing on your end needs fixing AFAICT.  Thanks!

Asher, there are 2 flags you can pass that will result in mirroring all manifests of a multi-arch image:

$ oc image mirror registry.redhat.io/openshift4/ose-local-storage-operator@sha256:7bf8f73cb99ae708679348a9375a672e616404c81210997ef95a536432cfde7c=local-registry/openshift4/ose-local-storage-operator --keep-manifest-list

$ oc image mirror registry.redhat.io/openshift4/ose-local-storage-operator@sha256:7bf8f73cb99ae708679348a9375a672e616404c81210997ef95a536432cfde7c=localhost:5000/openshift4/ose-local-storage-operator --filter-by-os=.*

--keep-manifest-list and --filter-by-os=.* result in mirroring all manifests - I'll be updating the help menu and documentation to make this more clear.

Comment 12 Sally 2020-07-30 21:28:38 UTC
Setting status to 'Upcoming Sprint' as I'm actively working on this bug.

Comment 13 Sally 2020-08-20 21:38:05 UTC
I've improved the help menu to guide users when using 'oc image mirror' and multi-arch images.  

The help menu clarifies that to mirror multi-arch image you should use the flags --filter-by-os=.* to mirror all architectures, --filter-by-os=linux/s390x to mirror only a particular os/arch, or nothing to default to mirroring whatever your system os/arch is.  That's all that's required to resolve this issue.  Closing as not a bug, since the digest is preserved, just that there was the unexpected os/arch digest different than the manifest list digest.


Note You need to log in before you can comment on or make changes to this bug.