Bug 1800674 - oc image mirror doesn't mirror manifestlist for single image manifestlists [NEEDINFO]
Summary: oc image mirror doesn't mirror manifestlist for single image manifestlists
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: OLM
Version: 4.4
Hardware: x86_64
OS: Unspecified
urgent
urgent
Target Milestone: ---
: 4.6.0
Assignee: Kevin Rizza
QA Contact: Jian Zhang
URL:
Whiteboard:
: 1797203 (view as bug list)
Depends On:
Blocks: 1780703 1804600
TreeView+ depends on / blocked
 
Reported: 2020-02-07 16:55 UTC by Erik Lalancette
Modified: 2023-12-15 17:17 UTC (History)
37 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
: 1804600 (view as bug list)
Environment:
Last Closed: 2020-08-21 11:28:39 UTC
Target Upstream Version:
Embargoed:
bparees: needinfo? (susuresh)


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Github openshift oc pull 298 0 None closed Bug 1800674: Option to force RepoDigest mirror 2021-02-16 15:24:53 UTC
Github openshift oc pull 301 0 None closed Bug 1800674: Option to force RepoDigest mirror 2021-02-16 15:24:54 UTC
Red Hat Product Errata RHBA-2020:2409 0 None None None 2020-07-13 17:14:49 UTC

Description Erik Lalancette 2020-02-07 16:55:36 UTC
Description of problem:

4.3 channel is publishing the image url as 
registry.redhat.io/openshift4/ose-elasticsearch-operator@sha256:5e50822dc262ddcec0ce65fd1dee14f61398047b95b5dfc438a3043b54d2f93d

I would expect registry.redhat.io/openshift4/ose-elasticsearch-operator@sha256:5e50822dc262ddcec0ce65fd1dee14f61398047b95b5dfc438a3043b54d2f93d to be the sh256 Digest of the ose-elasticsearch-operator image, but it is not.

sha256:5e50822dc262ddcec0ce65fd1dee14f61398047b95b5dfc438a3043b54d2f93d is the RepoDigest of the image, not the Digest of the image.

Version-Release number of selected component (if applicable):

4.3

How reproducible:

Manually pulling the image referenced in the 4.3 update channel
registry.redhat.io/openshift4/ose-elasticsearch-operator@sha256:5e50822dc262ddcec0ce65fd1dee14f61398047b95b5dfc438a3043b54d2f93d

$ podman images | grep ose-elasticsearch  # just to show that i don't have any existing images

$ podman pull registry.redhat.io/openshift4/ose-elasticsearch-operator@sha256:5e50822dc262ddcec0ce65fd1dee14f61398047b95b5dfc438a3043b54d2f93d 
Trying to pull registry.redhat.io/openshift4/ose-elasticsearch-operator@sha256:5e50822dc262ddcec0ce65fd1dee14f61398047b95b5dfc438a3043b54d2f93d...
Getting image source signatures
Copying blob 4fbc3bafa3d4 skipped: already exists  
Copying blob 8d4203411daf skipped: already exists  
Copying blob e232ae6317c2 skipped: already exists  
Copying blob 34971b2d1eb9 skipped: already exists  
Copying blob c53747ad4b97 done  
Copying config b045933af4 done  
Writing manifest to image destination
Storing signatures
b045933af4f7a0883f1a0366c2377a03b94a77bba24e7c15b4b4171dbfdb5638

$ podman inspect b045933af4f7a0883f1a0366c2377a03b94a77bba24e7c15b4b4171dbfdb5638 | jq '.[0]|.Digest,.RepoDigests'
"sha256:4b790902039f3ea801643011f807d7fdc20c1bf014c923351f5eebba46b3d327"
[
  "registry.redhat.io/openshift4/ose-elasticsearch-operator@sha256:4b790902039f3ea801643011f807d7fdc20c1bf014c923351f5eebba46b3d327",
  "registry.redhat.io/openshift4/ose-elasticsearch-operator@sha256:5e50822dc262ddcec0ce65fd1dee14f61398047b95b5dfc438a3043b54d2f93d"
] 
$ podman rmi b045933af4f7a0883f1a0366c2377a03b94a77bba24e7c15b4b4171dbfdb5638
Untagged: registry.redhat.io/openshift4/ose-elasticsearch-operator@sha256:5e50822dc262ddcec0ce65fd1dee14f61398047b95b5dfc438a3043b54d2f93d
Deleted: b045933af4f7a0883f1a0366c2377a03b94a77bba24e7c15b4b4171dbfdb5638
$ 
When I use 
jq '.[0]|.Digest,.RepoDigests'
 to extract the Digest and RepoDigest fields for the image
you can see that sha256:5e50822dc262ddcec0ce65fd1dee14f61398047b95b5dfc438a3043b54d2f93d is the RepoDigest and not the Digest for the image




Actual results:

When I use 
jq '.[0]|.Digest,.RepoDigests'
 to extract the Digest and RepoDigest fields for the image
you can see that sha256:5e50822dc262ddcec0ce65fd1dee14f61398047b95b5dfc438a3043b54d2f93d is the RepoDigest and not the Digest for the image

Expected results:

published in the catalogue as 
registry.redhat.io/openshift4/ose-elasticsearch-operator@sha256:4b790902039f3ea801643011f807d7fdc20c1bf014c923351f5eebba46b3d327
Additional info:

Comment 1 Anping Li 2020-02-08 05:49:34 UTC
Operator Registry start to RepoDigest as image url from 4.3 to support  multiple-artchuture. Could you check if your podman support multiple-artchuture. I can pull the image_url attached above using Docker.

Comment 2 Sushil 2020-02-10 04:18:54 UTC
I am not sure why you are asking me if, I can pull the image using podman?

If you look at the example quoted, i can clearly pull the image using podman.
However the issue is. once I mirror this image to a docker registry, i am not able to pull this image from the mirror'd docker registry 


Going to try explaining the issue one more time.

In the 4.2 stable channel the image published for ose-elasticsearch-operator is
registry.redhat.io/openshift4/ose-elasticsearch-operator@sha256:d94eebefb44bed0bcaab45fd3e59554c7d44caf37a4c57e9cd340a00dd2d490d

I attempt to mirror this image using the `oc image mirror` command to my local docker registry. and then pull the image using podman.

Everything works as expected. (notsure if the below formatting works or not)

```
export MIRROR_REGISTRY=helper.ocp4.example.com:5000
export NAMESPACE=olmtest-take2


[sushil@rhltop ~]$ oc image mirror registry.redhat.io/openshift4/ose-elasticsearch-operator@sha256:d94eebefb44bed0bcaab45fd3e59554c7d44caf37a4c57e9cd340a00dd2d490d ${MIRROR_REGISTRY}/${NAMESPACE}/ose-elasticsearch-operator
helper.ocp4.example.com:5000/
  olmtest-take2/ose-elasticsearch-operator
    blobs:
      registry.redhat.io/openshift4/ose-elasticsearch-operator sha256:4fbc3bafa3d4400bb97a733c1fe12f2f99bf38b9d5b913d5034f29798739654d 1.585KiB
      registry.redhat.io/openshift4/ose-elasticsearch-operator sha256:3d5c8986188005c86ab63feaa2c62a6dadccf0709f8e8596e9320d89d35748bd 5.176KiB
      registry.redhat.io/openshift4/ose-elasticsearch-operator sha256:4a21444dc61515c74392d648347471d1ebc8cd8ee35bb734dcfc9978bec41837 3.336MiB
      registry.redhat.io/openshift4/ose-elasticsearch-operator sha256:eb8b56a4e1384d8bf72a65f3953d4d3e9c2980191ec75ed41751f1b560e247b8 7.844MiB
      registry.redhat.io/openshift4/ose-elasticsearch-operator sha256:92b84f826d5dd4245465279ad2178fc1a7010209a721ccc753ab6fae91d55a3c 16.62MiB
      registry.redhat.io/openshift4/ose-elasticsearch-operator sha256:34971b2d1eb98e410c2802e6bb3a7f499f9c5bff1b870ed329089ecdb21cf856 72.71MiB
    manifests:
      sha256:d94eebefb44bed0bcaab45fd3e59554c7d44caf37a4c57e9cd340a00dd2d490d
  stats: shared=0 unique=6 size=100.5MiB ratio=1.00

phase 0:
  helper.ocp4.example.com:5000 olmtest-take2/ose-elasticsearch-operator blobs=6 mounts=0 manifests=1 shared=0

info: Planning completed in 7.33s
uploading: helper.ocp4.example.com:5000/olmtest-take2/ose-elasticsearch-operator sha256:4a21444dc61515c74392d648347471d1ebc8cd8ee35bb734dcfc9978bec41837 3.336MiB
uploading: helper.ocp4.example.com:5000/olmtest-take2/ose-elasticsearch-operator sha256:92b84f826d5dd4245465279ad2178fc1a7010209a721ccc753ab6fae91d55a3c 16.62MiB
uploading: helper.ocp4.example.com:5000/olmtest-take2/ose-elasticsearch-operator sha256:34971b2d1eb98e410c2802e6bb3a7f499f9c5bff1b870ed329089ecdb21cf856 72.71MiB
uploading: helper.ocp4.example.com:5000/olmtest-take2/ose-elasticsearch-operator sha256:eb8b56a4e1384d8bf72a65f3953d4d3e9c2980191ec75ed41751f1b560e247b8 7.844MiB
sha256:d94eebefb44bed0bcaab45fd3e59554c7d44caf37a4c57e9cd340a00dd2d490d helper.ocp4.example.com:5000/olmtest-take2/ose-elasticsearch-operator
info: Mirroring completed in 7.51s (14.02MB/s)
[sushil@rhltop ~]$ 
[sushil@rhltop ~]$ 
[sushil@rhltop ~]$ 
[sushil@rhltop ~]$ 
[sushil@rhltop ~]$ podman pull ${MIRROR_REGISTRY}/${NAMESPACE}/ose-elasticsearch-operator@sha256:d94eebefb44bed0bcaab45fd3e59554c7d44caf37a4c57e9cd340a00dd2d490d
             Trying to pull helper.ocp4.example.com:5000/olmtest-take2/ose-elasticsearch-operator@sha256:d94eebefb44bed0bcaab45fd3e59554c7d44caf37a4c57e9cd340a00dd2d490d...
Getting image source signatures
Copying blob 34971b2d1eb9 skipped: already exists  
Copying blob 4fbc3bafa3d4 skipped: already exists  
Copying blob 92b84f826d5d done  
Copying blob 4a21444dc615 done  
Copying blob eb8b56a4e138 done  
Copying config 3d5c898618 done  
Writing manifest to image destination
Storing signatures
3d5c8986188005c86ab63feaa2c62a6dadccf0709f8e8596e9320d89d35748bd
[sushil@rhltop ~]$ 
[sushil@rhltop ~]$ 

```

Now trying to do the same for the 4.3 stable channel the image published for ose-elasticsearch-operator
registry.redhat.io/openshift4/ose-elasticsearch-operator@sha256:5e50822dc262ddcec0ce65fd1dee14f61398047b95b5dfc438a3043b54d2f93d

```
export MIRROR_REGISTRY=helper.ocp4.example.com:5000
export NAMESPACE=olmtest-take3

[sushil@rhltop ~]$ oc image mirror registry.redhat.io/openshift4/ose-elasticsearch-operator@sha256:5e50822dc262ddcec0ce65fd1dee14f61398047b95b5dfc438a3043b54d2f93d ${MIRROR_REGISTRY}/${NAMESPACE}/ose-elasticsearch-operator
helper.ocp4.example.com:5000/
  olmtest-take3/ose-elasticsearch-operator
    blobs:
      registry.redhat.io/openshift4/ose-elasticsearch-operator sha256:4fbc3bafa3d4400bb97a733c1fe12f2f99bf38b9d5b913d5034f29798739654d 1.585KiB
      registry.redhat.io/openshift4/ose-elasticsearch-operator sha256:b045933af4f7a0883f1a0366c2377a03b94a77bba24e7c15b4b4171dbfdb5638 5.171KiB
      registry.redhat.io/openshift4/ose-elasticsearch-operator sha256:8d4203411daf0c5f64ba4820c5cfca11c859871429f7ca8b02eebf810962ea7d 3.336MiB
      registry.redhat.io/openshift4/ose-elasticsearch-operator sha256:e232ae6317c2547a0c0941579360e4600a08bdd3398fc22ad2160e187256e94f 7.842MiB
      registry.redhat.io/openshift4/ose-elasticsearch-operator sha256:c53747ad4b973dea6b6c85e215eadb4a7a1a16d530c87e9db3f31a252edc8f99 18.51MiB
      registry.redhat.io/openshift4/ose-elasticsearch-operator sha256:34971b2d1eb98e410c2802e6bb3a7f499f9c5bff1b870ed329089ecdb21cf856 72.71MiB
    manifests:
      sha256:4b790902039f3ea801643011f807d7fdc20c1bf014c923351f5eebba46b3d327
  stats: shared=0 unique=6 size=102.4MiB ratio=1.00

phase 0:
  helper.ocp4.example.com:5000 olmtest-take3/ose-elasticsearch-operator blobs=6 mounts=0 manifests=1 shared=0

info: Planning completed in 7.29s
uploading: helper.ocp4.example.com:5000/olmtest-take3/ose-elasticsearch-operator sha256:c53747ad4b973dea6b6c85e215eadb4a7a1a16d530c87e9db3f31a252edc8f99 18.51MiB
uploading: helper.ocp4.example.com:5000/olmtest-take3/ose-elasticsearch-operator sha256:8d4203411daf0c5f64ba4820c5cfca11c859871429f7ca8b02eebf810962ea7d 3.336MiB
uploading: helper.ocp4.example.com:5000/olmtest-take3/ose-elasticsearch-operator sha256:e232ae6317c2547a0c0941579360e4600a08bdd3398fc22ad2160e187256e94f 7.842MiB
uploading: helper.ocp4.example.com:5000/olmtest-take3/ose-elasticsearch-operator sha256:34971b2d1eb98e410c2802e6bb3a7f499f9c5bff1b870ed329089ecdb21cf856 72.71MiB
sha256:4b790902039f3ea801643011f807d7fdc20c1bf014c923351f5eebba46b3d327 helper.ocp4.example.com:5000/olmtest-take3/ose-elasticsearch-operator
info: Mirroring completed in 7s (15.32MB/s)
[sushil@rhltop ~]$ 
[sushil@rhltop ~]$ 
[sushil@rhltop ~]$ podman pull ${MIRROR_REGISTRY}/${NAMESPACE}/ose-elasticsearch-operator@sha256:5e50822dc262ddcec0ce65fd1dee14f61398047b95b5dfc438a3043b54d2f93d
             Trying to pull helper.ocp4.example.com:5000/olmtest-take3/ose-elasticsearch-operator@sha256:5e50822dc262ddcec0ce65fd1dee14f61398047b95b5dfc438a3043b54d2f93d...
  manifest unknown: manifest unknown
Error: error pulling image "helper.ocp4.example.com:5000/olmtest-take3/ose-elasticsearch-operator@sha256:5e50822dc262ddcec0ce65fd1dee14f61398047b95b5dfc438a3043b54d2f93d": unable to pull helper.ocp4.example.com:5000/olmtest-take3/ose-elasticsearch-operator@sha256:5e50822dc262ddcec0ce65fd1dee14f61398047b95b5dfc438a3043b54d2f93d: unable to pull image: Error initializing source docker://helper.ocp4.example.com:5000/olmtest-take3/ose-elasticsearch-operator@sha256:5e50822dc262ddcec0ce65fd1dee14f61398047b95b5dfc438a3043b54d2f93d: Error reading manifest sha256:5e50822dc262ddcec0ce65fd1dee14f61398047b95b5dfc438a3043b54d2f93d in helper.ocp4.example.com:5000/olmtest-take3/ose-elasticsearch-operator: manifest unknown: manifest unknown
[sushil@rhltop ~]$ 

```

As a result I am not able to seutp a mirror'd repo and use it for setting up disconnected Operators. Because once mirrored the image reference for ose-elasticsearch-operator doesn't work.

First time updating a BZ, so not sure sure if the above formating works.
Dumping the above output into a gist as well, hopeing it will make it easier to read.
https://gist.github.com/sushilsuresh/e1b3ab9ba9be6dfc232611eb8145f41f

Comment 3 Qiaoling Tang 2020-02-10 07:03:46 UTC
From 4.3, all the 3rd level operators are changed to use manifest lists in the CSV files. 

Looks like your MIRROR_REGISTRY doesn't support manifest lists. And I guess the manifest of the ose-elasticsearch-operator in your MIRROR_REGISTRY is `4b790902039f3ea801643011f807d7fdc20c1bf014c923351f5eebba46b3d327`, not `5e50822dc262ddcec0ce65fd1dee14f61398047b95b5dfc438a3043b54d2f93d`. Maybe you could try to mirror the images to a registry that supports v2 schema2.

Comment 4 Sushil 2020-02-10 11:06:01 UTC
The version of my docker image registry is docker.io/library/registry:2.6.2

If I were to check https://docs.docker.com/registry/compatibility/
As far as I understand, Docker Registry v2.3 and above supports and defaults to schema2 format.

What am I missing?

Thank you.

--
Regards
Sushil Suresh.

Comment 5 Sushil 2020-02-10 14:00:16 UTC
I have tried replicating the same process using the latest docker image.

Below is the command I use to start the docker image registry

    podman run -td --name {{ registry_container_name }} -p 5000:5000            
    -v {{ registry_base_path }}/data:/var/lib/registry:z                        
    -v {{ registry_base_path }}/certs:/certs:z                                  
    -v {{ registry_base_path }}/auth:/auth:z                                    
    -e REGISTRY_AUTH=htpasswd                                                   
    -e REGISTRY_AUTH_HTPASSWD_REALM=Registry                                    
    -e REGISTRY_AUTH_HTPASSWD_PATH=/auth/htpasswd                               
    -e REGISTRY_HTTP_TLS_CERTIFICATE=/certs/domain.crt                          
    -e REGISTRY_HTTP_TLS_KEY=/certs/domain.key                                  
    docker.io/library/registry:latest     


I get the exact same error as before.
My understanding is that the latest docker registry images remove support for v2 schema1

So this clearly is not an issue with the docker registry not supporting v2 schema2.

Either the oc image mirror command is messing things up.

OR

The original publied image manifest is wrong.

Happy to be proved wrong. I am fairly new at this.
I just want to know what the root cause is and a being able to mirror the ose-elasticsearch-operator image in a disconnectd environment and have it working.

in case you are wondering, my laptop is running Fedora 31 and below are my podman and oc command versions.

[sushil@rhltop ocp4-upi-helpernode]$ podman version
             Version:            1.8.0
RemoteAPI Version:  1
Go Version:         go1.13.6
OS/Arch:            linux/amd64


[sushil@rhltop ocp4-upi-helpernode]$ oc version
             Client Version: 4.3.1


I would encourage you to setup a local docker image registry and try mirroring the below image and pulling the same from the mirror's registry. The suspect image in question is
registry.redhat.io/openshift4/ose-elasticsearch-operator@sha256:5e50822dc262ddcec0ce65fd1dee14f61398047b95b5dfc438a3043b54d2f93d


Thanks again.

--
Regards
Sushil Suresh.

Comment 6 Ben Parees 2020-02-10 21:48:12 UTC
i'm told you need to add "--filter-by-os=‘.*’" to your oc image mirror command to make this work, can you try that?

Comment 7 Sushil 2020-02-10 23:00:39 UTC
Nope, doesn't work either.

Exact same error - 

manifest unknown: manifest unknown

see Gist - https://gist.github.com/sushilsuresh/268ff30de35dd2825f1bf758b991c7e8

For your benifit, going to paste the console output below as welll 

-----------------------------------------------------------------------------------
[sushil@rhltop ~]$ 
[sushil@rhltop ~]$ export MIRROR_REGISTRY=helper.ocp4.example.com:5000
[sushil@rhltop ~]$ export NAMESPACE=olmtest-take1
[sushil@rhltop ~]$ 
[sushil@rhltop ~]$ 
[sushil@rhltop ~]$ oc image mirror --filter-by-os=.* registry.redhat.io/openshift4/ose-elasticsearch-operator@sha256:5e50822dc262ddcec0ce65fd1dee14f61398047b95b5dfc438a3043b54d2f93d ${MIRROR_REGISTRY}/${NAMESPACE}/ose-elasticsearch-operator
             helper.ocp4.example.com:5000/
  olmtest-take1/ose-elasticsearch-operator
    blobs:
      registry.redhat.io/openshift4/ose-elasticsearch-operator sha256:4fbc3bafa3d4400bb97a733c1fe12f2f99bf38b9d5b913d5034f29798739654d 1.585KiB
      registry.redhat.io/openshift4/ose-elasticsearch-operator sha256:b045933af4f7a0883f1a0366c2377a03b94a77bba24e7c15b4b4171dbfdb5638 5.171KiB
      registry.redhat.io/openshift4/ose-elasticsearch-operator sha256:8d4203411daf0c5f64ba4820c5cfca11c859871429f7ca8b02eebf810962ea7d 3.336MiB
      registry.redhat.io/openshift4/ose-elasticsearch-operator sha256:e232ae6317c2547a0c0941579360e4600a08bdd3398fc22ad2160e187256e94f 7.842MiB
      registry.redhat.io/openshift4/ose-elasticsearch-operator sha256:c53747ad4b973dea6b6c85e215eadb4a7a1a16d530c87e9db3f31a252edc8f99 18.51MiB
      registry.redhat.io/openshift4/ose-elasticsearch-operator sha256:34971b2d1eb98e410c2802e6bb3a7f499f9c5bff1b870ed329089ecdb21cf856 72.71MiB
    manifests:
      sha256:4b790902039f3ea801643011f807d7fdc20c1bf014c923351f5eebba46b3d327
  stats: shared=0 unique=6 size=102.4MiB ratio=1.00

phase 0:
  helper.ocp4.example.com:5000 olmtest-take1/ose-elasticsearch-operator blobs=6 mounts=0 manifests=1 shared=0

info: Planning completed in 3.32s
uploading: helper.ocp4.example.com:5000/olmtest-take1/ose-elasticsearch-operator sha256:e232ae6317c2547a0c0941579360e4600a08bdd3398fc22ad2160e187256e94f 7.842MiB
uploading: helper.ocp4.example.com:5000/olmtest-take1/ose-elasticsearch-operator sha256:34971b2d1eb98e410c2802e6bb3a7f499f9c5bff1b870ed329089ecdb21cf856 72.71MiB
uploading: helper.ocp4.example.com:5000/olmtest-take1/ose-elasticsearch-operator sha256:8d4203411daf0c5f64ba4820c5cfca11c859871429f7ca8b02eebf810962ea7d 3.336MiB
uploading: helper.ocp4.example.com:5000/olmtest-take1/ose-elasticsearch-operator sha256:c53747ad4b973dea6b6c85e215eadb4a7a1a16d530c87e9db3f31a252edc8f99 18.51MiB
sha256:4b790902039f3ea801643011f807d7fdc20c1bf014c923351f5eebba46b3d327 helper.ocp4.example.com:5000/olmtest-take1/ose-elasticsearch-operator
info: Mirroring completed in 21.07s (5.096MB/s)
[sushil@rhltop ~]$ 
[sushil@rhltop ~]$ 
[sushil@rhltop ~]$ 
[sushil@rhltop ~]$ podman pull ${MIRROR_REGISTRY}/${NAMESPACE}/ose-elasticsearch-operator@sha256:5e50822dc262ddcec0ce65fd1dee14f61398047b95b5dfc438a3043b54d2f93d
             Trying to pull helper.ocp4.example.com:5000/olmtest-take1/ose-elasticsearch-operator@sha256:5e50822dc262ddcec0ce65fd1dee14f61398047b95b5dfc438a3043b54d2f93d...
  manifest unknown: manifest unknown
Error: error pulling image "helper.ocp4.example.com:5000/olmtest-take1/ose-elasticsearch-operator@sha256:5e50822dc262ddcec0ce65fd1dee14f61398047b95b5dfc438a3043b54d2f93d": unable to pull helper.ocp4.example.com:5000/olmtest-take1/ose-elasticsearch-operator@sha256:5e50822dc262ddcec0ce65fd1dee14f61398047b95b5dfc438a3043b54d2f93d: unable to pull image: Error initializing source docker://helper.ocp4.example.com:5000/olmtest-take1/ose-elasticsearch-operator@sha256:5e50822dc262ddcec0ce65fd1dee14f61398047b95b5dfc438a3043b54d2f93d: Error reading manifest sha256:5e50822dc262ddcec0ce65fd1dee14f61398047b95b5dfc438a3043b54d2f93d in helper.ocp4.example.com:5000/olmtest-take1/ose-elasticsearch-operator: manifest unknown: manifest unknown
[sushil@rhltop ~]$ 
-----------------------------------------------------------------------------------

What next ?

Thank you.
--
Regards
Sushil Suresh.

Comment 8 Ben Parees 2020-02-11 15:31:20 UTC
Thanks to Shawn Hurley for digging into this a bit, there appears to be a bug(well, intentional decision but one that needs to be revisited) in oc image mirror in which if the manifestlist being mirrored has only a single manifest in it, oc image mirror only mirrors the single manifest instead of the manifestlist:

https://github.com/openshift/oc/blob/680ad4847f7de413f5fa28ed31bd59fd4ba4af10/pkg/cli/image/manifest/manifest.go#L382-L388

So that's causing the behavior seen here even when --filter-by-os=.* is being passed.

The workaround in the short term is to use "skopeo copy --all" to mirror the content instead, until this bug is resolved such that oc image mirror can do the right thing (possibly by introducing a new flag that allows you to choose whether you want the existing behavior or the full manifestlist mirroring, in order to preserve backward compatibility).


Note that there is also a related bug in "oc adm catalog mirror" which prevents it from doing the right thing in these scenarios, i'll update this bug when i can track that down.

Comment 9 Sushil 2020-02-12 13:50:07 UTC
Finally i have it working. Yay !!!

So as suggested I attempted to use skopeo copy --all
To mirror the image and, I do not have an issue pulling down the image.


So would you please mind referencing the bugs for both the

oc image mirror

and 

od adm catalog mirror

commands.

Thanks again. for helping me get to the bottom of this.
Appreciate the help.

Gist for readabiliy
https://gist.github.com/sushilsuresh/c792fa207ecf94192a7005aeb2d222bb


Pasting the output below as well.

---------------------------------------------------------------------------------------
[sushil@rhltop ocp4-ansible-roles]$                                                                                                                                                                                                           
[sushil@rhltop ocp4-ansible-roles]$ export MIRROR_REGISTRY=helper.ocp4.example.com:5000                                                                                                                                                       
[sushil@rhltop ocp4-ansible-roles]$ export NAMESPACE=olmtest-take1                                                                                                                                                                                       6]
[sushil@rhltop ocp4-ansible-roles]$                                                                                                                                                                                                         6]
[sushil@rhltop ocp4-ansible-roles]$                                                                                                                                                                                                           
[sushil@rhltop ocp4-ansible-roles]$ skopeo copy --all docker://registry.redhat.io/openshift4/ose-elasticsearch-operator@sha256:5e50822dc262ddcec0ce65fd1dee14f61398047b95b5dfc438a3043b54d2f93d docker://${MIRROR_REGISTRY}/${NAMESPACE}/ose-e
lasticsearch-operator                                                                                                                                                                                                                         
Getting image list signatures                                                                                                                                                                                                               6]
Copying 1 of 1 images in list                                                                                                                                                                                                                 
Copying image sha256:4b790902039f3ea801643011f807d7fdc20c1bf014c923351f5eebba46b3d327 (1/1)                                                                                                                                                   
Getting image source signatures                                                                                                                                                                                                               
Copying blob e232ae6317c2 done                                                                                                                                                                                                                
Copying blob c53747ad4b97 done                                                                                                                                                                                                                
Copying blob 34971b2d1eb9 done                                                                                                                                                                                                                
Copying blob 8d4203411daf done                                                                                                                                                                                                                
Copying blob 4fbc3bafa3d4 done                                                                                                                                                                                                                
Copying config b045933af4 done                                                                                                                                                                                                                
Writing manifest to image destination                                                                                                                                                                                                         
Storing signatures                                                                                                                                                                                                                            
Writing manifest list to image destination                                                                                                                                                                                                    
Storing list signatures                                                                                                                                                                                                                       
[sushil@rhltop ocp4-ansible-roles]$                                                                                                                                                                                                           
[sushil@rhltop ocp4-ansible-roles]$                                                                                                                                                                                                           
[sushil@rhltop ocp4-ansible-roles]$                                                              


------------------------------------------

[root@rhltop ~]# 
[root@rhltop ~]# export MIRROR_REGISTRY=helper.ocp4.example.com:5000
[root@rhltop ~]# export NAMESPACE=olmtest-take1
[root@rhltop ~]# 
[root@rhltop ~]# 
[root@rhltop ~]# podman pull ${MIRROR_REGISTRY}/${NAMESPACE}/ose-elasticsearch-operator@sha256:5e50822dc262ddcec0ce65fd1dee14f61398047b95b5dfc438a3043b54d2f93d
Trying to pull helper.ocp4.example.com:5000/olmtest-take1/ose-elasticsearch-operator@sha256:5e50822dc262ddcec0ce65fd1dee14f61398047b95b5dfc438a3043b54d2f93d...
Getting image source signatures
Copying blob 4fbc3bafa3d4 done  
Copying blob c53747ad4b97 done  
Copying blob 34971b2d1eb9 done  
Copying blob e232ae6317c2 done  
Copying blob 8d4203411daf done  
Copying config b045933af4 done  
Writing manifest to image destination
Storing signatures
b045933af4f7a0883f1a0366c2377a03b94a77bba24e7c15b4b4171dbfdb5638
[root@rhltop ~]# 

---------------------------------------------------------------------------------------

Comment 10 Shawn Hurley 2020-02-12 14:18:24 UTC
This is the 4.3 bugs for the `oc adm catalog mirror` issue with mirroring manifest lists.

https://bugzilla.redhat.com/show_bug.cgi?id=1797026

Comment 11 Ben Parees 2020-02-12 15:11:29 UTC
and this bug itself is the one for the oc image mirror behavior (that it doesn't copy the manifestlist if the manifestlist only contains a single manifest).  I retitled it accordingly.

Comment 12 Sushil 2020-02-12 15:26:03 UTC
Sweet

Thank you sir.

Will track this.

--
Regards
Sushil Suresh.

Comment 13 Anshul Verma 2020-02-14 02:31:50 UTC
(In reply to Ben Parees from comment #8)
> Thanks to Shawn Hurley for digging into this a bit, there appears to be a
> bug(well, intentional decision but one that needs to be revisited) in oc
> image mirror in which if the manifestlist being mirrored has only a single
> manifest in it, oc image mirror only mirrors the single manifest instead of
> the manifestlist:
> 
> https://github.com/openshift/oc/blob/
> 680ad4847f7de413f5fa28ed31bd59fd4ba4af10/pkg/cli/image/manifest/manifest.
> go#L382-L388
> 
> So that's causing the behavior seen here even when --filter-by-os=.* is
> being passed.
> 
> The workaround in the short term is to use "skopeo copy --all" to mirror the
> content instead, until this bug is resolved such that oc image mirror can do
> the right thing (possibly by introducing a new flag that allows you to
> choose whether you want the existing behavior or the full manifestlist
> mirroring, in order to preserve backward compatibility).
> 
> 
> Note that there is also a related bug in "oc adm catalog mirror" which
> prevents it from doing the right thing in these scenarios, i'll update this
> bug when i can track that down.

@Ben 
Actually '--all' option is not there in 'skopeo-0.1.37-3.el7.x86_64' and this is the latest version in RHEL 7.7 , is there any other workaround which we can use in RHEL 7.7 which can only update to 'skopeo-0.1.37-3.el7.x86_64' or maybe if any repo present which has the latest official versin of skopeo which is 0.1.40.

Comment 18 Maciej Szulik 2020-02-14 16:02:07 UTC
Spoke too soon, I'm moving this to OLM team to decide about how they are planning to handle this.

Comment 19 Evan Cordell 2020-02-17 14:53:33 UTC
*** Bug 1797203 has been marked as a duplicate of this bug. ***

Comment 32 akamarathunga 2020-06-01 01:40:43 UTC
I have encountered the same error where OC cli change digest of the mirrored image. The skopeo workaround helped. However, it'll be better if we can use OC cli. Any ETA on a fix?

Comment 33 Michael Riedmann 2020-06-23 18:02:49 UTC
We also hit that with a customer in a fully air-gapped scenario (Openshift 4.4.7, OC 4.4.7) during the attempt to mirror the OLM catalog to an internal registry.
The Skopeo workaround worked, but we also observed an interesting fact: Focus on the last lines of the snippet below. 2 of 3 manifests are copied but the first is repeated and the one is missing. The missing hash is always the one that is referred to as the digest. Maybe that's a clue about what is going on. Also, only images with 3 manifests are affected, most other images only have 2, which works flawlessly. Please fix this issue, air-gapped installations are hard enough and bugs like this do raise concerns about the quality and stability of Openshift as an enterprise application ecosystem.

---

oc image mirror -a pull-secret.txt --keep-manifest-list=true registry.redhat.io/openshift4/ose-elasticsearch-operator@sha256:882fff755fb513b7af659e5b47643bffe9d2ce9d492e208bbc4aa9b390570310=mirror.lab.example.com:5000/openshift4/ose-elasticsearch-operator
mirror.lab.example.com:5000/
  openshift4/ose-elasticsearch-operator
    blobs:
      registry.redhat.io/openshift4/ose-elasticsearch-operator sha256:82a8f4ea76cb6f833c5f179b3e6eda9f2267ed8ac7d1bf652f88ac3e9cc453d1 1.561KiB
      registry.redhat.io/openshift4/ose-elasticsearch-operator sha256:5caacc3f346e3cae733746c1d97e5b259333610acb9977953c8193646d61b65c 5.521KiB
      registry.redhat.io/openshift4/ose-elasticsearch-operator sha256:5808abd2c28ac0118bd858443443074e64a1819defecfaffd8b64f34e492fcf0 3.339MiB
      registry.redhat.io/openshift4/ose-elasticsearch-operator sha256:9f932d1a696e1cb4b35e1ac5e48b96a55a9b30abdbd11d312fd6e52ec5ead728 7.858MiB
      registry.redhat.io/openshift4/ose-elasticsearch-operator sha256:68b2a20998ac18623266c87e0ad6297e00a080f5b54c05de5c49738901f81fb1 17.48MiB
      registry.redhat.io/openshift4/ose-elasticsearch-operator sha256:a3ac36470b00df382448e79f7a749aa6833e4ac9cc90e3391f778820db9fa407 72.74MiB
    manifests:
      sha256:79238e45bffc48b49c69ece3e28a205c25e93e717665341cf7f1a8d6fe4e9424
      sha256:882fff755fb513b7af659e5b47643bffe9d2ce9d492e208bbc4aa9b390570310
      sha256:f142729ac4b87b219cf0871f91d736de96bf5c885e7ebe2c7caf876693e2614f
  stats: shared=0 unique=6 size=101.4MiB ratio=1.00

phase 0:
  mirror.lab.example.com:5000 openshift4/ose-elasticsearch-operator blobs=6 mounts=0 manifests=3 shared=0

info: Planning completed in 1.89s
sha256:79238e45bffc48b49c69ece3e28a205c25e93e717665341cf7f1a8d6fe4e9424 mirror.lab.example.com:5000/openshift4/ose-elasticsearch-operator       
sha256:f142729ac4b87b219cf0871f91d736de96bf5c885e7ebe2c7caf876693e2614f mirror.lab.example.com:5000/openshift4/ose-elasticsearch-operator       
sha256:79238e45bffc48b49c69ece3e28a205c25e93e717665341cf7f1a8d6fe4e9424 mirror.lab.example.com:5000/openshift4/ose-elasticsearch-operator
info: Mirroring completed in 430ms (0B/s)

Comment 35 errata-xmlrpc 2020-07-13 17:14:25 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:2409

Comment 40 Maciej Szulik 2020-08-20 09:48:06 UTC
The 4.4 fix

Comment 41 W. Trevor King 2020-08-20 19:16:21 UTC
Changing a CLOSED ERRATA bug back to NEW is not great, and will make the shipped 4.5.1 errata very confusing.  Can we move this bug back to CLOSED ERRATA and create new bugs for any remaining issues?

Comment 42 Kevin Rizza 2020-08-21 11:28:39 UTC
Selim,

I think the issue your command has is that you are using `--filter-by-os='linux/amd64`, to get the mirror to work you will need to use `--filter-by-os=.*`

oc adm catalog mirror registry.home:7000/mirror/redhat-operators:test mirror.registry --filter-by-os='.*' --to-manifests=/var/lib/import/olm/redhat-operators/ --path=/:/var/lib/import/olm/redhat-operators/image

Operator manifests reference images that are built with manifestlists, and in the background cri-o will pull the underlying image with the correct architecture.

Marking this back as CLOSED ERRATA. If there is indeed an issue after this, feel free to open another bug and we can triage from there.


Note You need to log in before you can comment on or make changes to this bug.