Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

Bug 1884693

Summary: oc image mirror missing Manifest List sha256
Product: OpenShift Container Platform Reporter: Muhammad Aizuddin Zali <mzali>
Component: ocAssignee: Maciej Szulik <maszulik>
Status: CLOSED DUPLICATE QA Contact: zhou ying <yinzhou>
Severity: medium Docs Contact:
Priority: unspecified    
Version: 4.5CC: aos-bugs, jokerman, mfojtik, mzali
Target Milestone: ---   
Target Release: ---   
Hardware: All   
OS: All   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2020-10-05 08:14:19 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description Muhammad Aizuddin Zali 2020-10-02 17:09:34 UTC 
Description of problem:
`oc image mirror` does not mirror Manifest List. As such for restricted OLM, operator pod with "Error reading manifest" logged.

Version-Release number of selected component (if applicable):
#  oc version
Client Version: 4.5.13
Server Version: 4.5.6
Kubernetes Version: v1.18.3+002a51f


How reproducible:
Always

Steps to Reproduce:
1. Run mirroring command:
# cat mapping.txt 
registry.redhat.io/openshift4/ose-local-storage-operator@sha256:06991e3dded419e7f976561d97ba40aad1cdaabb3966c1dce3d5577c9a1d4e3a=registry.local.bytewise.my:5000/openshift4/ose-local-storage-operator:844ec5b6

# oc image mirror -a /opt/registry/pull-secret.json -f mapping.txt  --insecure  --insecure --force
registry.local.bytewise.my:5000/
  openshift4/ose-local-storage-operator
    blobs:
      registry.redhat.io/openshift4/ose-local-storage-operator sha256:74cbb6607642df5f9f70e8588e3c56d6de795d1a9af22866ea4cc82f2dad4f14 1.68KiB
      registry.redhat.io/openshift4/ose-local-storage-operator sha256:3559c47a79020fca933df438e1288c7ba7d0d21e03e61b8b1d4c3fe751ba5e3d 5.361KiB
      registry.redhat.io/openshift4/ose-local-storage-operator sha256:9ebdc795180d997ec9611acb8a453717f63fa2227b003db6c7cc911482e3e43d 3.347MiB
      registry.redhat.io/openshift4/ose-local-storage-operator sha256:4406a9beeab462c05c8e03f0c6eccd2902cb4cde00caee21cf13ff6dfaab53f4 9.145MiB
      registry.redhat.io/openshift4/ose-local-storage-operator sha256:e89199065ebee8c2d04e516c5b1db25f74b5e532166ac18d8e02e7b5c5d40399 16.16MiB
      registry.redhat.io/openshift4/ose-local-storage-operator sha256:c9fa7d57b9028d4bd02b51cef3c3039fa7b23a8b2d9d26a6ce66b3428f6e2457 72.71MiB
    manifests:
      sha256:465675042abb7922dd18386b6f1b13cc4d9f24edf67a36a436980e67a8ff5392 -> 844ec5b6
  stats: shared=0 unique=6 size=101.4MiB ratio=1.00

phase 0:
  registry.local.bytewise.my:5000 openshift4/ose-local-storage-operator blobs=6 mounts=0 manifests=1 shared=0

info: Planning completed in 2.33s
uploading: registry.local.bytewise.my:5000/openshift4/ose-local-storage-operator sha256:4406a9beeab462c05c8e03f0c6eccd2902cb4cde00caee21cf13ff6dfaab53f4 9.145MiB
uploading: registry.local.bytewise.my:5000/openshift4/ose-local-storage-operator sha256:9ebdc795180d997ec9611acb8a453717f63fa2227b003db6c7cc911482e3e43d 3.347MiB
uploading: registry.local.bytewise.my:5000/openshift4/ose-local-storage-operator sha256:c9fa7d57b9028d4bd02b51cef3c3039fa7b23a8b2d9d26a6ce66b3428f6e2457 72.71MiB
uploading: registry.local.bytewise.my:5000/openshift4/ose-local-storage-operator sha256:e89199065ebee8c2d04e516c5b1db25f74b5e532166ac18d8e02e7b5c5d40399 16.16MiB
sha256:465675042abb7922dd18386b6f1b13cc4d9f24edf67a36a436980e67a8ff5392 registry.local.bytewise.my:5000/openshift4/ose-local-storage-operator:844ec5b6
info: Mirroring completed in 3.79s (28.01MB/s)


2. Verify image info

[root@bastion redhat-operators-manifests-old]# oc image info -a /opt/registry/pull-secret.json registry.local.bytewise.my:5000/openshift4/ose-local-storage-operator:844ec5b6 --insecure  
Name:        registry.local.bytewise.my:5000/openshift4/ose-local-storage-operator:844ec5b6
Digest:      sha256:465675042abb7922dd18386b6f1b13cc4d9f24edf67a36a436980e67a8ff5392
Media Type:  application/vnd.docker.distribution.manifest.v2+json
Created:     27d ago
Image Size:  106.3MB in 5 layers
Layers:      76.25MB sha256:c9fa7d57b9028d4bd02b51cef3c3039fa7b23a8b2d9d26a6ce66b3428f6e2457
             1.72kB  sha256:74cbb6607642df5f9f70e8588e3c56d6de795d1a9af22866ea4cc82f2dad4f14
             3.509MB sha256:9ebdc795180d997ec9611acb8a453717f63fa2227b003db6c7cc911482e3e43d
             9.59MB  sha256:4406a9beeab462c05c8e03f0c6eccd2902cb4cde00caee21cf13ff6dfaab53f4
             16.94MB sha256:e89199065ebee8c2d04e516c5b1db25f74b5e532166ac18d8e02e7b5c5d40399
OS:          linux
Arch:        amd64
Entrypoint:  /usr/bin/local-storage-operator
User:        0
Environment: __doozer=merge
             BUILD_RELEASE=202009042055.p0
             BUILD_VERSION=v4.5.0
             OS_GIT_MAJOR=4
             OS_GIT_MINOR=5
             OS_GIT_PATCH=0
             OS_GIT_TREE_STATE=clean
             OS_GIT_VERSION=4.5.0-202009042055.p0-5f424ba
             SOURCE_GIT_TREE_STATE=clean
             OS_GIT_COMMIT=5f424ba
             SOURCE_DATE_EPOCH=1599245670
             SOURCE_GIT_COMMIT=5f424ba9d0fb147616eaf5740ff73c9cc53ebb9c
             SOURCE_GIT_TAG=5f424ba
             SOURCE_GIT_URL=https://github.com/openshift/local-storage-operator
             PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
             container=oci
Labels:      License=GPLv2+
             architecture=x86_64
             build-date=2020-09-04T21:18:10.860156
             com.redhat.build-host=cpt-1005.osbs.prod.upshift.rdu2.redhat.com
             com.redhat.component=local-storage-operator-container
             com.redhat.delivery.appregistry=false
             com.redhat.license_terms=https://www.redhat.com/agreements
             description=This is a component of OpenShift and manages local volumes.
             distribution-scope=public
             io.k8s.description=This is a component of OpenShift and manages local volumes.
             io.k8s.display-name=OpenShift local-storage-operator
             io.openshift.build.commit.id=5f424ba9d0fb147616eaf5740ff73c9cc53ebb9c
             io.openshift.build.commit.url=https://github.com/openshift/local-storage-operator/commit/5f424ba9d0fb147616eaf5740ff73c9cc53ebb9c
             io.openshift.build.source-location=https://github.com/openshift/local-storage-operator
             io.openshift.maintainer.component=Storage
             io.openshift.maintainer.product=OpenShift Container Platform
             io.openshift.tags=openshift,base
             maintainer=Hemant Kumar <hekumar>
             name=openshift/ose-local-storage-operator
             release=202009042055.p0
             summary=Provides the latest release of the Red Hat Universal Base Image 7.
             url=https://access.redhat.com/containers/#/registry.access.redhat.com/openshift/ose-local-storage-operator/images/v4.5.0-202009042055.p0
             vcs-ref=2d0ab1047253a30d2c6bff9f4754bb10eae8c8c4
             vcs-type=git
             vendor=Red Hat, Inc.
             version=v4.5.0



3. Inpect operator pod deployment with error:
Failed to pull image "registry.redhat.io/openshift4/ose-local-storage-operator@sha256:06991e3dded419e7f976561d97ba40aad1cdaabb3966c1dce3d5577c9a1d4e3a": rpc error: code = Unknown desc = (Mirrors also failed: [registry.local.bytewise.my:5000/openshift4/ose-local-storage-operator@sha256:06991e3dded419e7f976561d97ba40aad1cdaabb3966c1dce3d5577c9a1d4e3a: Error reading manifest sha256:06991e3dded419e7f976561d97ba40aad1cdaabb3966c1dce3d5577c9a1d4e3a in registry.local.bytewise.my:5000/openshift4/ose-local-storage-operator: manifest unknown: manifest unknown]): registry.redhat.io/openshift4/ose-local-storage-operator@sha256:06991e3dded419e7f976561d97ba40aad1cdaabb3966c1dce3d5577c9a1d4e3a: unable to retrieve auth token: invalid username/password: unauthorized: Please login to the Red Hat Registry using your Customer Portal credentials. Further instructions can be found here: https://access.redhat.com/RegistryAuthentication

4. To resolved this we need to use skopeo copy:

# skopeo copy -a docker://registry.redhat.io/openshift4/ose-local-storage-operator@sha256:06991e3dded419e7f976561d97ba40aad1cdaabb3966c1dce3d5577c9a1d4e3a docker://registry.local.bytewise.my:5000/openshift4/ose-local-storage-operator:844ec5b6 --authfile=/opt/registry/pull-secret.json --dest-tls-verify=false
Getting image list signatures
Copying 3 of 3 images in list
Copying image sha256:465675042abb7922dd18386b6f1b13cc4d9f24edf67a36a436980e67a8ff5392 (1/3)
Getting image source signatures
Copying blob c9fa7d57b902 skipped: already exists  
Copying blob 74cbb6607642 skipped: already exists  
Copying blob 9ebdc795180d skipped: already exists  
Copying blob 4406a9beeab4 skipped: already exists  
Copying blob e89199065ebe [--------------------------------------] 0.0b / 0.0b
Writing manifest to image destination
Storing signatures
Copying image sha256:7e653affb84e8d8576896519ee73be046bbb697dbc8bdfa72a7290e9fb7200a1 (2/3)
Getting image source signatures
Copying blob f9e75f9bb0ee done  
Copying blob 1d2139ac6200 done  
Copying blob 035969daab35 done  
Copying blob 39bc543eae60 done  
Copying blob ea743b6b7299 done  
Copying config 0db3132ea4 done  
Writing manifest to image destination
Storing signatures
Copying image sha256:78d42ab1d104377681ccceb0a742385473b8cb5279d3c70b9a53a3651a9332a8 (3/3)
Getting image source signatures
Copying blob c4579cf2e782 done  
Copying blob 2a0d867c6ee2 done  
Copying blob 5381955e35b0 done  
Copying blob a5187e0ae00a done  
Copying blob a27e2f5b7047 done  
Copying config 31349a8dfa done  
Writing manifest to image destination
Storing signatures
Writing manifest list to image destination
Storing list signatures


5. Verify image info that mirrored with skopeo copy command is having Manifest List digest:

# oc image info -a /opt/registry/pull-secret.json registry.local.bytewise.my:5000/openshift4/ose-local-storage-operator:844ec5b6 --insecure  --filter-by-os=linux/amd64
Name:          registry.local.bytewise.my:5000/openshift4/ose-local-storage-operator:844ec5b6
Digest:        sha256:465675042abb7922dd18386b6f1b13cc4d9f24edf67a36a436980e67a8ff5392
Manifest List: sha256:06991e3dded419e7f976561d97ba40aad1cdaabb3966c1dce3d5577c9a1d4e3a
Media Type:    application/vnd.docker.distribution.manifest.v2+json
Created:       27d ago
Image Size:    106.3MB in 5 layers
Layers:        76.25MB sha256:c9fa7d57b9028d4bd02b51cef3c3039fa7b23a8b2d9d26a6ce66b3428f6e2457
               1.72kB  sha256:74cbb6607642df5f9f70e8588e3c56d6de795d1a9af22866ea4cc82f2dad4f14
               3.509MB sha256:9ebdc795180d997ec9611acb8a453717f63fa2227b003db6c7cc911482e3e43d
               9.59MB  sha256:4406a9beeab462c05c8e03f0c6eccd2902cb4cde00caee21cf13ff6dfaab53f4
               16.94MB sha256:e89199065ebee8c2d04e516c5b1db25f74b5e532166ac18d8e02e7b5c5d40399
OS:            linux
Arch:          amd64
Entrypoint:    /usr/bin/local-storage-operator
User:          0
Environment:   __doozer=merge
               BUILD_RELEASE=202009042055.p0
               BUILD_VERSION=v4.5.0
               OS_GIT_MAJOR=4
               OS_GIT_MINOR=5
               OS_GIT_PATCH=0
               OS_GIT_TREE_STATE=clean
               OS_GIT_VERSION=4.5.0-202009042055.p0-5f424ba
               SOURCE_GIT_TREE_STATE=clean
               OS_GIT_COMMIT=5f424ba
               SOURCE_DATE_EPOCH=1599245670
               SOURCE_GIT_COMMIT=5f424ba9d0fb147616eaf5740ff73c9cc53ebb9c
               SOURCE_GIT_TAG=5f424ba
               SOURCE_GIT_URL=https://github.com/openshift/local-storage-operator
               PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
               container=oci
Labels:        License=GPLv2+
               architecture=x86_64
               build-date=2020-09-04T21:18:10.860156
               com.redhat.build-host=cpt-1005.osbs.prod.upshift.rdu2.redhat.com
               com.redhat.component=local-storage-operator-container
               com.redhat.delivery.appregistry=false
               com.redhat.license_terms=https://www.redhat.com/agreements
               description=This is a component of OpenShift and manages local volumes.
               distribution-scope=public
               io.k8s.description=This is a component of OpenShift and manages local volumes.
               io.k8s.display-name=OpenShift local-storage-operator
               io.openshift.build.commit.id=5f424ba9d0fb147616eaf5740ff73c9cc53ebb9c
               io.openshift.build.commit.url=https://github.com/openshift/local-storage-operator/commit/5f424ba9d0fb147616eaf5740ff73c9cc53ebb9c
               io.openshift.build.source-location=https://github.com/openshift/local-storage-operator
               io.openshift.maintainer.component=Storage
               io.openshift.maintainer.product=OpenShift Container Platform
               io.openshift.tags=openshift,base
               maintainer=Hemant Kumar <hekumar>
               name=openshift/ose-local-storage-operator
               release=202009042055.p0
               summary=Provides the latest release of the Red Hat Universal Base Image 7.
               url=https://access.redhat.com/containers/#/registry.access.redhat.com/openshift/ose-local-storage-operator/images/v4.5.0-202009042055.p0
               vcs-ref=2d0ab1047253a30d2c6bff9f4754bb10eae8c8c4
               vcs-type=git
               vendor=Red Hat, Inc.
               version=v4.5.0

6. Now image able to be pulled and pod is running.
Successfully pulled image "registry.redhat.io/openshift4/ose-local-storage-operator@sha256:06991e3dded419e7f976561d97ba40aad1cdaabb3966c1dce3d5577c9a1d4e3a"

Actual results:
oc image mirror command missing Manifest List digest.

Expected results:
oc image mirror should mirror Manifest List digest.

Additional info:
Comment 1 Muhammad Aizuddin Zali 2020-10-05 05:58:45 UTC 
Seems adding `--filter-by-os=".*"` fixed the missing manifest list checksum. 


# oc image mirror -a /opt/registry/pull-secret.json -f logging.txt --insecure --force --filter-by-os=".*"


# oc image info -a /opt/registry/pull-secret.json registry.local.bytewise.my:5000/openshift4/ose-cluster-logging-operator:44164da --insecure --filter-by-os="linux/amd64" | head
Name:          registry.local.bytewise.my:5000/openshift4/ose-cluster-logging-operator:44164da
Digest:        sha256:512997105871d1121e6490b55b14795d35413cd3ef762f60d68354da18be2753
Manifest List: sha256:57fe7f01330e58b87ab925c068348e4d762a20f37f229410c548b2fc597cf3fe
Media Type:    application/vnd.docker.distribution.manifest.v2+json
Created:       30d ago
Image Size:    111.4MB in 5 layers
Layers:        76.25MB sha256:c9fa7d57b9028d4bd02b51cef3c3039fa7b23a8b2d9d26a6ce66b3428f6e2457
               1.72kB  sha256:74cbb6607642df5f9f70e8588e3c56d6de795d1a9af22866ea4cc82f2dad4f14
               3.509MB sha256:9ebdc795180d997ec9611acb8a453717f63fa2227b003db6c7cc911482e3e43d
               9.59MB  sha256:4406a9beeab462c05c8e03f0c6eccd2902cb4cde00caee21cf13ff6dfaab53f4
Comment 2 Maciej Szulik 2020-10-05 08:14:19 UTC 

*** This bug has been marked as a duplicate of bug 1883605 ***