Bug 1850931 - oc image mirror into localdisk will change the original sha256 in manifest schemaVersion1 images
Summary: oc image mirror into localdisk will change the original sha256 in manifest sc...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: oc
Version: 4.4
Hardware: x86_64
OS: Linux
medium
high
Target Milestone: ---
: 4.6.0
Assignee: Sally
QA Contact: zhou ying
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2020-06-25 08:27 UTC by kevin
Modified: 2020-10-27 16:10 UTC (History)
7 users (show)

Fixed In Version:
Doc Type: No Doc Update
Doc Text:
Clone Of:
Environment:
Last Closed: 2020-10-27 16:09:42 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Github openshift oc pull 493 0 None closed Bug 1850931: 'oc image mirror' Add warning when mirroring SchemaVersion:1 images 2021-02-18 18:51:08 UTC
Red Hat Product Errata RHBA-2020:4196 0 None None None 2020-10-27 16:10:26 UTC

Description kevin 2020-06-25 08:27:36 UTC
Description of problem:

for manifest schemaVersion1 images, use oc image mirror will change the original image sha256 value

Version-Release number of selected component (if applicable):

oc version
Client Version: 4.4.9


How reproducible:

Steps to Reproduce:

1.image manifest schema V1 use "oc image mirror" command mirror into local disk will change the original sha256 value!!!

now, I use operatorhub etcd operator image as example, this image is very important, because currently our ACM operator(Advanced Cluster Management rely on it !)

the image url is:
quay.io/coreos/etcd-operator@sha256:66a37fd61a06a43969854ee6d3e21087a98b93838e284a6086b13917f96b0d9b

(1) check the image's schema version
you can use the following command to check the image schema version number:

[COMMAND]
# skopeo inspect --raw docker://quay.io/coreos/etcd-operator@sha256:66a37fd61a06a43969854ee6d3e21087a98b93838e284a6086b13917f96b0d9b |jq -r .
[OUTPUT]
{
  "schemaVersion": 1,
...

You can see this image's schema version is 1

(2) mirror the images into local disk

I use following command to download the image into local disk 

oc image mirror <imageURL> --dir=<DIR> file://<fileURL>

[COMMAND]
# oc image mirror quay.io/coreos/etcd-operator@sha256:66a37fd61a06a43969854ee6d3e21087a98b93838e284a6086b13917f96b0d9b --dir=images file://coreos/etcd-operator
[OUTPUT]
<dir>
  coreos/etcd-operator
    blobs:
      quay.io/coreos/etcd-operator sha256:59265c40e257554058624f35856dafd82d135c4ef406de298cb1fee647867381
      quay.io/coreos/etcd-operator sha256:678f2cfebea627b8b3d5bed91a6fe1d3749421fc504a568a7977ce912a953763
      quay.io/coreos/etcd-operator sha256:6d4b4c91b3ff6970e22c704f59d45a5945065cb80bd7055114f6d27432e2205a
      quay.io/coreos/etcd-operator sha256:80b1b554e5b4c4a4334fd25764d6c3d78bafd04ef727714dfb25b1356cc3a265
      quay.io/coreos/etcd-operator sha256:8ea0315241b46a55ff650eddc8d06bfe7bfbdc072b4f3878d6c4598fc7b015d3
      quay.io/coreos/etcd-operator sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4
      quay.io/coreos/etcd-operator sha256:e05d0d7eb99a1d91fa7349dd9a98a8b2d9e222b39fa305543e1ac4c380147a4b
    manifests:
      sha256:66a37fd61a06a43969854ee6d3e21087a98b93838e284a6086b13917f96b0d9b
  stats: shared=0 unique=7 size=0B

phase 0:
   coreos/etcd-operator blobs=7 mounts=0 manifests=1 shared=0

info: Planning completed in 1.2s
uploading: file://coreos/etcd-operator sha256:59265c40e257554058624f35856dafd82d135c4ef406de298cb1fee647867381 0B
uploading: file://coreos/etcd-operator sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4 0B
uploading: file://coreos/etcd-operator sha256:6d4b4c91b3ff6970e22c704f59d45a5945065cb80bd7055114f6d27432e2205a 0B
uploading: file://coreos/etcd-operator sha256:80b1b554e5b4c4a4334fd25764d6c3d78bafd04ef727714dfb25b1356cc3a265 0B
uploading: file://coreos/etcd-operator sha256:e05d0d7eb99a1d91fa7349dd9a98a8b2d9e222b39fa305543e1ac4c380147a4b 0B
uploading: file://coreos/etcd-operator sha256:678f2cfebea627b8b3d5bed91a6fe1d3749421fc504a568a7977ce912a953763 0B
warning: Layer size mismatch for sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4: had 0, wrote 32
warning: Layer size mismatch for sha256:6d4b4c91b3ff6970e22c704f59d45a5945065cb80bd7055114f6d27432e2205a: had 0, wrote 1251
warning: Layer size mismatch for sha256:59265c40e257554058624f35856dafd82d135c4ef406de298cb1fee647867381: had 0, wrote 2016712
warning: Layer size mismatch for sha256:80b1b554e5b4c4a4334fd25764d6c3d78bafd04ef727714dfb25b1356cc3a265: had 0, wrote 15521633
uploading: file://coreos/etcd-operator sha256:8ea0315241b46a55ff650eddc8d06bfe7bfbdc072b4f3878d6c4598fc7b015d3 0B
warning: Layer size mismatch for sha256:e05d0d7eb99a1d91fa7349dd9a98a8b2d9e222b39fa305543e1ac4c380147a4b: had 0, wrote 17235178
warning: Layer size mismatch for sha256:678f2cfebea627b8b3d5bed91a6fe1d3749421fc504a568a7977ce912a953763: had 0, wrote 17555180
warning: Layer size mismatch for sha256:8ea0315241b46a55ff650eddc8d06bfe7bfbdc072b4f3878d6c4598fc7b015d3: had 0, wrote 351520
sha256:ac5496c52004b990f981ee419eb338921753c35db97e382496b21f15c7d23792 file://coreos/etcd-operator
info: Mirroring completed in 640ms (81.8MB/s)
--------------------------------------
!!!!!!Note: we can see the real sha256 have been changed into sha256:ac5496c52004b990f981ee419eb338921753c35db97e382496b21f15c7d23792 , this is totally different from original sha256:66a37fd61a06a43969854ee6d3e21087a98b93838e284a6086b13917f96b0d9b
--------------------------------------

(3) use the original sha256 check the downloaded image from local disk

I use following command to check the image into local disk 

oc image info --dir=<DIR> file://<fileURL>

[COMMAND]
# oc image info --dir=images file://coreos/etcd-operator@sha256:66a37fd61a06a43969854ee6d3e21087a98b93838e284a6086b13917f96b0d9b
[OUTPUT]
error: unable to read image file://coreos/etcd-operator@sha256:66a37fd61a06a43969854ee6d3e21087a98b93838e284a6086b13917f96b0d9b: open images/v2/coreos/etcd-operator/manifests/sha256:66a37fd61a06a43969854ee6d3e21087a98b93838e284a6086b13917f96b0d9b: no such file or directory

--------------------
You can see original sha256 have faild
--------------------

(4) check the real sha256 value from downloaded image of local disk 

[COMMAND]
# oc image info --dir=images file://coreos/etcd-operator@sha256:ac5496c52004b990f981ee419eb338921753c35db97e382496b21f15c7d23792
[OUTPUT]
Name:           coreos/etcd-operator@sha256:ac5496c52004b990f981ee419eb338921753c35db97e382496b21f15c7d23792
Content Digest: sha256:66a37fd61a06a43969854ee6d3e21087a98b93838e284a6086b13917f96b0d9b
                ERROR: the image contents do not match the requested digest, this image has been tampered with
Media Type:     application/vnd.docker.distribution.manifest.v1+prettyjws
Created:        1y ago
Image Size:     8 layers (size unavailable)
Layers:         -- sha256:59265c40e257554058624f35856dafd82d135c4ef406de298cb1fee647867381
                -- sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4
                -- sha256:8ea0315241b46a55ff650eddc8d06bfe7bfbdc072b4f3878d6c4598fc7b015d3
                -- sha256:678f2cfebea627b8b3d5bed91a6fe1d3749421fc504a568a7977ce912a953763
                -- sha256:e05d0d7eb99a1d91fa7349dd9a98a8b2d9e222b39fa305543e1ac4c380147a4b
                -- sha256:80b1b554e5b4c4a4334fd25764d6c3d78bafd04ef727714dfb25b1356cc3a265
                -- sha256:6d4b4c91b3ff6970e22c704f59d45a5945065cb80bd7055114f6d27432e2205a
                -- sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4
OS:             linux
Arch:           amd64
Command:        /bin/sh
User:           etcd-operator
Environment:    PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin

(5)conclusion

the original sha256 for schemaVersion1 image will be changed when use "oc image mirror" command mirror into local disk


2.image manifest schema V2 use "oc image mirror" command will will mantain original sha256

(1) check the image's schema version

[COMMAND]
# skopeo inspect --raw docker://docker.io/strimzi/jmxtrans@sha256:ab65157523eaaa25cf44fc273a01ac701dee00e5cd4f0c378b700e0f0f795b73 |jq -r .
[OUTPUT]
{
  "schemaVersion": 2,
...

You can see this image's schema version is 2

(2) mirror the images into local disk

[COMMAND]
# oc image mirror docker.io/strimzi/jmxtrans@sha256:ab65157523eaaa25cf44fc273a01ac701dee00e5cd4f0c378b700e0f0f795b73 --dir=images file://strimzi/jmxtrans
[OUTPUT]
<dir>
  strimzi/jmxtrans
    blobs:
      docker.io/strimzi/jmxtrans sha256:e87f8ee3aa3255c300490c1a183abc72010781f573cf8b924204c5eefdc5e03e 176B
      docker.io/strimzi/jmxtrans sha256:4e0c4e9806a135838d6f6be7c142de1932e532187839d076dcedd6ee3baa89c7 192B
      docker.io/strimzi/jmxtrans sha256:0fbfee66f135772127845b671742e4c619d0836a32d4e33113f6cb4bdcfb2297 401B
      docker.io/strimzi/jmxtrans sha256:c176ec8c02ed7f9f883dc6705af6fe6739a47195d980c5dc12ae57d972837568 619B
      docker.io/strimzi/jmxtrans sha256:634e7b6429a390e732225ae3ee179f58b9f8dae7484e619517cc282d4d4a6373 8.655KiB
      docker.io/strimzi/jmxtrans sha256:25436adb983683597f4da7cec06b65dccf85cc16bd90a163b084ff8926725f68 9.316KiB
      docker.io/strimzi/jmxtrans sha256:23c47d033b0e9c42515cb92474a5058f04b993123f07af2b44d54adbbf6bfe02 9.318KiB
      docker.io/strimzi/jmxtrans sha256:72d219db4282b110a0856d5f55394cdf1ecf850a6df78c21969430cae5bfec1a 18.74MiB
      docker.io/strimzi/jmxtrans sha256:682ec5793ad32c8c49553b46bf4f00c6b685a26b3743568b0b5e4a17a1f14e43 23.82MiB
      docker.io/strimzi/jmxtrans sha256:06ea991a3b933c49058585f82006648f8702a33b5de8725e2fe85724f18a2ff4 71.76MiB
      docker.io/strimzi/jmxtrans sha256:524b0c1e57f8ee5fee01a1decba2f301c324a6513ca3551021264e3aa7341ebc 72.36MiB
    manifests:
      sha256:ab65157523eaaa25cf44fc273a01ac701dee00e5cd4f0c378b700e0f0f795b73
  stats: shared=0 unique=11 size=186.7MiB ratio=1.00

phase 0:
   strimzi/jmxtrans blobs=11 mounts=0 manifests=1 shared=0

info: Planning completed in 2.33s
uploading: file://strimzi/jmxtrans sha256:524b0c1e57f8ee5fee01a1decba2f301c324a6513ca3551021264e3aa7341ebc 72.36MiB
uploading: file://strimzi/jmxtrans sha256:06ea991a3b933c49058585f82006648f8702a33b5de8725e2fe85724f18a2ff4 71.76MiB
uploading: file://strimzi/jmxtrans sha256:682ec5793ad32c8c49553b46bf4f00c6b685a26b3743568b0b5e4a17a1f14e43 23.82MiB
uploading: file://strimzi/jmxtrans sha256:72d219db4282b110a0856d5f55394cdf1ecf850a6df78c21969430cae5bfec1a 18.74MiB
sha256:ab65157523eaaa25cf44fc273a01ac701dee00e5cd4f0c378b700e0f0f795b73 file://strimzi/jmxtrans
info: Mirroring completed in 1.59s (122.9MB/s)

-----------------------------
We can see the original sha256 is well maintain into local disk
-----------------------------

(3) use the original sha256 check the downloaded image from local disk

[COMMAND]
# oc image info --dir=images file://strimzi/jmxtrans@sha256:ab65157523eaaa25cf44fc273a01ac701dee00e5cd4f0c378b700e0f0f795b73
[OUTPUT]
Name:        strimzi/jmxtrans@sha256:ab65157523eaaa25cf44fc273a01ac701dee00e5cd4f0c378b700e0f0f795b73
Media Type:  application/vnd.docker.distribution.manifest.v2+json
Created:     35d ago
Image Size:  195.8MB in 10 layers
Layers:      75.88MB sha256:524b0c1e57f8ee5fee01a1decba2f301c324a6513ca3551021264e3aa7341ebc
             75.24MB sha256:06ea991a3b933c49058585f82006648f8702a33b5de8725e2fe85724f18a2ff4
             176B    sha256:e87f8ee3aa3255c300490c1a183abc72010781f573cf8b924204c5eefdc5e03e
             192B    sha256:4e0c4e9806a135838d6f6be7c142de1932e532187839d076dcedd6ee3baa89c7
             19.65MB sha256:72d219db4282b110a0856d5f55394cdf1ecf850a6df78c21969430cae5bfec1a
             619B    sha256:c176ec8c02ed7f9f883dc6705af6fe6739a47195d980c5dc12ae57d972837568
             401B    sha256:0fbfee66f135772127845b671742e4c619d0836a32d4e33113f6cb4bdcfb2297
             9.54kB  sha256:25436adb983683597f4da7cec06b65dccf85cc16bd90a163b084ff8926725f68
             9.542kB sha256:23c47d033b0e9c42515cb92474a5058f04b993123f07af2b44d54adbbf6bfe02
             24.98MB sha256:682ec5793ad32c8c49553b46bf4f00c6b685a26b3743568b0b5e4a17a1f14e43
OS:          linux
Arch:        amd64
Entrypoint:  /docker-entrypoint.sh start-without-jmx
Working Dir: /usr/share/jmxtrans
Environment: PATH=/usr/share/jmxtrans/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
             JMXTRANS_HOME=/usr/share/jmxtrans
             JAR_FILE=/usr/share/jmxtrans/lib/jmxtrans.jar
             JMXTRANS_VERSION=271
             JMXTRANS_CHECKSUM=9c9116b628be912a723fae8ab65853908cc0872139340827b4af3dbdb7274bc88956af7f33dc927a43ea291c721701fb52368ccd414218ef33e7de3060baf849  jmxtrans-271-all.jar
             HEAP_SIZE=512
             PERM_SIZE=384
             MAX_PERM_SIZE=384
             SECONDS_BETWEEN_RUNS=60
             CONTINUE_ON_ERROR=false
             JSON_DIR=/var/lib/jmxtrans
             TINI_VERSION=v0.18.0
             TINI_SHA256=12d20136605531b09a2c2dac02ccee85e1b874eb322ef6baf7561cd93f93c855
Labels:      org.label-schema.build-date=20200504
             org.label-schema.license=GPLv2
             org.label-schema.name=CentOS Base Image
             org.label-schema.schema-version=1.0
             org.label-schema.vendor=CentOS
             org.opencontainers.image.created=2020-05-04 00:00:00+01:00
             org.opencontainers.image.licenses=GPL-2.0-only
             org.opencontainers.image.title=CentOS Base Image
             org.opencontainers.image.vendor=CentOS
Volumes:     /var/lib/jmxtrans

-----------------------------
We can see the original sha256 is well maintain into local disk
-----------------------------

(4)conclusion

we can see schema version 2 image use "oc image mirror" command mirror into local disk will well mantain the original sha256


Expected results:

Why use "oc image mirror" command mirror "schema version 1 image" into local disk will change the original sha256, but "schema version 2 image" will well mantain the original sha256?

Is this a BUG???

Comment 1 kevin 2020-06-25 08:54:56 UTC
In OCP operatorHub, I have found many "schema version 1 image" as follwing:

quay.io/coreos/etcd-operator@sha256:66a37fd61a06a43969854ee6d3e21087a98b93838e284a6086b13917f96b0d9b=registry.example.internal:5000/coreos/etcd-operator:latest
This image's schema version is 1
quay.io/coreos/etcd-operator@sha256:bd944a211eaf8f31da5e6d69e8541e7cada8f16a9f7a5a570b22478997819943=registry.example.internal:5000/coreos/etcd-operator:latest
This image's schema version is 1
quay.io/coreos/etcd-operator@sha256:c0301e4686c3ed4206e370b42de5a3bd2229b9fb4906cf85f3f30650424abec2=registry.example.internal:5000/coreos/etcd-operator:latest
This image's schema version is 1
quay.io/coreos/etcd-operator@sha256:db563baa8194fcfe39d1df744ed70024b0f1f9e9b55b5923c2f3a413c44dc6b8=registry.example.internal:5000/coreos/etcd-operator:latest
This image's schema version is 1
quay.io/coreos/prometheus-operator@sha256:0e92dd9b5789c4b13d53e1319d0a6375bcca4caaf0d698af61198061222a576d=registry.example.internal:5000/coreos/prometheus-operator:latest
This image's schema version is 1
quay.io/coreos/prometheus-operator@sha256:3daa69a8c6c2f1d35dcf1fe48a7cd8b230e55f5229a1ded438f687debade5bcf=registry.example.internal:5000/coreos/prometheus-operator:latest
This image's schema version is 1
quay.io/coreos/prometheus-operator@sha256:5037b4e90dbb03ebdefaa547ddf6a1f748c8eeebeedf6b9d9f0913ad662b5731=registry.example.internal:5000/coreos/prometheus-operator:latest
This image's schema version is 1
quay.io/coreos/prometheus-operator@sha256:933cd5bf380cf7db330808ff54f75f26fda0b1501021d499a1766b7d16224188=registry.example.internal:5000/coreos/prometheus-operator:latest
This image's schema version is 1
quay.io/coreos/prometheus-operator@sha256:ed3ec0597c2d5b7102a7f62c661a23d8e4b34d910693fc23fd40bfb1d9404dcf=registry.example.internal:5000/coreos/prometheus-operator:latest
This image's schema version is 1
quay.io/jmckind/argocd-operator@sha256:0fa4b7709e1e9c9cb9ca064be50618f71ff3eef07e185c631b1227f8e5a57776=registry.example.internal:5000/jmckind/argocd-operator:8.1-328_linux-amd64
This image's schema version is 1
quay.io/jmckind/argocd-operator@sha256:5d7c4b0e8e0fea068e49a9718a35ae068fc267e607f7393374db39916d7186f4=registry.example.internal:5000/jmckind/argocd-operator:7.7-211_linux-amd64
This image's schema version is 1
quay.io/jmckind/argocd-operator@sha256:d1385d23a60205636bc3789b0127d6159d33d7a7521dd07d6b679b7f734ee4b3=registry.example.internal:5000/jmckind/argocd-operator:7.7-211_linux-amd64
This image's schema version is 1
quay.io/jmckind/argocd-operator@sha256:fd7aaf9a0b330d5f646aa69933c8149de60b680878208f473543fd3c43412096=registry.example.internal:5000/jmckind/argocd-operator:7.7-211_linux-amd64
This image's schema version is 1
quay.io/quay/container-security-operator@sha256:154d7e0295a94fb3d2a97309d711186a98a7308da37a5cd3d50360c6b2ba57de=registry.example.internal:5000/quay/container-security-operator:latest
This image's schema version is 1
quay.io/quay/container-security-operator@sha256:15a4b50d847512b5f404ec1cf72c30c98e073a7f26f1588213bd2e8b6331f016=registry.example.internal:5000/quay/container-security-operator:latest
This image's schema version is 1
quay.io/quay/container-security-operator@sha256:6eefeaee910251ba26c825746d11ae166a9781aeace5455b2766d26298911f13=registry.example.internal:5000/quay/container-security-operator:latest
This image's schema version is 1
quay.io/quay/container-security-operator@sha256:7998f9377973cdc22d8ad713ba1b81381db9782a4b58d4c89f4bed688e2ff461=registry.example.internal:5000/quay/container-security-operator:latest
This image's schema version is 1

Comment 2 Chris Doan 2020-06-25 19:20:45 UTC
The etcd images in quay have the schema1 manifests. In deploying RHACM 1.0 in disconnected install mode, we go through the disconnected OLM procedure to mirror down the the two operator catalogs--Red Hat Operators, and Community Operators, into a mirror registry. First, the mirror registry needs to allow schema1 compatibility. We can configure this like the following commands, adding the compatibility stanza to the registry config:

# podman exec -it registry /bin/sh

# vi etc/docker/registry/config.yml
compatibility:
  schema1:
    enabled: true

# podman restart registry

Then, you should be able to load schema1 images without a problem.

Comment 3 Chris Doan 2020-06-26 02:51:21 UTC
Also, use skopeo to copy will maintain the digest.

skopeo copy docker://quay.io/coreos/etcd-operator@sha256:66a37fd61a06a43969854ee6d3e21087a98b93838e284a6086b13917f96b0d9b dir:/tmp/foo/etcd-operator
skopeo inspect dir:/tmp/foo/etcd-operator
{
    "Tag": "v0.9.4",
    "Digest": "sha256:66a37fd61a06a43969854ee6d3e21087a98b93838e284a6086b13917f96b0d9b",
    "RepoTags": [],
    "Created": "2019-02-28T02:02:50.022009749Z",
    "DockerVersion": "18.09.2",
    "Labels": null,
    "Architecture": "amd64",
    "Os": "linux",
    "Layers": [
        "sha256:59265c40e257554058624f35856dafd82d135c4ef406de298cb1fee647867381",
        "sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4",
        "sha256:8ea0315241b46a55ff650eddc8d06bfe7bfbdc072b4f3878d6c4598fc7b015d3",
        "sha256:678f2cfebea627b8b3d5bed91a6fe1d3749421fc504a568a7977ce912a953763",
        "sha256:e05d0d7eb99a1d91fa7349dd9a98a8b2d9e222b39fa305543e1ac4c380147a4b",
        "sha256:80b1b554e5b4c4a4334fd25764d6c3d78bafd04ef727714dfb25b1356cc3a265",
        "sha256:6d4b4c91b3ff6970e22c704f59d45a5945065cb80bd7055114f6d27432e2205a",
        "sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4"
    ],
    "Env": [
        "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"
    ]
}

Comment 4 kevin 2020-06-26 07:26:29 UTC
SKOPEO Can Not solve this problem, reproduce following:

1- use skopeo download image into localdisk

[root@instance-micro ~]# skopeo copy docker://quay.io/coreos/etcd-operator@sha256:66a37fd61a06a43969854ee6d3e21087a98b93838e284a6086b13917f96b0d9b dir:./etcd-operator
Getting image source signatures
Copying blob 59265c40e257 done
Copying blob a3ed95caeb02 done
Copying blob 8ea0315241b4 done
Copying blob 678f2cfebea6 done
Copying blob e05d0d7eb99a done
Copying blob 80b1b554e5b4 done
Copying blob 6d4b4c91b3ff done
Copying blob a3ed95caeb02 skipped: already exists
Writing manifest to image destination
Storing signatures

2- use skopeo push the local disk image file into my private registry server

skopeo copy dir:./etcd-operator docker://localhost/coreos/etcd-operator
Getting image source signatures
Copying blob 59265c40e257 done
Copying blob a3ed95caeb02 done
Copying blob 8ea0315241b4 done
Copying blob 678f2cfebea6 done
Copying blob e05d0d7eb99a done
Copying blob 80b1b554e5b4 done
Copying blob 6d4b4c91b3ff done
Copying blob a3ed95caeb02 skipped: already exists
Writing manifest to image destination
Storing signatures

3- check the pushed image's sha256

[root@instance-micro ~]# oc image info localhost/coreos/etcd-operator
Name:        localhost/coreos/etcd-operator:latest
Digest:      sha256:1fb5a1ea2b048fb604cb4fb4076fbc5356977423ca65006b3b8c89a8fbbcffb7
Media Type:  application/vnd.docker.distribution.manifest.v1+prettyjws
Created:     1y ago
Image Size:  8 layers (size unavailable)
Layers:      -- sha256:59265c40e257554058624f35856dafd82d135c4ef406de298cb1fee647867381
             -- sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4
             -- sha256:8ea0315241b46a55ff650eddc8d06bfe7bfbdc072b4f3878d6c4598fc7b015d3
             -- sha256:678f2cfebea627b8b3d5bed91a6fe1d3749421fc504a568a7977ce912a953763
             -- sha256:e05d0d7eb99a1d91fa7349dd9a98a8b2d9e222b39fa305543e1ac4c380147a4b
             -- sha256:80b1b554e5b4c4a4334fd25764d6c3d78bafd04ef727714dfb25b1356cc3a265
             -- sha256:6d4b4c91b3ff6970e22c704f59d45a5945065cb80bd7055114f6d27432e2205a
             -- sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4
OS:          linux
Arch:        amd64
Command:     /bin/sh
User:        etcd-operator
Environment: PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin

conclusion: You Will See the sha256 have been changed as sha256:1fb5a1ea2b048fb604cb4fb4076fbc5356977423ca65006b3b8c89a8fbbcffb7

Comment 5 Chris Doan 2020-06-26 19:59:17 UTC
Yes, going to back to `oc image mirror`, everything works as expected when mirroring between two registry. But when mirroring to a local file, the digest is not maintained.

Comment 6 Lucian Maly 2020-06-29 03:02:12 UTC
Might be related to #02584003? When the image has certain amount of layers, last layers do not get mirrored for some reason.

Comment 7 Lucian Maly 2020-06-29 03:06:10 UTC
Ignore the number in my comment #6. The right bug is https://bugzilla.redhat.com/show_bug.cgi?id=1797203

Comment 8 kevin 2020-06-29 08:21:04 UTC
I am also try to use "skopeo copy --all", the SHA256 Still have been changed!!!

[root@instance-micro ~]# skopeo copy --all docker://quay.io/coreos/etcd-operator@sha256:66a37fd61a06a43969854ee6d3e21087a98b93838e284a6086b13917f96b0d9b dir:./etcd-operator
Getting image source signatures
Copying blob 59265c40e257 done
Copying blob a3ed95caeb02 done
Copying blob 8ea0315241b4 done
Copying blob 678f2cfebea6 done
Copying blob e05d0d7eb99a done
Copying blob 80b1b554e5b4 done
Copying blob 6d4b4c91b3ff done
Copying blob a3ed95caeb02 skipped: already exists
Writing manifest to image destination
Storing signatures

[root@instance-micro ~]# skopeo copy --all dir:./etcd-operator docker://localhost/coreos/etcd-operator
Getting image source signatures
Copying blob 59265c40e257 skipped: already exists
Copying blob a3ed95caeb02 skipped: already exists
Copying blob 8ea0315241b4 skipped: already exists
Copying blob 678f2cfebea6 skipped: already exists
Copying blob e05d0d7eb99a skipped: already exists
Copying blob 80b1b554e5b4 skipped: already exists
Copying blob 6d4b4c91b3ff skipped: already exists
Copying blob a3ed95caeb02 skipped: already exists
Writing manifest to image destination
Storing signatures

[root@instance-micro ~]# oc image info localhost/coreos/etcd-operator
Name:        localhost/coreos/etcd-operator:latest
Digest:      sha256:1fb5a1ea2b048fb604cb4fb4076fbc5356977423ca65006b3b8c89a8fbbcffb7
Media Type:  application/vnd.docker.distribution.manifest.v1+prettyjws
Created:     1y ago
Image Size:  8 layers (size unavailable)
Layers:      -- sha256:59265c40e257554058624f35856dafd82d135c4ef406de298cb1fee647867381
             -- sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4
             -- sha256:8ea0315241b46a55ff650eddc8d06bfe7bfbdc072b4f3878d6c4598fc7b015d3
             -- sha256:678f2cfebea627b8b3d5bed91a6fe1d3749421fc504a568a7977ce912a953763
             -- sha256:e05d0d7eb99a1d91fa7349dd9a98a8b2d9e222b39fa305543e1ac4c380147a4b
             -- sha256:80b1b554e5b4c4a4334fd25764d6c3d78bafd04ef727714dfb25b1356cc3a265
             -- sha256:6d4b4c91b3ff6970e22c704f59d45a5945065cb80bd7055114f6d27432e2205a
             -- sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4
OS:          linux
Arch:        amd64
Command:     /bin/sh
User:        etcd-operator
Environment: PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin

Comment 9 Miloslav Trmač 2020-06-30 17:28:38 UTC
FWIW, `skopeo copy` does not change the schema1 manifests _if_ the image is signed (and the signature is visible to Skopeo), or if the destination uses a digested reference.

So,
> skopeo copy docker://$src@$digest docker://$dest@$digest
should work. But this is impossible with an intermediate `dir:` step.

Comment 10 Miloslav Trmač 2020-06-30 17:31:09 UTC
… actually, let me correct that:
> skopeo copy docker://… dir:…
never modifies the manifest (unless extra options are used to explicitly request changes to the image)
> skopeo copy dir:… docker://…
does not modify schema1 manifests if the destination uses a digested reference. (But if you do need to create a tag for that image pushed by digest, Skopeo can’t do that.)

Comment 11 Sally 2020-06-30 18:13:59 UTC
Summary:
  - schema version 1 contains the registry name as well as its tag - if either of those change the digest will change
  - schema version 2 is a pure content-reference, so digest will never change
  - older images (from before quay supported schema 2) are schema 1, and only way to update to schema 2 is to build/push new images. 

  - with skopeo copy, if you don't modify name:tag while doing the copy, the schema1 digest will be preserved.  
  ex:
   $ skopeo copy $source:tag $dest:tag will cause the manifest to be edited
   $ skopeo copy $source@digest $dest@digest will preserve 

 
Running with -v=8 I see this:

$ oc image mirror quay.io/coreos/etcd-operator@sha256:66a37fd61a06a43969854ee6d3e21087a98b93838e284a6086b13917f96b0d9b --dir=/tmp/images file://coreos/etcd-operator -v=8
---
plan.go:344] Associated digest sha256:66a37fd61a06a43969854ee6d3e21087a98b93838e284a6086b13917f96b0d9b with converted digest sha256:db9eac85a5ca921e38713d0bbf8372b1d354538887b952dd547db0c86a82d2da
sha256:db9eac85a5ca921e38713d0bbf8372b1d354538887b952dd547db0c86a82d2da file://coreos/etcd-operator
info: Mirroring completed in 6.76s (7.791MB/s)

You can see the original digest is logged w/ the converted digest.  
Should oc image mirror be updated to force preserving the original digest? Not sure, will discuss among workloads team and report back.

Comment 12 Sally 2020-07-10 19:27:37 UTC
This is an issue with schema v1 images in general.  We are adding a warning in 'oc image mirror' that digests are only guaranteed to remain the same w/ V2 images. Also, we'll add a warning that states support for schema V1 images will be dropped in the future.  Once those changes are in oc code, I'll be closing this bug as there is no 'fix' other than move all images to V2. Those warnings will be added in the upcoming sprint.

Comment 13 kevin 2020-07-13 01:43:14 UTC
If we cannot maintain the digest by oc image mirror command, I recommend the operator which images base on schema v1 would not use sha256 (such as etcd operator)instead use tag, I have found some images in Community Channel in OperatorHub use schema v1

Comment 14 Miloslav Trmač 2020-07-13 17:49:55 UTC
(In reply to kevin from comment #13)
> If we cannot maintain the digest by oc image mirror command, I recommend the
> operator which images base on schema v1 would not use sha256 (such as etcd
> operator)instead use tag

Note that tags don’t work with the way disconnected clusters use imageContentSourcePoilcy, so if the operator hub disconnected access depends on ICSP, that’s not an option.

Comment 18 zhou ying 2020-07-21 09:45:44 UTC
Confirmed with oc [root@dhcp-140-138 ~]# oc version --client
Client Version: 4.6.0-202007171623.p0-c33851e

We could see the warning now:

[root@dhcp-140-138 ~]# oc image mirror quay.io/coreos/etcd-operator@sha256:66a37fd61a06a43969854ee6d3e21087a98b93838e284a6086b13917f96b0d9b --dir=/tmp/images file://coreos/etcd-operator 
.......
warning: Digests are not preserved with schema version 1 images. Support for schema version 1 images will be removed in a future release.
sha256:db9eac85a5ca921e38713d0bbf8372b1d354538887b952dd547db0c86a82d2da file://coreos/etcd-operator
info: Mirroring completed in 12.93s (4.072MB/s)

Comment 20 errata-xmlrpc 2020-10-27 16:09:42 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory (OpenShift Container Platform 4.6 GA Images), and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2020:4196


Note You need to log in before you can comment on or make changes to this bug.