Description of problem: for manifest schemaVersion1 images, use oc image mirror will change the original image sha256 value Version-Release number of selected component (if applicable): oc version Client Version: 4.4.9 How reproducible: Steps to Reproduce: 1.image manifest schema V1 use "oc image mirror" command mirror into local disk will change the original sha256 value!!! now, I use operatorhub etcd operator image as example, this image is very important, because currently our ACM operator(Advanced Cluster Management rely on it !) the image url is: quay.io/coreos/etcd-operator@sha256:66a37fd61a06a43969854ee6d3e21087a98b93838e284a6086b13917f96b0d9b (1) check the image's schema version you can use the following command to check the image schema version number: [COMMAND] # skopeo inspect --raw docker://quay.io/coreos/etcd-operator@sha256:66a37fd61a06a43969854ee6d3e21087a98b93838e284a6086b13917f96b0d9b |jq -r . [OUTPUT] { "schemaVersion": 1, ... You can see this image's schema version is 1 (2) mirror the images into local disk I use following command to download the image into local disk oc image mirror <imageURL> --dir=<DIR> file://<fileURL> [COMMAND] # oc image mirror quay.io/coreos/etcd-operator@sha256:66a37fd61a06a43969854ee6d3e21087a98b93838e284a6086b13917f96b0d9b --dir=images file://coreos/etcd-operator [OUTPUT] <dir> coreos/etcd-operator blobs: quay.io/coreos/etcd-operator sha256:59265c40e257554058624f35856dafd82d135c4ef406de298cb1fee647867381 quay.io/coreos/etcd-operator sha256:678f2cfebea627b8b3d5bed91a6fe1d3749421fc504a568a7977ce912a953763 quay.io/coreos/etcd-operator sha256:6d4b4c91b3ff6970e22c704f59d45a5945065cb80bd7055114f6d27432e2205a quay.io/coreos/etcd-operator sha256:80b1b554e5b4c4a4334fd25764d6c3d78bafd04ef727714dfb25b1356cc3a265 quay.io/coreos/etcd-operator sha256:8ea0315241b46a55ff650eddc8d06bfe7bfbdc072b4f3878d6c4598fc7b015d3 quay.io/coreos/etcd-operator sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4 quay.io/coreos/etcd-operator sha256:e05d0d7eb99a1d91fa7349dd9a98a8b2d9e222b39fa305543e1ac4c380147a4b manifests: sha256:66a37fd61a06a43969854ee6d3e21087a98b93838e284a6086b13917f96b0d9b stats: shared=0 unique=7 size=0B phase 0: coreos/etcd-operator blobs=7 mounts=0 manifests=1 shared=0 info: Planning completed in 1.2s uploading: file://coreos/etcd-operator sha256:59265c40e257554058624f35856dafd82d135c4ef406de298cb1fee647867381 0B uploading: file://coreos/etcd-operator sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4 0B uploading: file://coreos/etcd-operator sha256:6d4b4c91b3ff6970e22c704f59d45a5945065cb80bd7055114f6d27432e2205a 0B uploading: file://coreos/etcd-operator sha256:80b1b554e5b4c4a4334fd25764d6c3d78bafd04ef727714dfb25b1356cc3a265 0B uploading: file://coreos/etcd-operator sha256:e05d0d7eb99a1d91fa7349dd9a98a8b2d9e222b39fa305543e1ac4c380147a4b 0B uploading: file://coreos/etcd-operator sha256:678f2cfebea627b8b3d5bed91a6fe1d3749421fc504a568a7977ce912a953763 0B warning: Layer size mismatch for sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4: had 0, wrote 32 warning: Layer size mismatch for sha256:6d4b4c91b3ff6970e22c704f59d45a5945065cb80bd7055114f6d27432e2205a: had 0, wrote 1251 warning: Layer size mismatch for sha256:59265c40e257554058624f35856dafd82d135c4ef406de298cb1fee647867381: had 0, wrote 2016712 warning: Layer size mismatch for sha256:80b1b554e5b4c4a4334fd25764d6c3d78bafd04ef727714dfb25b1356cc3a265: had 0, wrote 15521633 uploading: file://coreos/etcd-operator sha256:8ea0315241b46a55ff650eddc8d06bfe7bfbdc072b4f3878d6c4598fc7b015d3 0B warning: Layer size mismatch for sha256:e05d0d7eb99a1d91fa7349dd9a98a8b2d9e222b39fa305543e1ac4c380147a4b: had 0, wrote 17235178 warning: Layer size mismatch for sha256:678f2cfebea627b8b3d5bed91a6fe1d3749421fc504a568a7977ce912a953763: had 0, wrote 17555180 warning: Layer size mismatch for sha256:8ea0315241b46a55ff650eddc8d06bfe7bfbdc072b4f3878d6c4598fc7b015d3: had 0, wrote 351520 sha256:ac5496c52004b990f981ee419eb338921753c35db97e382496b21f15c7d23792 file://coreos/etcd-operator info: Mirroring completed in 640ms (81.8MB/s) -------------------------------------- !!!!!!Note: we can see the real sha256 have been changed into sha256:ac5496c52004b990f981ee419eb338921753c35db97e382496b21f15c7d23792 , this is totally different from original sha256:66a37fd61a06a43969854ee6d3e21087a98b93838e284a6086b13917f96b0d9b -------------------------------------- (3) use the original sha256 check the downloaded image from local disk I use following command to check the image into local disk oc image info --dir=<DIR> file://<fileURL> [COMMAND] # oc image info --dir=images file://coreos/etcd-operator@sha256:66a37fd61a06a43969854ee6d3e21087a98b93838e284a6086b13917f96b0d9b [OUTPUT] error: unable to read image file://coreos/etcd-operator@sha256:66a37fd61a06a43969854ee6d3e21087a98b93838e284a6086b13917f96b0d9b: open images/v2/coreos/etcd-operator/manifests/sha256:66a37fd61a06a43969854ee6d3e21087a98b93838e284a6086b13917f96b0d9b: no such file or directory -------------------- You can see original sha256 have faild -------------------- (4) check the real sha256 value from downloaded image of local disk [COMMAND] # oc image info --dir=images file://coreos/etcd-operator@sha256:ac5496c52004b990f981ee419eb338921753c35db97e382496b21f15c7d23792 [OUTPUT] Name: coreos/etcd-operator@sha256:ac5496c52004b990f981ee419eb338921753c35db97e382496b21f15c7d23792 Content Digest: sha256:66a37fd61a06a43969854ee6d3e21087a98b93838e284a6086b13917f96b0d9b ERROR: the image contents do not match the requested digest, this image has been tampered with Media Type: application/vnd.docker.distribution.manifest.v1+prettyjws Created: 1y ago Image Size: 8 layers (size unavailable) Layers: -- sha256:59265c40e257554058624f35856dafd82d135c4ef406de298cb1fee647867381 -- sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4 -- sha256:8ea0315241b46a55ff650eddc8d06bfe7bfbdc072b4f3878d6c4598fc7b015d3 -- sha256:678f2cfebea627b8b3d5bed91a6fe1d3749421fc504a568a7977ce912a953763 -- sha256:e05d0d7eb99a1d91fa7349dd9a98a8b2d9e222b39fa305543e1ac4c380147a4b -- sha256:80b1b554e5b4c4a4334fd25764d6c3d78bafd04ef727714dfb25b1356cc3a265 -- sha256:6d4b4c91b3ff6970e22c704f59d45a5945065cb80bd7055114f6d27432e2205a -- sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4 OS: linux Arch: amd64 Command: /bin/sh User: etcd-operator Environment: PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin (5)conclusion the original sha256 for schemaVersion1 image will be changed when use "oc image mirror" command mirror into local disk 2.image manifest schema V2 use "oc image mirror" command will will mantain original sha256 (1) check the image's schema version [COMMAND] # skopeo inspect --raw docker://docker.io/strimzi/jmxtrans@sha256:ab65157523eaaa25cf44fc273a01ac701dee00e5cd4f0c378b700e0f0f795b73 |jq -r . [OUTPUT] { "schemaVersion": 2, ... You can see this image's schema version is 2 (2) mirror the images into local disk [COMMAND] # oc image mirror docker.io/strimzi/jmxtrans@sha256:ab65157523eaaa25cf44fc273a01ac701dee00e5cd4f0c378b700e0f0f795b73 --dir=images file://strimzi/jmxtrans [OUTPUT] <dir> strimzi/jmxtrans blobs: docker.io/strimzi/jmxtrans sha256:e87f8ee3aa3255c300490c1a183abc72010781f573cf8b924204c5eefdc5e03e 176B docker.io/strimzi/jmxtrans sha256:4e0c4e9806a135838d6f6be7c142de1932e532187839d076dcedd6ee3baa89c7 192B docker.io/strimzi/jmxtrans sha256:0fbfee66f135772127845b671742e4c619d0836a32d4e33113f6cb4bdcfb2297 401B docker.io/strimzi/jmxtrans sha256:c176ec8c02ed7f9f883dc6705af6fe6739a47195d980c5dc12ae57d972837568 619B docker.io/strimzi/jmxtrans sha256:634e7b6429a390e732225ae3ee179f58b9f8dae7484e619517cc282d4d4a6373 8.655KiB docker.io/strimzi/jmxtrans sha256:25436adb983683597f4da7cec06b65dccf85cc16bd90a163b084ff8926725f68 9.316KiB docker.io/strimzi/jmxtrans sha256:23c47d033b0e9c42515cb92474a5058f04b993123f07af2b44d54adbbf6bfe02 9.318KiB docker.io/strimzi/jmxtrans sha256:72d219db4282b110a0856d5f55394cdf1ecf850a6df78c21969430cae5bfec1a 18.74MiB docker.io/strimzi/jmxtrans sha256:682ec5793ad32c8c49553b46bf4f00c6b685a26b3743568b0b5e4a17a1f14e43 23.82MiB docker.io/strimzi/jmxtrans sha256:06ea991a3b933c49058585f82006648f8702a33b5de8725e2fe85724f18a2ff4 71.76MiB docker.io/strimzi/jmxtrans sha256:524b0c1e57f8ee5fee01a1decba2f301c324a6513ca3551021264e3aa7341ebc 72.36MiB manifests: sha256:ab65157523eaaa25cf44fc273a01ac701dee00e5cd4f0c378b700e0f0f795b73 stats: shared=0 unique=11 size=186.7MiB ratio=1.00 phase 0: strimzi/jmxtrans blobs=11 mounts=0 manifests=1 shared=0 info: Planning completed in 2.33s uploading: file://strimzi/jmxtrans sha256:524b0c1e57f8ee5fee01a1decba2f301c324a6513ca3551021264e3aa7341ebc 72.36MiB uploading: file://strimzi/jmxtrans sha256:06ea991a3b933c49058585f82006648f8702a33b5de8725e2fe85724f18a2ff4 71.76MiB uploading: file://strimzi/jmxtrans sha256:682ec5793ad32c8c49553b46bf4f00c6b685a26b3743568b0b5e4a17a1f14e43 23.82MiB uploading: file://strimzi/jmxtrans sha256:72d219db4282b110a0856d5f55394cdf1ecf850a6df78c21969430cae5bfec1a 18.74MiB sha256:ab65157523eaaa25cf44fc273a01ac701dee00e5cd4f0c378b700e0f0f795b73 file://strimzi/jmxtrans info: Mirroring completed in 1.59s (122.9MB/s) ----------------------------- We can see the original sha256 is well maintain into local disk ----------------------------- (3) use the original sha256 check the downloaded image from local disk [COMMAND] # oc image info --dir=images file://strimzi/jmxtrans@sha256:ab65157523eaaa25cf44fc273a01ac701dee00e5cd4f0c378b700e0f0f795b73 [OUTPUT] Name: strimzi/jmxtrans@sha256:ab65157523eaaa25cf44fc273a01ac701dee00e5cd4f0c378b700e0f0f795b73 Media Type: application/vnd.docker.distribution.manifest.v2+json Created: 35d ago Image Size: 195.8MB in 10 layers Layers: 75.88MB sha256:524b0c1e57f8ee5fee01a1decba2f301c324a6513ca3551021264e3aa7341ebc 75.24MB sha256:06ea991a3b933c49058585f82006648f8702a33b5de8725e2fe85724f18a2ff4 176B sha256:e87f8ee3aa3255c300490c1a183abc72010781f573cf8b924204c5eefdc5e03e 192B sha256:4e0c4e9806a135838d6f6be7c142de1932e532187839d076dcedd6ee3baa89c7 19.65MB sha256:72d219db4282b110a0856d5f55394cdf1ecf850a6df78c21969430cae5bfec1a 619B sha256:c176ec8c02ed7f9f883dc6705af6fe6739a47195d980c5dc12ae57d972837568 401B sha256:0fbfee66f135772127845b671742e4c619d0836a32d4e33113f6cb4bdcfb2297 9.54kB sha256:25436adb983683597f4da7cec06b65dccf85cc16bd90a163b084ff8926725f68 9.542kB sha256:23c47d033b0e9c42515cb92474a5058f04b993123f07af2b44d54adbbf6bfe02 24.98MB sha256:682ec5793ad32c8c49553b46bf4f00c6b685a26b3743568b0b5e4a17a1f14e43 OS: linux Arch: amd64 Entrypoint: /docker-entrypoint.sh start-without-jmx Working Dir: /usr/share/jmxtrans Environment: PATH=/usr/share/jmxtrans/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin JMXTRANS_HOME=/usr/share/jmxtrans JAR_FILE=/usr/share/jmxtrans/lib/jmxtrans.jar JMXTRANS_VERSION=271 JMXTRANS_CHECKSUM=9c9116b628be912a723fae8ab65853908cc0872139340827b4af3dbdb7274bc88956af7f33dc927a43ea291c721701fb52368ccd414218ef33e7de3060baf849 jmxtrans-271-all.jar HEAP_SIZE=512 PERM_SIZE=384 MAX_PERM_SIZE=384 SECONDS_BETWEEN_RUNS=60 CONTINUE_ON_ERROR=false JSON_DIR=/var/lib/jmxtrans TINI_VERSION=v0.18.0 TINI_SHA256=12d20136605531b09a2c2dac02ccee85e1b874eb322ef6baf7561cd93f93c855 Labels: org.label-schema.build-date=20200504 org.label-schema.license=GPLv2 org.label-schema.name=CentOS Base Image org.label-schema.schema-version=1.0 org.label-schema.vendor=CentOS org.opencontainers.image.created=2020-05-04 00:00:00+01:00 org.opencontainers.image.licenses=GPL-2.0-only org.opencontainers.image.title=CentOS Base Image org.opencontainers.image.vendor=CentOS Volumes: /var/lib/jmxtrans ----------------------------- We can see the original sha256 is well maintain into local disk ----------------------------- (4)conclusion we can see schema version 2 image use "oc image mirror" command mirror into local disk will well mantain the original sha256 Expected results: Why use "oc image mirror" command mirror "schema version 1 image" into local disk will change the original sha256, but "schema version 2 image" will well mantain the original sha256? Is this a BUG???
In OCP operatorHub, I have found many "schema version 1 image" as follwing: quay.io/coreos/etcd-operator@sha256:66a37fd61a06a43969854ee6d3e21087a98b93838e284a6086b13917f96b0d9b=registry.example.internal:5000/coreos/etcd-operator:latest This image's schema version is 1 quay.io/coreos/etcd-operator@sha256:bd944a211eaf8f31da5e6d69e8541e7cada8f16a9f7a5a570b22478997819943=registry.example.internal:5000/coreos/etcd-operator:latest This image's schema version is 1 quay.io/coreos/etcd-operator@sha256:c0301e4686c3ed4206e370b42de5a3bd2229b9fb4906cf85f3f30650424abec2=registry.example.internal:5000/coreos/etcd-operator:latest This image's schema version is 1 quay.io/coreos/etcd-operator@sha256:db563baa8194fcfe39d1df744ed70024b0f1f9e9b55b5923c2f3a413c44dc6b8=registry.example.internal:5000/coreos/etcd-operator:latest This image's schema version is 1 quay.io/coreos/prometheus-operator@sha256:0e92dd9b5789c4b13d53e1319d0a6375bcca4caaf0d698af61198061222a576d=registry.example.internal:5000/coreos/prometheus-operator:latest This image's schema version is 1 quay.io/coreos/prometheus-operator@sha256:3daa69a8c6c2f1d35dcf1fe48a7cd8b230e55f5229a1ded438f687debade5bcf=registry.example.internal:5000/coreos/prometheus-operator:latest This image's schema version is 1 quay.io/coreos/prometheus-operator@sha256:5037b4e90dbb03ebdefaa547ddf6a1f748c8eeebeedf6b9d9f0913ad662b5731=registry.example.internal:5000/coreos/prometheus-operator:latest This image's schema version is 1 quay.io/coreos/prometheus-operator@sha256:933cd5bf380cf7db330808ff54f75f26fda0b1501021d499a1766b7d16224188=registry.example.internal:5000/coreos/prometheus-operator:latest This image's schema version is 1 quay.io/coreos/prometheus-operator@sha256:ed3ec0597c2d5b7102a7f62c661a23d8e4b34d910693fc23fd40bfb1d9404dcf=registry.example.internal:5000/coreos/prometheus-operator:latest This image's schema version is 1 quay.io/jmckind/argocd-operator@sha256:0fa4b7709e1e9c9cb9ca064be50618f71ff3eef07e185c631b1227f8e5a57776=registry.example.internal:5000/jmckind/argocd-operator:8.1-328_linux-amd64 This image's schema version is 1 quay.io/jmckind/argocd-operator@sha256:5d7c4b0e8e0fea068e49a9718a35ae068fc267e607f7393374db39916d7186f4=registry.example.internal:5000/jmckind/argocd-operator:7.7-211_linux-amd64 This image's schema version is 1 quay.io/jmckind/argocd-operator@sha256:d1385d23a60205636bc3789b0127d6159d33d7a7521dd07d6b679b7f734ee4b3=registry.example.internal:5000/jmckind/argocd-operator:7.7-211_linux-amd64 This image's schema version is 1 quay.io/jmckind/argocd-operator@sha256:fd7aaf9a0b330d5f646aa69933c8149de60b680878208f473543fd3c43412096=registry.example.internal:5000/jmckind/argocd-operator:7.7-211_linux-amd64 This image's schema version is 1 quay.io/quay/container-security-operator@sha256:154d7e0295a94fb3d2a97309d711186a98a7308da37a5cd3d50360c6b2ba57de=registry.example.internal:5000/quay/container-security-operator:latest This image's schema version is 1 quay.io/quay/container-security-operator@sha256:15a4b50d847512b5f404ec1cf72c30c98e073a7f26f1588213bd2e8b6331f016=registry.example.internal:5000/quay/container-security-operator:latest This image's schema version is 1 quay.io/quay/container-security-operator@sha256:6eefeaee910251ba26c825746d11ae166a9781aeace5455b2766d26298911f13=registry.example.internal:5000/quay/container-security-operator:latest This image's schema version is 1 quay.io/quay/container-security-operator@sha256:7998f9377973cdc22d8ad713ba1b81381db9782a4b58d4c89f4bed688e2ff461=registry.example.internal:5000/quay/container-security-operator:latest This image's schema version is 1
The etcd images in quay have the schema1 manifests. In deploying RHACM 1.0 in disconnected install mode, we go through the disconnected OLM procedure to mirror down the the two operator catalogs--Red Hat Operators, and Community Operators, into a mirror registry. First, the mirror registry needs to allow schema1 compatibility. We can configure this like the following commands, adding the compatibility stanza to the registry config: # podman exec -it registry /bin/sh # vi etc/docker/registry/config.yml compatibility: schema1: enabled: true # podman restart registry Then, you should be able to load schema1 images without a problem.
Also, use skopeo to copy will maintain the digest. skopeo copy docker://quay.io/coreos/etcd-operator@sha256:66a37fd61a06a43969854ee6d3e21087a98b93838e284a6086b13917f96b0d9b dir:/tmp/foo/etcd-operator skopeo inspect dir:/tmp/foo/etcd-operator { "Tag": "v0.9.4", "Digest": "sha256:66a37fd61a06a43969854ee6d3e21087a98b93838e284a6086b13917f96b0d9b", "RepoTags": [], "Created": "2019-02-28T02:02:50.022009749Z", "DockerVersion": "18.09.2", "Labels": null, "Architecture": "amd64", "Os": "linux", "Layers": [ "sha256:59265c40e257554058624f35856dafd82d135c4ef406de298cb1fee647867381", "sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4", "sha256:8ea0315241b46a55ff650eddc8d06bfe7bfbdc072b4f3878d6c4598fc7b015d3", "sha256:678f2cfebea627b8b3d5bed91a6fe1d3749421fc504a568a7977ce912a953763", "sha256:e05d0d7eb99a1d91fa7349dd9a98a8b2d9e222b39fa305543e1ac4c380147a4b", "sha256:80b1b554e5b4c4a4334fd25764d6c3d78bafd04ef727714dfb25b1356cc3a265", "sha256:6d4b4c91b3ff6970e22c704f59d45a5945065cb80bd7055114f6d27432e2205a", "sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4" ], "Env": [ "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin" ] }
SKOPEO Can Not solve this problem, reproduce following: 1- use skopeo download image into localdisk [root@instance-micro ~]# skopeo copy docker://quay.io/coreos/etcd-operator@sha256:66a37fd61a06a43969854ee6d3e21087a98b93838e284a6086b13917f96b0d9b dir:./etcd-operator Getting image source signatures Copying blob 59265c40e257 done Copying blob a3ed95caeb02 done Copying blob 8ea0315241b4 done Copying blob 678f2cfebea6 done Copying blob e05d0d7eb99a done Copying blob 80b1b554e5b4 done Copying blob 6d4b4c91b3ff done Copying blob a3ed95caeb02 skipped: already exists Writing manifest to image destination Storing signatures 2- use skopeo push the local disk image file into my private registry server skopeo copy dir:./etcd-operator docker://localhost/coreos/etcd-operator Getting image source signatures Copying blob 59265c40e257 done Copying blob a3ed95caeb02 done Copying blob 8ea0315241b4 done Copying blob 678f2cfebea6 done Copying blob e05d0d7eb99a done Copying blob 80b1b554e5b4 done Copying blob 6d4b4c91b3ff done Copying blob a3ed95caeb02 skipped: already exists Writing manifest to image destination Storing signatures 3- check the pushed image's sha256 [root@instance-micro ~]# oc image info localhost/coreos/etcd-operator Name: localhost/coreos/etcd-operator:latest Digest: sha256:1fb5a1ea2b048fb604cb4fb4076fbc5356977423ca65006b3b8c89a8fbbcffb7 Media Type: application/vnd.docker.distribution.manifest.v1+prettyjws Created: 1y ago Image Size: 8 layers (size unavailable) Layers: -- sha256:59265c40e257554058624f35856dafd82d135c4ef406de298cb1fee647867381 -- sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4 -- sha256:8ea0315241b46a55ff650eddc8d06bfe7bfbdc072b4f3878d6c4598fc7b015d3 -- sha256:678f2cfebea627b8b3d5bed91a6fe1d3749421fc504a568a7977ce912a953763 -- sha256:e05d0d7eb99a1d91fa7349dd9a98a8b2d9e222b39fa305543e1ac4c380147a4b -- sha256:80b1b554e5b4c4a4334fd25764d6c3d78bafd04ef727714dfb25b1356cc3a265 -- sha256:6d4b4c91b3ff6970e22c704f59d45a5945065cb80bd7055114f6d27432e2205a -- sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4 OS: linux Arch: amd64 Command: /bin/sh User: etcd-operator Environment: PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin conclusion: You Will See the sha256 have been changed as sha256:1fb5a1ea2b048fb604cb4fb4076fbc5356977423ca65006b3b8c89a8fbbcffb7
Yes, going to back to `oc image mirror`, everything works as expected when mirroring between two registry. But when mirroring to a local file, the digest is not maintained.
Might be related to #02584003? When the image has certain amount of layers, last layers do not get mirrored for some reason.
Ignore the number in my comment #6. The right bug is https://bugzilla.redhat.com/show_bug.cgi?id=1797203
I am also try to use "skopeo copy --all", the SHA256 Still have been changed!!! [root@instance-micro ~]# skopeo copy --all docker://quay.io/coreos/etcd-operator@sha256:66a37fd61a06a43969854ee6d3e21087a98b93838e284a6086b13917f96b0d9b dir:./etcd-operator Getting image source signatures Copying blob 59265c40e257 done Copying blob a3ed95caeb02 done Copying blob 8ea0315241b4 done Copying blob 678f2cfebea6 done Copying blob e05d0d7eb99a done Copying blob 80b1b554e5b4 done Copying blob 6d4b4c91b3ff done Copying blob a3ed95caeb02 skipped: already exists Writing manifest to image destination Storing signatures [root@instance-micro ~]# skopeo copy --all dir:./etcd-operator docker://localhost/coreos/etcd-operator Getting image source signatures Copying blob 59265c40e257 skipped: already exists Copying blob a3ed95caeb02 skipped: already exists Copying blob 8ea0315241b4 skipped: already exists Copying blob 678f2cfebea6 skipped: already exists Copying blob e05d0d7eb99a skipped: already exists Copying blob 80b1b554e5b4 skipped: already exists Copying blob 6d4b4c91b3ff skipped: already exists Copying blob a3ed95caeb02 skipped: already exists Writing manifest to image destination Storing signatures [root@instance-micro ~]# oc image info localhost/coreos/etcd-operator Name: localhost/coreos/etcd-operator:latest Digest: sha256:1fb5a1ea2b048fb604cb4fb4076fbc5356977423ca65006b3b8c89a8fbbcffb7 Media Type: application/vnd.docker.distribution.manifest.v1+prettyjws Created: 1y ago Image Size: 8 layers (size unavailable) Layers: -- sha256:59265c40e257554058624f35856dafd82d135c4ef406de298cb1fee647867381 -- sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4 -- sha256:8ea0315241b46a55ff650eddc8d06bfe7bfbdc072b4f3878d6c4598fc7b015d3 -- sha256:678f2cfebea627b8b3d5bed91a6fe1d3749421fc504a568a7977ce912a953763 -- sha256:e05d0d7eb99a1d91fa7349dd9a98a8b2d9e222b39fa305543e1ac4c380147a4b -- sha256:80b1b554e5b4c4a4334fd25764d6c3d78bafd04ef727714dfb25b1356cc3a265 -- sha256:6d4b4c91b3ff6970e22c704f59d45a5945065cb80bd7055114f6d27432e2205a -- sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4 OS: linux Arch: amd64 Command: /bin/sh User: etcd-operator Environment: PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
FWIW, `skopeo copy` does not change the schema1 manifests _if_ the image is signed (and the signature is visible to Skopeo), or if the destination uses a digested reference. So, > skopeo copy docker://$src@$digest docker://$dest@$digest should work. But this is impossible with an intermediate `dir:` step.
… actually, let me correct that: > skopeo copy docker://… dir:… never modifies the manifest (unless extra options are used to explicitly request changes to the image) > skopeo copy dir:… docker://… does not modify schema1 manifests if the destination uses a digested reference. (But if you do need to create a tag for that image pushed by digest, Skopeo can’t do that.)
Summary: - schema version 1 contains the registry name as well as its tag - if either of those change the digest will change - schema version 2 is a pure content-reference, so digest will never change - older images (from before quay supported schema 2) are schema 1, and only way to update to schema 2 is to build/push new images. - with skopeo copy, if you don't modify name:tag while doing the copy, the schema1 digest will be preserved. ex: $ skopeo copy $source:tag $dest:tag will cause the manifest to be edited $ skopeo copy $source@digest $dest@digest will preserve Running with -v=8 I see this: $ oc image mirror quay.io/coreos/etcd-operator@sha256:66a37fd61a06a43969854ee6d3e21087a98b93838e284a6086b13917f96b0d9b --dir=/tmp/images file://coreos/etcd-operator -v=8 --- plan.go:344] Associated digest sha256:66a37fd61a06a43969854ee6d3e21087a98b93838e284a6086b13917f96b0d9b with converted digest sha256:db9eac85a5ca921e38713d0bbf8372b1d354538887b952dd547db0c86a82d2da sha256:db9eac85a5ca921e38713d0bbf8372b1d354538887b952dd547db0c86a82d2da file://coreos/etcd-operator info: Mirroring completed in 6.76s (7.791MB/s) You can see the original digest is logged w/ the converted digest. Should oc image mirror be updated to force preserving the original digest? Not sure, will discuss among workloads team and report back.
This is an issue with schema v1 images in general. We are adding a warning in 'oc image mirror' that digests are only guaranteed to remain the same w/ V2 images. Also, we'll add a warning that states support for schema V1 images will be dropped in the future. Once those changes are in oc code, I'll be closing this bug as there is no 'fix' other than move all images to V2. Those warnings will be added in the upcoming sprint.
If we cannot maintain the digest by oc image mirror command, I recommend the operator which images base on schema v1 would not use sha256 (such as etcd operator)instead use tag, I have found some images in Community Channel in OperatorHub use schema v1
(In reply to kevin from comment #13) > If we cannot maintain the digest by oc image mirror command, I recommend the > operator which images base on schema v1 would not use sha256 (such as etcd > operator)instead use tag Note that tags don’t work with the way disconnected clusters use imageContentSourcePoilcy, so if the operator hub disconnected access depends on ICSP, that’s not an option.
Confirmed with oc [root@dhcp-140-138 ~]# oc version --client Client Version: 4.6.0-202007171623.p0-c33851e We could see the warning now: [root@dhcp-140-138 ~]# oc image mirror quay.io/coreos/etcd-operator@sha256:66a37fd61a06a43969854ee6d3e21087a98b93838e284a6086b13917f96b0d9b --dir=/tmp/images file://coreos/etcd-operator ....... warning: Digests are not preserved with schema version 1 images. Support for schema version 1 images will be removed in a future release. sha256:db9eac85a5ca921e38713d0bbf8372b1d354538887b952dd547db0c86a82d2da file://coreos/etcd-operator info: Mirroring completed in 12.93s (4.072MB/s)
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (OpenShift Container Platform 4.6 GA Images), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2020:4196