Bug 1850931
Summary: | oc image mirror into localdisk will change the original sha256 in manifest schemaVersion1 images | ||
---|---|---|---|
Product: | OpenShift Container Platform | Reporter: | kevin <welin> |
Component: | oc | Assignee: | Sally <somalley> |
Status: | CLOSED ERRATA | QA Contact: | zhou ying <yinzhou> |
Severity: | high | Docs Contact: | |
Priority: | medium | ||
Version: | 4.4 | CC: | aos-bugs, cdoan, jokerman, lmaly, mfojtik, mitr, susuresh |
Target Milestone: | --- | ||
Target Release: | 4.6.0 | ||
Hardware: | x86_64 | ||
OS: | Linux | ||
Whiteboard: | |||
Fixed In Version: | Doc Type: | No Doc Update | |
Doc Text: | Story Points: | --- | |
Clone Of: | Environment: | ||
Last Closed: | 2020-10-27 16:09:42 UTC | Type: | Bug |
Regression: | --- | Mount Type: | --- |
Documentation: | --- | CRM: | |
Verified Versions: | Category: | --- | |
oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |
Cloudforms Team: | --- | Target Upstream Version: | |
Embargoed: |
Description
kevin
2020-06-25 08:27:36 UTC
In OCP operatorHub, I have found many "schema version 1 image" as follwing: quay.io/coreos/etcd-operator@sha256:66a37fd61a06a43969854ee6d3e21087a98b93838e284a6086b13917f96b0d9b=registry.example.internal:5000/coreos/etcd-operator:latest This image's schema version is 1 quay.io/coreos/etcd-operator@sha256:bd944a211eaf8f31da5e6d69e8541e7cada8f16a9f7a5a570b22478997819943=registry.example.internal:5000/coreos/etcd-operator:latest This image's schema version is 1 quay.io/coreos/etcd-operator@sha256:c0301e4686c3ed4206e370b42de5a3bd2229b9fb4906cf85f3f30650424abec2=registry.example.internal:5000/coreos/etcd-operator:latest This image's schema version is 1 quay.io/coreos/etcd-operator@sha256:db563baa8194fcfe39d1df744ed70024b0f1f9e9b55b5923c2f3a413c44dc6b8=registry.example.internal:5000/coreos/etcd-operator:latest This image's schema version is 1 quay.io/coreos/prometheus-operator@sha256:0e92dd9b5789c4b13d53e1319d0a6375bcca4caaf0d698af61198061222a576d=registry.example.internal:5000/coreos/prometheus-operator:latest This image's schema version is 1 quay.io/coreos/prometheus-operator@sha256:3daa69a8c6c2f1d35dcf1fe48a7cd8b230e55f5229a1ded438f687debade5bcf=registry.example.internal:5000/coreos/prometheus-operator:latest This image's schema version is 1 quay.io/coreos/prometheus-operator@sha256:5037b4e90dbb03ebdefaa547ddf6a1f748c8eeebeedf6b9d9f0913ad662b5731=registry.example.internal:5000/coreos/prometheus-operator:latest This image's schema version is 1 quay.io/coreos/prometheus-operator@sha256:933cd5bf380cf7db330808ff54f75f26fda0b1501021d499a1766b7d16224188=registry.example.internal:5000/coreos/prometheus-operator:latest This image's schema version is 1 quay.io/coreos/prometheus-operator@sha256:ed3ec0597c2d5b7102a7f62c661a23d8e4b34d910693fc23fd40bfb1d9404dcf=registry.example.internal:5000/coreos/prometheus-operator:latest This image's schema version is 1 quay.io/jmckind/argocd-operator@sha256:0fa4b7709e1e9c9cb9ca064be50618f71ff3eef07e185c631b1227f8e5a57776=registry.example.internal:5000/jmckind/argocd-operator:8.1-328_linux-amd64 This image's schema version is 1 quay.io/jmckind/argocd-operator@sha256:5d7c4b0e8e0fea068e49a9718a35ae068fc267e607f7393374db39916d7186f4=registry.example.internal:5000/jmckind/argocd-operator:7.7-211_linux-amd64 This image's schema version is 1 quay.io/jmckind/argocd-operator@sha256:d1385d23a60205636bc3789b0127d6159d33d7a7521dd07d6b679b7f734ee4b3=registry.example.internal:5000/jmckind/argocd-operator:7.7-211_linux-amd64 This image's schema version is 1 quay.io/jmckind/argocd-operator@sha256:fd7aaf9a0b330d5f646aa69933c8149de60b680878208f473543fd3c43412096=registry.example.internal:5000/jmckind/argocd-operator:7.7-211_linux-amd64 This image's schema version is 1 quay.io/quay/container-security-operator@sha256:154d7e0295a94fb3d2a97309d711186a98a7308da37a5cd3d50360c6b2ba57de=registry.example.internal:5000/quay/container-security-operator:latest This image's schema version is 1 quay.io/quay/container-security-operator@sha256:15a4b50d847512b5f404ec1cf72c30c98e073a7f26f1588213bd2e8b6331f016=registry.example.internal:5000/quay/container-security-operator:latest This image's schema version is 1 quay.io/quay/container-security-operator@sha256:6eefeaee910251ba26c825746d11ae166a9781aeace5455b2766d26298911f13=registry.example.internal:5000/quay/container-security-operator:latest This image's schema version is 1 quay.io/quay/container-security-operator@sha256:7998f9377973cdc22d8ad713ba1b81381db9782a4b58d4c89f4bed688e2ff461=registry.example.internal:5000/quay/container-security-operator:latest This image's schema version is 1 The etcd images in quay have the schema1 manifests. In deploying RHACM 1.0 in disconnected install mode, we go through the disconnected OLM procedure to mirror down the the two operator catalogs--Red Hat Operators, and Community Operators, into a mirror registry. First, the mirror registry needs to allow schema1 compatibility. We can configure this like the following commands, adding the compatibility stanza to the registry config: # podman exec -it registry /bin/sh # vi etc/docker/registry/config.yml compatibility: schema1: enabled: true # podman restart registry Then, you should be able to load schema1 images without a problem. Also, use skopeo to copy will maintain the digest. skopeo copy docker://quay.io/coreos/etcd-operator@sha256:66a37fd61a06a43969854ee6d3e21087a98b93838e284a6086b13917f96b0d9b dir:/tmp/foo/etcd-operator skopeo inspect dir:/tmp/foo/etcd-operator { "Tag": "v0.9.4", "Digest": "sha256:66a37fd61a06a43969854ee6d3e21087a98b93838e284a6086b13917f96b0d9b", "RepoTags": [], "Created": "2019-02-28T02:02:50.022009749Z", "DockerVersion": "18.09.2", "Labels": null, "Architecture": "amd64", "Os": "linux", "Layers": [ "sha256:59265c40e257554058624f35856dafd82d135c4ef406de298cb1fee647867381", "sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4", "sha256:8ea0315241b46a55ff650eddc8d06bfe7bfbdc072b4f3878d6c4598fc7b015d3", "sha256:678f2cfebea627b8b3d5bed91a6fe1d3749421fc504a568a7977ce912a953763", "sha256:e05d0d7eb99a1d91fa7349dd9a98a8b2d9e222b39fa305543e1ac4c380147a4b", "sha256:80b1b554e5b4c4a4334fd25764d6c3d78bafd04ef727714dfb25b1356cc3a265", "sha256:6d4b4c91b3ff6970e22c704f59d45a5945065cb80bd7055114f6d27432e2205a", "sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4" ], "Env": [ "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin" ] } SKOPEO Can Not solve this problem, reproduce following: 1- use skopeo download image into localdisk [root@instance-micro ~]# skopeo copy docker://quay.io/coreos/etcd-operator@sha256:66a37fd61a06a43969854ee6d3e21087a98b93838e284a6086b13917f96b0d9b dir:./etcd-operator Getting image source signatures Copying blob 59265c40e257 done Copying blob a3ed95caeb02 done Copying blob 8ea0315241b4 done Copying blob 678f2cfebea6 done Copying blob e05d0d7eb99a done Copying blob 80b1b554e5b4 done Copying blob 6d4b4c91b3ff done Copying blob a3ed95caeb02 skipped: already exists Writing manifest to image destination Storing signatures 2- use skopeo push the local disk image file into my private registry server skopeo copy dir:./etcd-operator docker://localhost/coreos/etcd-operator Getting image source signatures Copying blob 59265c40e257 done Copying blob a3ed95caeb02 done Copying blob 8ea0315241b4 done Copying blob 678f2cfebea6 done Copying blob e05d0d7eb99a done Copying blob 80b1b554e5b4 done Copying blob 6d4b4c91b3ff done Copying blob a3ed95caeb02 skipped: already exists Writing manifest to image destination Storing signatures 3- check the pushed image's sha256 [root@instance-micro ~]# oc image info localhost/coreos/etcd-operator Name: localhost/coreos/etcd-operator:latest Digest: sha256:1fb5a1ea2b048fb604cb4fb4076fbc5356977423ca65006b3b8c89a8fbbcffb7 Media Type: application/vnd.docker.distribution.manifest.v1+prettyjws Created: 1y ago Image Size: 8 layers (size unavailable) Layers: -- sha256:59265c40e257554058624f35856dafd82d135c4ef406de298cb1fee647867381 -- sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4 -- sha256:8ea0315241b46a55ff650eddc8d06bfe7bfbdc072b4f3878d6c4598fc7b015d3 -- sha256:678f2cfebea627b8b3d5bed91a6fe1d3749421fc504a568a7977ce912a953763 -- sha256:e05d0d7eb99a1d91fa7349dd9a98a8b2d9e222b39fa305543e1ac4c380147a4b -- sha256:80b1b554e5b4c4a4334fd25764d6c3d78bafd04ef727714dfb25b1356cc3a265 -- sha256:6d4b4c91b3ff6970e22c704f59d45a5945065cb80bd7055114f6d27432e2205a -- sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4 OS: linux Arch: amd64 Command: /bin/sh User: etcd-operator Environment: PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin conclusion: You Will See the sha256 have been changed as sha256:1fb5a1ea2b048fb604cb4fb4076fbc5356977423ca65006b3b8c89a8fbbcffb7 Yes, going to back to `oc image mirror`, everything works as expected when mirroring between two registry. But when mirroring to a local file, the digest is not maintained. Might be related to #02584003? When the image has certain amount of layers, last layers do not get mirrored for some reason. Ignore the number in my comment #6. The right bug is https://bugzilla.redhat.com/show_bug.cgi?id=1797203 I am also try to use "skopeo copy --all", the SHA256 Still have been changed!!! [root@instance-micro ~]# skopeo copy --all docker://quay.io/coreos/etcd-operator@sha256:66a37fd61a06a43969854ee6d3e21087a98b93838e284a6086b13917f96b0d9b dir:./etcd-operator Getting image source signatures Copying blob 59265c40e257 done Copying blob a3ed95caeb02 done Copying blob 8ea0315241b4 done Copying blob 678f2cfebea6 done Copying blob e05d0d7eb99a done Copying blob 80b1b554e5b4 done Copying blob 6d4b4c91b3ff done Copying blob a3ed95caeb02 skipped: already exists Writing manifest to image destination Storing signatures [root@instance-micro ~]# skopeo copy --all dir:./etcd-operator docker://localhost/coreos/etcd-operator Getting image source signatures Copying blob 59265c40e257 skipped: already exists Copying blob a3ed95caeb02 skipped: already exists Copying blob 8ea0315241b4 skipped: already exists Copying blob 678f2cfebea6 skipped: already exists Copying blob e05d0d7eb99a skipped: already exists Copying blob 80b1b554e5b4 skipped: already exists Copying blob 6d4b4c91b3ff skipped: already exists Copying blob a3ed95caeb02 skipped: already exists Writing manifest to image destination Storing signatures [root@instance-micro ~]# oc image info localhost/coreos/etcd-operator Name: localhost/coreos/etcd-operator:latest Digest: sha256:1fb5a1ea2b048fb604cb4fb4076fbc5356977423ca65006b3b8c89a8fbbcffb7 Media Type: application/vnd.docker.distribution.manifest.v1+prettyjws Created: 1y ago Image Size: 8 layers (size unavailable) Layers: -- sha256:59265c40e257554058624f35856dafd82d135c4ef406de298cb1fee647867381 -- sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4 -- sha256:8ea0315241b46a55ff650eddc8d06bfe7bfbdc072b4f3878d6c4598fc7b015d3 -- sha256:678f2cfebea627b8b3d5bed91a6fe1d3749421fc504a568a7977ce912a953763 -- sha256:e05d0d7eb99a1d91fa7349dd9a98a8b2d9e222b39fa305543e1ac4c380147a4b -- sha256:80b1b554e5b4c4a4334fd25764d6c3d78bafd04ef727714dfb25b1356cc3a265 -- sha256:6d4b4c91b3ff6970e22c704f59d45a5945065cb80bd7055114f6d27432e2205a -- sha256:a3ed95caeb02ffe68cdd9fd84406680ae93d633cb16422d00e8a7c22955b46d4 OS: linux Arch: amd64 Command: /bin/sh User: etcd-operator Environment: PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin FWIW, `skopeo copy` does not change the schema1 manifests _if_ the image is signed (and the signature is visible to Skopeo), or if the destination uses a digested reference.
So,
> skopeo copy docker://$src@$digest docker://$dest@$digest
should work. But this is impossible with an intermediate `dir:` step.
… actually, let me correct that: > skopeo copy docker://… dir:… never modifies the manifest (unless extra options are used to explicitly request changes to the image) > skopeo copy dir:… docker://… does not modify schema1 manifests if the destination uses a digested reference. (But if you do need to create a tag for that image pushed by digest, Skopeo can’t do that.) Summary: - schema version 1 contains the registry name as well as its tag - if either of those change the digest will change - schema version 2 is a pure content-reference, so digest will never change - older images (from before quay supported schema 2) are schema 1, and only way to update to schema 2 is to build/push new images. - with skopeo copy, if you don't modify name:tag while doing the copy, the schema1 digest will be preserved. ex: $ skopeo copy $source:tag $dest:tag will cause the manifest to be edited $ skopeo copy $source@digest $dest@digest will preserve Running with -v=8 I see this: $ oc image mirror quay.io/coreos/etcd-operator@sha256:66a37fd61a06a43969854ee6d3e21087a98b93838e284a6086b13917f96b0d9b --dir=/tmp/images file://coreos/etcd-operator -v=8 --- plan.go:344] Associated digest sha256:66a37fd61a06a43969854ee6d3e21087a98b93838e284a6086b13917f96b0d9b with converted digest sha256:db9eac85a5ca921e38713d0bbf8372b1d354538887b952dd547db0c86a82d2da sha256:db9eac85a5ca921e38713d0bbf8372b1d354538887b952dd547db0c86a82d2da file://coreos/etcd-operator info: Mirroring completed in 6.76s (7.791MB/s) You can see the original digest is logged w/ the converted digest. Should oc image mirror be updated to force preserving the original digest? Not sure, will discuss among workloads team and report back. This is an issue with schema v1 images in general. We are adding a warning in 'oc image mirror' that digests are only guaranteed to remain the same w/ V2 images. Also, we'll add a warning that states support for schema V1 images will be dropped in the future. Once those changes are in oc code, I'll be closing this bug as there is no 'fix' other than move all images to V2. Those warnings will be added in the upcoming sprint. If we cannot maintain the digest by oc image mirror command, I recommend the operator which images base on schema v1 would not use sha256 (such as etcd operator)instead use tag, I have found some images in Community Channel in OperatorHub use schema v1 (In reply to kevin from comment #13) > If we cannot maintain the digest by oc image mirror command, I recommend the > operator which images base on schema v1 would not use sha256 (such as etcd > operator)instead use tag Note that tags don’t work with the way disconnected clusters use imageContentSourcePoilcy, so if the operator hub disconnected access depends on ICSP, that’s not an option. Confirmed with oc [root@dhcp-140-138 ~]# oc version --client Client Version: 4.6.0-202007171623.p0-c33851e We could see the warning now: [root@dhcp-140-138 ~]# oc image mirror quay.io/coreos/etcd-operator@sha256:66a37fd61a06a43969854ee6d3e21087a98b93838e284a6086b13917f96b0d9b --dir=/tmp/images file://coreos/etcd-operator ....... warning: Digests are not preserved with schema version 1 images. Support for schema version 1 images will be removed in a future release. sha256:db9eac85a5ca921e38713d0bbf8372b1d354538887b952dd547db0c86a82d2da file://coreos/etcd-operator info: Mirroring completed in 12.93s (4.072MB/s) Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (OpenShift Container Platform 4.6 GA Images), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2020:4196 |