Description of problem: `oc adm release mirror` has started uploading some manifests as schema 1 Version-Release number of selected component (if applicable): Latest from https://mirror.openshift.com/pub/openshift-v4/x86_64/clients/ocp/stable/openshift-client-linux.tar.gz Client Version: 4.6.6 How reproducible: Always Steps to Reproduce: 1. Get pullspec of the image from payload: $ RELEASE="4.6.0-0.okd-2020-12-07-083734" $ oc adm release info --image-for aws-ebs-csi-driver-operator registry.svc.ci.openshift.org/origin/release:${RELEASE} registry.svc.ci.openshift.org/origin/4.6-2020-12-07-083734@sha256:52f962cc969eaf5ab3c94e8c87eea75f12310dbde6ed3b0f7596c6d7200de08f 2. Check its schema version $ skopeo inspect --authfile ~/src/github.com/vrutkovs/okd-installer/pull_secret.json docker://registry.svc.ci.openshift.org/origin/4.6-2020-12-07-083734@sha256:52f962cc969eaf5ab3c94e8c87eea75f12310dbde6ed3b0f7596c6d7200de08f --raw | jq '.schemaVersion' 2 (as expected) 3. Mirror the release: $ oc adm -a /run/user/0/containers/auth.json release new --from-release registry.svc.ci.openshift.org/origin/release:${RELEASE} --mirror quay.io/openshift/okd-content --to-image quay.io/vrutkovs/okd-release:${RELEASE} --name=${RELEASE} 4. Check schema version of uploaded image: $ oc adm release info --image-for aws-ebs-csi-driver-operator quay.io/vrutkovs/okd-release:${RELEASE} quay.io/vrutkovs/okd-release@sha256:ce862b5f752bea5fdd0d2c3b197c3b0362e02e3e433ad168bfc8ca380082e429 $ skopeo inspect --authfile ~/src/github.com/vrutkovs/okd-installer/pull_secret.json docker://quay.io/vrutkovs/okd-release@sha256:ce862b5f752bea5fdd0d2c3b197c3b0362e02e3e433ad168bfc8ca380082e429 --raw | jq '.schemaVersion' 1 Actual results: Some images are uploaded as schema 1 and throw signature verification errors during upload (as schema 1 digest doesn't match the expected) Expected results: All manifests are uploaded as schema 2 Additional info:
Created attachment 1737353 [details] oc adm mirror from quay to quay with v=5
Comment on attachment 1737353 [details] oc adm mirror from quay to quay with v=5 oc -v=5 adm -a ~/src/github.com/vrutkovs/okd-installer/pull_secret_quay.json release mirror \ --from=quay.io/openshift/okd:4.6.0-0.okd-2020-11-27-200126 \ --to=quay.io/vrutkovs/okd-release \ --to-release-image=quay.io/vrutkovs/okd-release:4.6-mirroring-bug
Created attachment 1737369 [details] oc adm mirror from registry.svc to quay >I1207 16:50:22.072264 1595705 manifest.go:498] Registry reported invalid manifest error, attempting to convert to v2schema1 as ref vrutkovs/okd-release:4.6.0- 0.okd-2020-12-07-083734-jenkins
Seems Quay is rejecting some images and oc tries to push v2schema1: ``` I1207 17:57:09.238677 1620987 manifest.go:461] Put manifest vrutkovs/okd-release:4.6.0-0.okd-2020-12-07-083734-oauth-proxy I1207 17:57:09.238757 1620987 round_trippers.go:423] curl -k -v -XPUT -H "Content-Type: application/vnd.docker.distribution.manifest.v2+json" -H "Authorization: Bearer ..." 'https://quay.io/v2/vrutkovs/okd-release/manifests/4.6.0-0.okd-2020-12-07-083734-oauth-proxy' I1207 17:57:09.336905 1620987 round_trippers.go:443] PUT https://quay.io/v2/vrutkovs/okd-release/manifests/4.6.0-0.okd-2020-12-07-083734-machine-config-operator 400 Bad Request in 98 milliseconds I1207 17:57:09.337047 1620987 round_trippers.go:449] Response Headers: I1207 17:57:09.337084 1620987 round_trippers.go:452] Date: Mon, 07 Dec 2020 17:57:09 GMT I1207 17:57:09.337130 1620987 round_trippers.go:452] Content-Type: application/json I1207 17:57:09.337160 1620987 round_trippers.go:452] Content-Length: 979 I1207 17:57:09.337398 1620987 round_trippers.go:452] Server: nginx/1.12.1 I1207 17:57:09.337900 1620987 manifest.go:498] Registry reported invalid manifest error, attempting to convert to v2schema1 as ref vrutkovs/okd-release:4.6.0-0.okd-2020-12-07-083734-machine-config-operator ```
Apparently some images have broken manifest: ``` $ skopeo inspect docker://registry.ci.openshift.org/origin/4.6:tools ... "Env": [ "GODEBUG=x509ignoreCN=0", ... $ oc -v=10 -a ~/src/github.com/vrutkovs/okd-installer/pull_secret_quay.json image mirror registry.ci.openshift.org/origin/4.6:tools quay.io/vrutkovs/okd-release .... PUT https://quay.io/v2/vrutkovs/okd-release/manifests/tools 400 Bad Request in 207 milliseconds ... ``` This was built on 4.6.7 Some, like `registry.ci.openshift.org/origin/4.6:thanos` were built on 4.5 and are mirrored correctly
{"errors":[{"code":"MANIFEST_INVALID","detail":{"message":"failed to parse manifest: manifest data does not match schema: u'application/vnd.docker.image.rootfs.diff.tar' is not one of ['application/vnd.docker.image.rootfs.diff.tar.gzip', 'application/vnd.docker.image.rootfs.foreign.diff.tar.gzip']\n\nFailed validating 'enum' in schema['properties']['layers']['items']['properties']['mediaType']:\n {'description': 'The MIME type of the referenced object. This should generally be application/vnd.docker.image.rootfs.diff.tar.gzip. Layers of type application/vnd.docker.image.rootfs.foreign.diff.tar.gzip may be pulled from a remote location but they should never be pushed.',\n 'enum': ['applica… from quay output, which seems to be https://github.com/containers/image/issues/733
Nalin has a PR up for https://github.com/containers/image/issues/733 Ultimately, we'll need to bump the buildah dep in openshift/builder to pick it up
Summary of issue: 1. OCP CI environments (4.6 OSD) import UBI and build a base image - that is done using buildah and the images already exist in the target registry (since base is super common), so the buildah bug is triggered 2. The base image is used as the parent for all CI images created on those clusters for all CI created releases (4.2 -> 4.8) 3. The CI tests use those images directly and appear to pull and mirror correctly within our CI clusters 4. The job that mirrors CI images to quay for OKD and developer access is rejected by quay for those images (rejects the application/vnd.docker.image.rootfs.diff.tar media type on the layer), which causes oc image mirror to fall back to a v2schema1 image, which causes the manifest to change 5. Payloads for OKD cannot be created for release that preserve SHAs because of this It is too hard to spot these errors in the mirror command, so the mirror command should be printing the quay.io error message about the invalid manifest. We need a fix to 4.6 and then rolled out to OSD in order to get out of this. Other components may fail to work with these images (although no evidence of that so far except a 3.11 registry reports a 500 when this image is pushed)
Progress on this issue is being made in https://github.com/containers/image/pull/1089
*** Bug 1908908 has been marked as a duplicate of this bug. ***
Moving this back to assigned. Once the we merge the upstream PR, we'll need a PR for the client, and then it'll be appropriate to move this back to POST, I think.
*** Bug 1937433 has been marked as a duplicate of this bug. ***
Current fix is awaiting CI verification. The following buildah dependencies are to be updated: 1. containers/image -> v5.10.5 2. containers/storage -> v1.24.8
Could reproduce it with builder image in ocp 4.5(will check on 4.6 tomorrow). steps: Push a builder image, $oc new-app ruby:latest~http://github.com/openshift/rails-ex.git Check the image schemeVersion $skopeo inspect docker://default-route-openshift-image-registry.apps.knarra4517.0317-3t7.qe.rhcloud.com/default/rails-ex:latest --raw | jq '.schemaVersion' 2 Mirror image to quay.io $oc image mirror default-route-openshift-image-registry.apps.knarra4517.0317-3t7.qe.rhcloud.com/default/rails-ex:latest quay.io/openshifttest/ocp:45 -a ~/.docker/quay-xiuwang Check the quay.io image schemaVersion $skopeo inspect docker://quay.io/openshifttest/ocp:45 --creds xiuwang:******* --raw | jq '.schemaVersion' 1 ======================================================================================================================== The bug fixs on ocp 4.8.0-0.nightly-2021-03-17-014745 cluster, validate it with same steps as above. $skopeo inspect docker://default-route-openshift-image-registry.apps.xxia17.qe.devcluster.openshift.com/xiuwang/ruby-hello-world:latest --raw | jq '.schemaVersion' 2 $oc image mirror default-route-openshift-image-registry.apps.xxia17.qe.devcluster.openshift.com/xiuwang/ruby-hello-world:latest quay.io/openshifttest/test:myimage -a ~/.docker/quay-xiuwang quay.io/ openshifttest/test blobs: default-route-openshift-image-registry.apps.xxia17.qe.devcluster.openshift.com/xiuwang/ruby-hello-world sha256:a858833a9239708c0c07c8fdf95218065c0605e14950051b009f9ad263f43511 1.765KiB default-route-openshift-image-registry.apps.xxia17.qe.devcluster.openshift.com/xiuwang/ruby-hello-world sha256:e10f2f601be71b985b78a7cb9002d952d8a30f8a5e526dd7265dbdc84b2da038 12.15KiB default-route-openshift-image-registry.apps.xxia17.qe.devcluster.openshift.com/xiuwang/ruby-hello-world sha256:608083cad0129a0f9240e5dcd4ceb087cc5ff025012277fc28bd77108e11a9bd 6.886MiB default-route-openshift-image-registry.apps.xxia17.qe.devcluster.openshift.com/xiuwang/ruby-hello-world sha256:c9caf8a93d59da90dcdfe05e92175baa22e7d6e443a95ffbbea3684e85bf46ba 10.55MiB default-route-openshift-image-registry.apps.xxia17.qe.devcluster.openshift.com/xiuwang/ruby-hello-world sha256:4f1355d64ea65ae6566038612e6e4d2d7384f5ac0323d7ca00f03cd84a9d6233 14MiB default-route-openshift-image-registry.apps.xxia17.qe.devcluster.openshift.com/xiuwang/ruby-hello-world sha256:b77f42d650dc7d0d6fa21f8661f03957cfe70fcf92e48245d2a7cad7d795eb56 72.89MiB default-route-openshift-image-registry.apps.xxia17.qe.devcluster.openshift.com/xiuwang/ruby-hello-world sha256:d15e5a5c8e28d36e53056a430b5aeb6a0d3fea187ebada478dd7cbb5524221bf 83.85MiB manifests: sha256:01c3e02a429e8096a9bb00653417691931562d260e3399d6b8520a020a613bae -> myimage stats: shared=0 unique=7 size=188.2MiB ratio=1.00 phase 0: quay.io openshifttest/test blobs=7 mounts=0 manifests=1 shared=0 info: Planning completed in 2.95s uploading: quay.io/openshifttest/test sha256:c9caf8a93d59da90dcdfe05e92175baa22e7d6e443a95ffbbea3684e85bf46ba 10.55MiB uploading: quay.io/openshifttest/test sha256:b77f42d650dc7d0d6fa21f8661f03957cfe70fcf92e48245d2a7cad7d795eb56 72.89MiB uploading: quay.io/openshifttest/test sha256:4f1355d64ea65ae6566038612e6e4d2d7384f5ac0323d7ca00f03cd84a9d6233 14MiB uploading: quay.io/openshifttest/test sha256:608083cad0129a0f9240e5dcd4ceb087cc5ff025012277fc28bd77108e11a9bd 6.886MiB uploading: quay.io/openshifttest/test sha256:d15e5a5c8e28d36e53056a430b5aeb6a0d3fea187ebada478dd7cbb5524221bf 83.85MiB sha256:01c3e02a429e8096a9bb00653417691931562d260e3399d6b8520a020a613bae quay.io/openshifttest/test:myimage info: Mirroring completed in 5m48.76s (565.8kB/s) $skopeo inspect docker://quay.io/openshifttest/test:myimage --creds xiuwang:******* --raw | jq '.schemaVersion' 2
Current fix is awaiting CI verification, the feature fixs are merged. Move to on_qa manually.
Per comment #25, and do regression test for build features on ocp 4.8.0-0.nightly-2021-03-17-014745 cluster, no new issue found. We could mark this bug as verified.
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory (Moderate: OpenShift Container Platform 4.8.2 bug fix and security update), and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHSA-2021:2438