Description of problem: `oc adm release mirror` has started uploading some manifests as schema 1. This was due to a bug in OCP builds pushing manifests that did not conform to the docker v2schema2 manifest spec. See https://bugzilla.redhat.com/show_bug.cgi?id=1905095 Once the dependent issue is fixed in OCP and backported to 4.6.z, our test platform needs to be updated to the patched z-stream and rebuild all images in the OCP/OKD payload. Once all images are rebuilt, they then can be mirrored to quay.io. Version-Release number of selected component (if applicable): Latest from https://mirror.openshift.com/pub/openshift-v4/x86_64/clients/ocp/stable/openshift-client-linux.tar.gz Client Version: 4.6.6 How reproducible: Always Steps to Reproduce: 1. Get pullspec of the image from payload: $ RELEASE="4.6.0-0.okd-2020-12-07-083734" $ oc adm release info --image-for aws-ebs-csi-driver-operator registry.svc.ci.openshift.org/origin/release:${RELEASE} registry.svc.ci.openshift.org/origin/4.6-2020-12-07-083734@sha256:52f962cc969eaf5ab3c94e8c87eea75f12310dbde6ed3b0f7596c6d7200de08f 2. Check its schema version $ skopeo inspect --authfile ~/src/github.com/vrutkovs/okd-installer/pull_secret.json docker://registry.svc.ci.openshift.org/origin/4.6-2020-12-07-083734@sha256:52f962cc969eaf5ab3c94e8c87eea75f12310dbde6ed3b0f7596c6d7200de08f --raw | jq '.schemaVersion' 2 (as expected) 3. Mirror the release: $ oc adm -a /run/user/0/containers/auth.json release new --from-release registry.svc.ci.openshift.org/origin/release:${RELEASE} --mirror quay.io/openshift/okd-content --to-image quay.io/vrutkovs/okd-release:${RELEASE} --name=${RELEASE} 4. Check schema version of uploaded image: $ oc adm release info --image-for aws-ebs-csi-driver-operator quay.io/vrutkovs/okd-release:${RELEASE} quay.io/vrutkovs/okd-release@sha256:ce862b5f752bea5fdd0d2c3b197c3b0362e02e3e433ad168bfc8ca380082e429 $ skopeo inspect --authfile ~/src/github.com/vrutkovs/okd-installer/pull_secret.json docker://quay.io/vrutkovs/okd-release@sha256:ce862b5f752bea5fdd0d2c3b197c3b0362e02e3e433ad168bfc8ca380082e429 --raw | jq '.schemaVersion' 1 Actual results: Some images are uploaded as schema 1 and throw signature verification errors during upload (as schema 1 digest doesn't match the expected) Expected results: All manifests are uploaded as schema 2
@akaplan what version of OCP must we be running before we trigger the mass-rebuild?
@Steve the following versions have the fix in: 1. OCP 4.7.4 or higher 2. OCP 4.6.23 or higher