Bug 1940203 - oc image mirror degrades to schema1 which loses consistent SHA's and prevents OKD releases
Summary: oc image mirror degrades to schema1 which loses consistent SHA's and prevents...
Keywords:
Status: CLOSED WONTFIX
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Test Infrastructure
Version: 4.6
Hardware: Unspecified
OS: Unspecified
urgent
high
Target Milestone: ---
: 4.6.z
Assignee: Steve Kuznetsov
QA Contact:
URL:
Whiteboard:
Depends On: 1939218 1939219
Blocks:
TreeView+ depends on / blocked
 
Reported: 2021-03-17 19:40 UTC by Adam Kaplan
Modified: 2022-08-24 15:27 UTC (History)
19 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of: 1939219
Environment:
Last Closed: 2022-08-24 15:27:23 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)

Description Adam Kaplan 2021-03-17 19:40:26 UTC
Description of problem:
`oc adm release mirror` has started uploading some manifests as schema 1. This was due to a bug in OCP builds pushing manifests that did not conform to the docker v2schema2 manifest spec. See https://bugzilla.redhat.com/show_bug.cgi?id=1905095

Once the dependent issue is fixed in OCP and backported to 4.6.z, our test platform needs to be updated to the patched z-stream and rebuild all images in the OCP/OKD payload.
Once all images are rebuilt, they then can be mirrored to quay.io.

Version-Release number of selected component (if applicable):
Latest from https://mirror.openshift.com/pub/openshift-v4/x86_64/clients/ocp/stable/openshift-client-linux.tar.gz
Client Version: 4.6.6

How reproducible:
Always

Steps to Reproduce:
1. Get pullspec of the image from payload:

$ RELEASE="4.6.0-0.okd-2020-12-07-083734"
$ oc adm release info --image-for aws-ebs-csi-driver-operator registry.svc.ci.openshift.org/origin/release:${RELEASE}
registry.svc.ci.openshift.org/origin/4.6-2020-12-07-083734@sha256:52f962cc969eaf5ab3c94e8c87eea75f12310dbde6ed3b0f7596c6d7200de08f

2. Check its schema version

$ skopeo inspect --authfile ~/src/github.com/vrutkovs/okd-installer/pull_secret.json docker://registry.svc.ci.openshift.org/origin/4.6-2020-12-07-083734@sha256:52f962cc969eaf5ab3c94e8c87eea75f12310dbde6ed3b0f7596c6d7200de08f --raw | jq '.schemaVersion'
2

(as expected)

3. Mirror the release:

$ oc adm -a /run/user/0/containers/auth.json release new --from-release registry.svc.ci.openshift.org/origin/release:${RELEASE} --mirror quay.io/openshift/okd-content --to-image quay.io/vrutkovs/okd-release:${RELEASE} --name=${RELEASE}

4. Check schema version of uploaded image:

$ oc adm release info --image-for aws-ebs-csi-driver-operator quay.io/vrutkovs/okd-release:${RELEASE}
quay.io/vrutkovs/okd-release@sha256:ce862b5f752bea5fdd0d2c3b197c3b0362e02e3e433ad168bfc8ca380082e429
$ skopeo inspect --authfile ~/src/github.com/vrutkovs/okd-installer/pull_secret.json docker://quay.io/vrutkovs/okd-release@sha256:ce862b5f752bea5fdd0d2c3b197c3b0362e02e3e433ad168bfc8ca380082e429 --raw | jq '.schemaVersion'
1


Actual results:
Some images are uploaded as schema 1 and throw signature verification errors during upload (as schema 1 digest doesn't match the expected)


Expected results:
All manifests are uploaded as schema 2

Comment 2 Steve Kuznetsov 2021-05-17 19:23:39 UTC
@akaplan what version of OCP must we be running before we trigger the mass-rebuild?

Comment 3 Adam Kaplan 2021-05-18 12:05:52 UTC
@Steve the following versions have the fix in:

1. OCP 4.7.4 or higher
2. OCP 4.6.23 or higher


Note You need to log in before you can comment on or make changes to this bug.