Bug 1940203

Summary: oc image mirror degrades to schema1 which loses consistent SHA's and prevents OKD releases
Product: OpenShift Container Platform Reporter: Adam Kaplan <adam.kaplan>
Component: Test InfrastructureAssignee: Steve Kuznetsov <skuznets>
Status: CLOSED WONTFIX QA Contact:
Severity: high Docs Contact:
Priority: urgent    
Version: 4.6CC: aaleman, adam.kaplan, andbartl, aos-bugs, cglombek, dmesser, gmontero, hchatter, jack.ottofaro, jluhrsen, mfojtik, nalin, openshift-bugs-escalate, openshift-bugzilla-robot, pmuller, scuppett, sdodson, wking, xiuwang
Target Milestone: ---   
Target Release: 4.6.z   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: If docs needed, set a value
Doc Text:
Story Points: ---
Clone Of: 1939219 Environment:
Last Closed: 2022-08-24 15:27:23 UTC Type: ---
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:
Bug Depends On: 1939218, 1939219    
Bug Blocks:    

Description Adam Kaplan 2021-03-17 19:40:26 UTC
Description of problem:
`oc adm release mirror` has started uploading some manifests as schema 1. This was due to a bug in OCP builds pushing manifests that did not conform to the docker v2schema2 manifest spec. See https://bugzilla.redhat.com/show_bug.cgi?id=1905095

Once the dependent issue is fixed in OCP and backported to 4.6.z, our test platform needs to be updated to the patched z-stream and rebuild all images in the OCP/OKD payload.
Once all images are rebuilt, they then can be mirrored to quay.io.

Version-Release number of selected component (if applicable):
Latest from https://mirror.openshift.com/pub/openshift-v4/x86_64/clients/ocp/stable/openshift-client-linux.tar.gz
Client Version: 4.6.6

How reproducible:
Always

Steps to Reproduce:
1. Get pullspec of the image from payload:

$ RELEASE="4.6.0-0.okd-2020-12-07-083734"
$ oc adm release info --image-for aws-ebs-csi-driver-operator registry.svc.ci.openshift.org/origin/release:${RELEASE}
registry.svc.ci.openshift.org/origin/4.6-2020-12-07-083734@sha256:52f962cc969eaf5ab3c94e8c87eea75f12310dbde6ed3b0f7596c6d7200de08f

2. Check its schema version

$ skopeo inspect --authfile ~/src/github.com/vrutkovs/okd-installer/pull_secret.json docker://registry.svc.ci.openshift.org/origin/4.6-2020-12-07-083734@sha256:52f962cc969eaf5ab3c94e8c87eea75f12310dbde6ed3b0f7596c6d7200de08f --raw | jq '.schemaVersion'
2

(as expected)

3. Mirror the release:

$ oc adm -a /run/user/0/containers/auth.json release new --from-release registry.svc.ci.openshift.org/origin/release:${RELEASE} --mirror quay.io/openshift/okd-content --to-image quay.io/vrutkovs/okd-release:${RELEASE} --name=${RELEASE}

4. Check schema version of uploaded image:

$ oc adm release info --image-for aws-ebs-csi-driver-operator quay.io/vrutkovs/okd-release:${RELEASE}
quay.io/vrutkovs/okd-release@sha256:ce862b5f752bea5fdd0d2c3b197c3b0362e02e3e433ad168bfc8ca380082e429
$ skopeo inspect --authfile ~/src/github.com/vrutkovs/okd-installer/pull_secret.json docker://quay.io/vrutkovs/okd-release@sha256:ce862b5f752bea5fdd0d2c3b197c3b0362e02e3e433ad168bfc8ca380082e429 --raw | jq '.schemaVersion'
1


Actual results:
Some images are uploaded as schema 1 and throw signature verification errors during upload (as schema 1 digest doesn't match the expected)


Expected results:
All manifests are uploaded as schema 2

Comment 2 Steve Kuznetsov 2021-05-17 19:23:39 UTC
@akaplan what version of OCP must we be running before we trigger the mass-rebuild?

Comment 3 Adam Kaplan 2021-05-18 12:05:52 UTC
@Steve the following versions have the fix in:

1. OCP 4.7.4 or higher
2. OCP 4.6.23 or higher