Note: This bug is displayed in read-only format because the product is no longer active in Red Hat Bugzilla.

Bug 2106163

Summary: Samples ImageStreams vs. registry.redhat.io: unsupported: V2 schema 1 manifest digests are no longer supported for image pulls
Product: OpenShift Container Platform Reporter: W. Trevor King <wking>
Component: SamplesAssignee: Feny Mehta <fmehta>
Status: CLOSED WORKSFORME QA Contact: Sushanta Das <susdas>
Severity: urgent Docs Contact:
Priority: urgent    
Version: 4.6CC: bleanhar, fmehta, shzhou, surya, tjungblu, yselkowi
Target Milestone: ---   
Target Release: 4.11.0   
Hardware: Unspecified   
OS: Unspecified   
Whiteboard:
Fixed In Version: Doc Type: No Doc Update
Doc Text:
Story Points: ---
Clone Of: Environment:
Last Closed: 2022-07-26 06:52:41 UTC Type: Bug
Regression: --- Mount Type: ---
Documentation: --- CRM:
Verified Versions: Category: ---
oVirt Team: --- RHEL 7.3 requirements from Atomic Host:
Cloudforms Team: --- Target Upstream Version:
Embargoed:

Description W. Trevor King 2022-07-11 21:49:52 UTC 
Since around 2022-07-11 14:50 UTC [1] CI runs like [2] have been failing with:

The following image streams are yet to be imported (attempt #20):
openshift/jboss-webserver31-tomcat7-openshift:1.0
openshift/jboss-webserver31-tomcat7-openshift:1.1
openshift/jboss-webserver31-tomcat8-openshift:1.0
openshift/jboss-webserver31-tomcat8-openshift:1.1
openshift/redhat-openjdk18-openshift:1.0
openshift/redhat-openjdk18-openshift:1.1
openshift/redhat-openjdk18-openshift:1.2
Failed while waiting on imagestream import

Looking in the must-gather for why:

$ curl -s https://gcsweb-ci.apps.ci.l2s4.p1.openshiftapps.com/gcs/origin-ci-test/logs/periodic-ci-openshift-release-master-nightly-4.11-e2e-azure-csi/1546489885890711552/artifacts/e2e-azure-csi/gather-must-gather/artifacts/must-gather.tar | tar xOz quay-io-openshift-release-dev-ocp-v4-0-art-dev-sha256-88d9b350be62e6314c2602112467e082bbf7d4c1538ae6a096c303d78ed5326f/namespaces/openshift/image.openshift.io/imagestreams/redhat-openjdk18-openshift.yaml | yaml2json | jq -r '.status.tags[] | (.conditions // [])[] | .lastTransitionTime + " " + .type + "=" + .status + " " + .reason + ": " + .message'
2022-07-11T14:11:12Z ImportSuccess=False InternalError: Internal error occurred: registry.redhat.io/redhat-openjdk-18/openjdk18-openshift:1.0: unsupported: V2 schema 1 manifest digests are no longer supported for image pulls. Use the equivalent V2 schema 2 manifest digest instead. For more information see https://access.redhat.com/articles/6138332
2022-07-11T14:11:12Z ImportSuccess=False InternalError: Internal error occurred: registry.redhat.io/redhat-openjdk-18/openjdk18-openshift:1.1: unsupported: V2 schema 1 manifest digests are no longer supported for image pulls. Use the equivalent V2 schema 2 manifest digest instead. For more information see https://access.redhat.com/articles/6138332
2022-07-11T14:11:12Z ImportSuccess=False InternalError: Internal error occurred: registry.redhat.io/redhat-openjdk-18/openjdk18-openshift:1.2: unsupported: V2 schema 1 manifest digests are no longer supported for image pulls. Use the equivalent V2 schema 2 manifest digest instead. For more information see https://access.redhat.com/articles/6138332

Perhaps registry.redhat.io no longer likes v2s1 manifests, and some of our samples depend on that manifest format?  Setting urgent/urgent until we understand whether the impact is specific to CI or also extends to customer clusters.

And checking the ClusterOperator conditions to see how this is reported there:

$ curl -s https://gcsweb-ci.apps.ci.l2s4.p1.openshiftapps.com/gcs/origin-ci-test/logs/periodic-ci-openshift-release-master-nightly-4.11-e2e-azure-csi/1546489885890711552/artifacts/e2e-azure-csi/gather-extra/artifacts/clusteroperators.json | jq -r '.items[] | select(.metadata.name == "openshift-samples").status.conditions[] | .lastTransitionTime + " " + .type + "=" + .status + " " + .reason + ": " + .message'
2022-07-11T14:09:21Z Degraded=False : 
2022-07-11T14:09:31Z Available=True : Samples installation successful at 4.11.0-0.nightly-2022-07-11-133844
2022-07-11T14:56:21Z Progressing=False FailedImageImports: Samples installed at 4.11.0-0.nightly-2022-07-11-133844, with image import failures for these imagestreams: openshift-service-ca.crt,kube-root-ca.crt,jboss-webserver31-tomcat7-openshift,jboss-webserver31-tomcat8-openshift,redhat-openjdk18-openshift; last import attempt 2022-07-11 14:11:03 +0000 UTC

So it's Progressing=False, which is something else we'll get back from Telemetry/Insights, but not something that the cluster-version operator will care about, so no impact on update completion or anything like that.

[1]: https://search.ci.openshift.org/chart?maxAge=10h&type=build-log&search=Failed+while+waiting+on+imagestream+import&name=periodic-
[2]: https://prow.ci.openshift.org/view/gs/origin-ci-test/logs/periodic-ci-openshift-release-master-nightly-4.11-e2e-azure-csi/1546489885890711552
Comment 1 David Peraza 2022-07-12 07:01:36 UTC 
This is the list of all images still at v2shema1 used in Samples from OCP 4.6 to 4.11:

registry.redhat.io/jboss-datagrid-6/datagrid65-client-openshift:1.0
registry.redhat.io/jboss-datagrid-6/datagrid65-client-openshift:1.1
registry.redhat.io/jboss-datagrid-7/datagrid71-openshift:1.0
registry.redhat.io/jboss-datagrid-7/datagrid71-openshift:1.1
registry.redhat.io/jboss-datagrid-7/datagrid71-openshift:1.2
registry.redhat.io/jboss-datagrid-6/datagrid65-openshift:1.2
registry.redhat.io/jboss-datagrid-6/datagrid65-openshift:1.3
registry.redhat.io/jboss-datagrid-6/datagrid65-openshift:1.4
registry.redhat.io/jboss-datagrid-6/datagrid65-openshift:1.5
registry.redhat.io/jboss-eap-6/eap64-openshift:1.1
registry.redhat.io/jboss-eap-6/eap64-openshift:1.2
registry.redhat.io/jboss-eap-6/eap64-openshift:1.3
registry.redhat.io/jboss-eap-6/eap64-openshift:1.4
registry.redhat.io/jboss-eap-6/eap64-openshift:1.5
registry.redhat.io/jboss-eap-6/eap64-openshift:1.6
registry.redhat.io/jboss-eap-6/eap64-openshift:1.7
registry.redhat.io/jboss-eap-7/eap71-openshift:1.1
registry.redhat.io/jboss-eap-7/eap70-openshift:1.3
registry.redhat.io/jboss-eap-7/eap70-openshift:1.4
registry.redhat.io/jboss-eap-7/eap70-openshift:1.5
registry.redhat.io/jboss-eap-7/eap70-openshift:1.6
registry.redhat.io/jboss-eap-7/eap70-openshift:1.7
registry.redhat.io/jboss-processserver-6/processserver64-openshift:1.0
registry.redhat.io/jboss-processserver-6/processserver64-openshift:1.1
registry.redhat.io/jboss-processserver-6/processserver64-openshift:1.2
registry.redhat.io/jboss-decisionserver-6/decisionserver64-openshift:1.0
registry.redhat.io/jboss-decisionserver-6/decisionserver64-openshift:1.1
registry.redhat.io/jboss-decisionserver-6/decisionserver64-openshift:1.2
registry.redhat.io/3scale-amp21/apicast-gateway:1.4-2
registry.redhat.io/redhat-openjdk-18/openjdk18-openshift:1.0
registry.redhat.io/redhat-openjdk-18/openjdk18-openshift:1.1
registry.redhat.io/redhat-openjdk-18/openjdk18-openshift:1.2
registry.redhat.io/jboss-amq-6/amq63-openshift:1.0
registry.redhat.io/jboss-amq-6/amq63-openshift:1.1
registry.redhat.io/jboss-amq-6/amq63-openshift:1.2
registry.redhat.io/jboss-amq-6/amq62-openshift:1.1
registry.redhat.io/jboss-amq-6/amq62-openshift:1.2
registry.redhat.io/jboss-amq-6/amq62-openshift:1.3
registry.redhat.io/jboss-amq-6/amq62-openshift:1.4
registry.redhat.io/jboss-amq-6/amq62-openshift:1.5
registry.redhat.io/jboss-amq-6/amq62-openshift:1.6
registry.redhat.io/jboss-webserver-3/webserver31-tomcat8-openshift:1.0
registry.redhat.io/jboss-webserver-3/webserver31-tomcat8-openshift:1.1
registry.redhat.io/jboss-webserver-3/webserver31-tomcat7-openshift:1.0
registry.redhat.io/jboss-webserver-3/webserver31-tomcat7-openshift:1.1
registry.redhat.io/redhat-sso-7/sso71-openshift:1.0
registry.redhat.io/redhat-sso-7/sso71-openshift:1.1
registry.redhat.io/redhat-sso-7/sso71-openshift:1.2
registry.redhat.io/redhat-sso-7/sso71-openshift:1.3
registry.redhat.io/redhat-sso-7/sso70-openshift:1.3
registry.redhat.io/redhat-sso-7/sso70-openshift:1.4
registry.redhat.io/redhat-sso-7/sso72-openshift:1.0
registry.redhat.io/openshift4/ose-jenkins-agent-nodejs:v4.0
registry.redhat.io/openshift4/ose-jenkins-agent-maven:v4.0
registry.redhat.io/openshift4/ose-jenkins:v4.0
Comment 2 David Peraza 2022-07-12 07:05:36 UTC 
@fmehta can you try to figure out the owners of these images so they can reset them to use v2schema2. See how to reset here: https://docs.docker.com/registry/spec/deprecated-schema-v1/

When you send correspondence point them to the root of the issue here: https://access.redhat.com/articles/6138332. Copy me, Jasper, Stevan and Linda Sharar
Comment 5 W. Trevor King 2022-07-12 17:08:10 UTC 
[1] should recover the bulk of OCP CI by ignoring the failed ImageStreams, but it doesn't actually fix the ImageStreams or do anything for external clusters.

[1]: https://github.com/openshift/release/pull/30367
Comment 6 Yaakov Selkowitz 2022-07-14 00:15:49 UTC 
The defunct openjdk18 and webserver31 tags have been removed in their upstreams.  For starters, should we sync just those to master now, and back to 4.11 (despite code freeze) so this isn't an issue at GA?
Comment 7 Feny Mehta 2022-07-20 06:53:44 UTC 
This was fixed by registry team to allow the pulls for schema1 (https://issues.redhat.com/browse/REGISTRY-326 )

Their intention to block v2s1 manifests and only shasum pulls were supposed to be affected, and it was a bug that tag pulls were affected.
Comment 11 W. Trevor King 2022-07-26 06:52:41 UTC 
As comment 7 points out, the:

  unsupported: V2 schema 1 manifest digests are no longer supported for image pulls

for by-tag registry.redhat.io access to older images was fixed on the registry.redhat.io side, so no OpenShift-side changes were needed.  The temporary breakage reminded us that there were some old samples that needed to be removed (e.g. bug 2110346 and similar series), but those will be tracked in their own series, and we don't need to keep this bug around or attach it to OpenShift errata, so I'm moving it to WORKSFORME.