Since around 2022-07-11 14:50 UTC [1] CI runs like [2] have been failing with: The following image streams are yet to be imported (attempt #20): openshift/jboss-webserver31-tomcat7-openshift:1.0 openshift/jboss-webserver31-tomcat7-openshift:1.1 openshift/jboss-webserver31-tomcat8-openshift:1.0 openshift/jboss-webserver31-tomcat8-openshift:1.1 openshift/redhat-openjdk18-openshift:1.0 openshift/redhat-openjdk18-openshift:1.1 openshift/redhat-openjdk18-openshift:1.2 Failed while waiting on imagestream import Looking in the must-gather for why: $ curl -s https://gcsweb-ci.apps.ci.l2s4.p1.openshiftapps.com/gcs/origin-ci-test/logs/periodic-ci-openshift-release-master-nightly-4.11-e2e-azure-csi/1546489885890711552/artifacts/e2e-azure-csi/gather-must-gather/artifacts/must-gather.tar | tar xOz quay-io-openshift-release-dev-ocp-v4-0-art-dev-sha256-88d9b350be62e6314c2602112467e082bbf7d4c1538ae6a096c303d78ed5326f/namespaces/openshift/image.openshift.io/imagestreams/redhat-openjdk18-openshift.yaml | yaml2json | jq -r '.status.tags[] | (.conditions // [])[] | .lastTransitionTime + " " + .type + "=" + .status + " " + .reason + ": " + .message' 2022-07-11T14:11:12Z ImportSuccess=False InternalError: Internal error occurred: registry.redhat.io/redhat-openjdk-18/openjdk18-openshift:1.0: unsupported: V2 schema 1 manifest digests are no longer supported for image pulls. Use the equivalent V2 schema 2 manifest digest instead. For more information see https://access.redhat.com/articles/6138332 2022-07-11T14:11:12Z ImportSuccess=False InternalError: Internal error occurred: registry.redhat.io/redhat-openjdk-18/openjdk18-openshift:1.1: unsupported: V2 schema 1 manifest digests are no longer supported for image pulls. Use the equivalent V2 schema 2 manifest digest instead. For more information see https://access.redhat.com/articles/6138332 2022-07-11T14:11:12Z ImportSuccess=False InternalError: Internal error occurred: registry.redhat.io/redhat-openjdk-18/openjdk18-openshift:1.2: unsupported: V2 schema 1 manifest digests are no longer supported for image pulls. Use the equivalent V2 schema 2 manifest digest instead. For more information see https://access.redhat.com/articles/6138332 Perhaps registry.redhat.io no longer likes v2s1 manifests, and some of our samples depend on that manifest format? Setting urgent/urgent until we understand whether the impact is specific to CI or also extends to customer clusters. And checking the ClusterOperator conditions to see how this is reported there: $ curl -s https://gcsweb-ci.apps.ci.l2s4.p1.openshiftapps.com/gcs/origin-ci-test/logs/periodic-ci-openshift-release-master-nightly-4.11-e2e-azure-csi/1546489885890711552/artifacts/e2e-azure-csi/gather-extra/artifacts/clusteroperators.json | jq -r '.items[] | select(.metadata.name == "openshift-samples").status.conditions[] | .lastTransitionTime + " " + .type + "=" + .status + " " + .reason + ": " + .message' 2022-07-11T14:09:21Z Degraded=False : 2022-07-11T14:09:31Z Available=True : Samples installation successful at 4.11.0-0.nightly-2022-07-11-133844 2022-07-11T14:56:21Z Progressing=False FailedImageImports: Samples installed at 4.11.0-0.nightly-2022-07-11-133844, with image import failures for these imagestreams: openshift-service-ca.crt,kube-root-ca.crt,jboss-webserver31-tomcat7-openshift,jboss-webserver31-tomcat8-openshift,redhat-openjdk18-openshift; last import attempt 2022-07-11 14:11:03 +0000 UTC So it's Progressing=False, which is something else we'll get back from Telemetry/Insights, but not something that the cluster-version operator will care about, so no impact on update completion or anything like that. [1]: https://search.ci.openshift.org/chart?maxAge=10h&type=build-log&search=Failed+while+waiting+on+imagestream+import&name=periodic- [2]: https://prow.ci.openshift.org/view/gs/origin-ci-test/logs/periodic-ci-openshift-release-master-nightly-4.11-e2e-azure-csi/1546489885890711552
This is the list of all images still at v2shema1 used in Samples from OCP 4.6 to 4.11: registry.redhat.io/jboss-datagrid-6/datagrid65-client-openshift:1.0 registry.redhat.io/jboss-datagrid-6/datagrid65-client-openshift:1.1 registry.redhat.io/jboss-datagrid-7/datagrid71-openshift:1.0 registry.redhat.io/jboss-datagrid-7/datagrid71-openshift:1.1 registry.redhat.io/jboss-datagrid-7/datagrid71-openshift:1.2 registry.redhat.io/jboss-datagrid-6/datagrid65-openshift:1.2 registry.redhat.io/jboss-datagrid-6/datagrid65-openshift:1.3 registry.redhat.io/jboss-datagrid-6/datagrid65-openshift:1.4 registry.redhat.io/jboss-datagrid-6/datagrid65-openshift:1.5 registry.redhat.io/jboss-eap-6/eap64-openshift:1.1 registry.redhat.io/jboss-eap-6/eap64-openshift:1.2 registry.redhat.io/jboss-eap-6/eap64-openshift:1.3 registry.redhat.io/jboss-eap-6/eap64-openshift:1.4 registry.redhat.io/jboss-eap-6/eap64-openshift:1.5 registry.redhat.io/jboss-eap-6/eap64-openshift:1.6 registry.redhat.io/jboss-eap-6/eap64-openshift:1.7 registry.redhat.io/jboss-eap-7/eap71-openshift:1.1 registry.redhat.io/jboss-eap-7/eap70-openshift:1.3 registry.redhat.io/jboss-eap-7/eap70-openshift:1.4 registry.redhat.io/jboss-eap-7/eap70-openshift:1.5 registry.redhat.io/jboss-eap-7/eap70-openshift:1.6 registry.redhat.io/jboss-eap-7/eap70-openshift:1.7 registry.redhat.io/jboss-processserver-6/processserver64-openshift:1.0 registry.redhat.io/jboss-processserver-6/processserver64-openshift:1.1 registry.redhat.io/jboss-processserver-6/processserver64-openshift:1.2 registry.redhat.io/jboss-decisionserver-6/decisionserver64-openshift:1.0 registry.redhat.io/jboss-decisionserver-6/decisionserver64-openshift:1.1 registry.redhat.io/jboss-decisionserver-6/decisionserver64-openshift:1.2 registry.redhat.io/3scale-amp21/apicast-gateway:1.4-2 registry.redhat.io/redhat-openjdk-18/openjdk18-openshift:1.0 registry.redhat.io/redhat-openjdk-18/openjdk18-openshift:1.1 registry.redhat.io/redhat-openjdk-18/openjdk18-openshift:1.2 registry.redhat.io/jboss-amq-6/amq63-openshift:1.0 registry.redhat.io/jboss-amq-6/amq63-openshift:1.1 registry.redhat.io/jboss-amq-6/amq63-openshift:1.2 registry.redhat.io/jboss-amq-6/amq62-openshift:1.1 registry.redhat.io/jboss-amq-6/amq62-openshift:1.2 registry.redhat.io/jboss-amq-6/amq62-openshift:1.3 registry.redhat.io/jboss-amq-6/amq62-openshift:1.4 registry.redhat.io/jboss-amq-6/amq62-openshift:1.5 registry.redhat.io/jboss-amq-6/amq62-openshift:1.6 registry.redhat.io/jboss-webserver-3/webserver31-tomcat8-openshift:1.0 registry.redhat.io/jboss-webserver-3/webserver31-tomcat8-openshift:1.1 registry.redhat.io/jboss-webserver-3/webserver31-tomcat7-openshift:1.0 registry.redhat.io/jboss-webserver-3/webserver31-tomcat7-openshift:1.1 registry.redhat.io/redhat-sso-7/sso71-openshift:1.0 registry.redhat.io/redhat-sso-7/sso71-openshift:1.1 registry.redhat.io/redhat-sso-7/sso71-openshift:1.2 registry.redhat.io/redhat-sso-7/sso71-openshift:1.3 registry.redhat.io/redhat-sso-7/sso70-openshift:1.3 registry.redhat.io/redhat-sso-7/sso70-openshift:1.4 registry.redhat.io/redhat-sso-7/sso72-openshift:1.0 registry.redhat.io/openshift4/ose-jenkins-agent-nodejs:v4.0 registry.redhat.io/openshift4/ose-jenkins-agent-maven:v4.0 registry.redhat.io/openshift4/ose-jenkins:v4.0
@fmehta can you try to figure out the owners of these images so they can reset them to use v2schema2. See how to reset here: https://docs.docker.com/registry/spec/deprecated-schema-v1/ When you send correspondence point them to the root of the issue here: https://access.redhat.com/articles/6138332. Copy me, Jasper, Stevan and Linda Sharar
[1] should recover the bulk of OCP CI by ignoring the failed ImageStreams, but it doesn't actually fix the ImageStreams or do anything for external clusters. [1]: https://github.com/openshift/release/pull/30367
The defunct openjdk18 and webserver31 tags have been removed in their upstreams. For starters, should we sync just those to master now, and back to 4.11 (despite code freeze) so this isn't an issue at GA?
This was fixed by registry team to allow the pulls for schema1 (https://issues.redhat.com/browse/REGISTRY-326 ) Their intention to block v2s1 manifests and only shasum pulls were supposed to be affected, and it was a bug that tag pulls were affected.
As comment 7 points out, the: unsupported: V2 schema 1 manifest digests are no longer supported for image pulls for by-tag registry.redhat.io access to older images was fixed on the registry.redhat.io side, so no OpenShift-side changes were needed. The temporary breakage reminded us that there were some old samples that needed to be removed (e.g. bug 2110346 and similar series), but those will be tracked in their own series, and we don't need to keep this bug around or attach it to OpenShift errata, so I'm moving it to WORKSFORME.