Description of problem: In disconnected environments, CRIO uses the imagecontentsourcepolicy to "fallback" to alternate registry sources when pulling sha-based image specs. The imagestream import controller should do the same, again only when importing a SHA based imagestreamtag. If the external references is a SHA, and the import against the original registry fails, it should use the imagecontentsourcepolicy to try alternate registry sources to import the SHA. If it succeeds, then the "status" for the tag should reflect the alternate registry. This is particularly needed to help resolve imagestreamtags that point to SHA content that's part of the release payload.
I'm not sure if this needs to be a blocker for 4.2 or not. As it currently stands, some imagestreamtags (notably "cli", "tests", and "must-gather") will fail to import. It's not clear if any components on the cluster actually attempt to use those imagestreamtags for anything, since most components could just get the pullspec for those images directly via an image-references definition, such that they'd have the pullspec available in their manifest if they needed it. (And then any pods they created using that pullspec would go through CRIO which would handle the registry fallback logic if needed). As such i'm not setting a severity right now, but if we can fix this for 4.2 we should, and it's possible it will become more critical if we identify something that's not working because this is missing.
Note: imagecontentsourcepolicy is a cluster resource, so the imagestream import controller should have access to it (the controller manager operator should watch it/set it in the config so the controller gets restarted if it changes. Or the imagestream controller itself should watch that config resource directly, which is probably easier and avoids restarts).
The jenkins related images failed to import too, maybe should reset the bug priority.
As I recall, the jenkins related images failed import because the samples operator registry hostname substitution logic didn't include them (because the registry hostname substitution logic ignored "registry.svc.ci.openshift.org", and Gabe created a PR to address that (in a different BZ i believe). Is it still broken? Has that PR merged?
(In reply to Ben Parees from comment #0) > Description of problem: > In disconnected environments, CRIO uses the imagecontentsourcepolicy to > "fallback" to alternate registry sources when pulling sha-based image specs. > > The imagestream import controller should do the same, again only when > importing a SHA based imagestreamtag. If the external references is a SHA, > and the import against the original registry fails, it should use the > imagecontentsourcepolicy to try alternate registry sources to import the SHA. I think it would be ideal to use the c/image client to access registries; that handles the MCO-managed registries.conf mirror configuration automatically, exactly consistently with CRI-O. (I have no idea whether the port to c/image would be more or less work than to handle ImageContentSourcePolicy explicitly in all the ImageStream callers.)
In response's to Ben's #comment 5 and XiuJuan's #comment 4 the samples operator change to include `registry.svc.ci.openshift.org` in the list of registries the samples operator will replace with his samplesRegistry config setting did land with commit https://github.com/openshift/cluster-samples-operator/commit/b0470e90ab6e9914f1c0914a3f995a29984d6f66 If XiuJuan thinks that commit was in the version being used, I would need to see the output from oc get is jenkins -n openshift -o yaml and oc get configs.samples -o yaml and open a separate bugzilla. I'm in the midst of some customer/3.11 testing right now and can't get a 4.2 cluster up, but if I can before XiuJuan updates.
Gabe, The latest disconnect env(4.2.0-0.nightly-2019-08-20-213632) has included the commit https://github.com/openshift/cluster-samples-operator/pull/173. So I will report a new bug to track the jenkins issue. Thanks
We are going to backport this to 4.2.z right?
Right, but we haven't merged it yet.
4.3 disconnected install failed due to 1770101, QE will verify this bug when 1770101 is fixed.
In disconnect env, Could import image with SHA imports $ oc get is cli -n openshift -o yaml apiVersion: image.openshift.io/v1 kind: ImageStream metadata: creationTimestamp: "2019-11-18T02:02:34Z" generation: 5 name: cli namespace: openshift resourceVersion: "35371" selfLink: /apis/image.openshift.io/v1/namespaces/openshift/imagestreams/cli uid: 3c6f08f0-7918-468c-ae8d-8b06377ddaf6 spec: lookupPolicy: local: false tags: - annotations: null from: kind: DockerImage name: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f572099c3c93261cab23376857c06757d1aa680414f25fa08709efdef38a7f1e generation: 4 importPolicy: scheduled: true name: latest referencePolicy: type: Source status: dockerImageRepository: image-registry.openshift-image-registry.svc:5000/openshift/cli publicDockerImageRepository: default-route-openshift-image-registry.apps.jialiu-663.qe.gcp.devcluster.openshift.com/openshift/cli tags: - items: - created: "2019-11-18T02:54:31Z" dockerImageReference: quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:f572099c3c93261cab23376857c06757d1aa680414f25fa08709efdef38a7f1e generation: 4 image: sha256:f572099c3c93261cab23376857c06757d1aa680414f25fa08709efdef38a7f1e tag: latest Before create imagecontentsourcepolicy for target image, import image with digest will fail, after creating imagecontentsourcepolicy, the importing process succeed. 1. Mirror image to mirror registry $oc image mirror centos/ruby-22-centos7@sha256:a18c8706118a5c4c9f1adf045024d2abf06ba632b5674b23421019ee4d3edcae jialiu-663.mirror-registry.qe.gcp.devcluster.openshift.com:5000/centos/ruby-22-centos7 2. Import image with digest before create imagecontentsourcepolicy $oc import-image rubycentos:v1 --from=docker.io/centos/ruby-22-centos7@sha256:a18c8706118a5c4c9f1adf045024d2abf06ba632b5674b23421019ee4d3edcae -n openshift --confirm error: tag failed: Internal error occurred: docker.io/centos/ruby-22-centos7@sha256:a18c8706118a5c4c9f1adf045024d2abf06ba632b5674b23421019ee4d3edcae: Get https://registry-1.docker.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers) imagestream.image.openshift.io/rubycentos imported with errors Name: rubycentos Namespace: openshift Created: 1 second ago Labels: <none> Annotations: openshift.io/image.dockerRepositoryCheck=2019-11-18T03:13:45Z Image Repository: image-registry.openshift-image-registry.svc:5000/openshift/rubycentos Image Lookup: local=false Unique Images: 0 Tags: 1 v1 tagged from docker.io/centos/ruby-22-centos7@sha256:a18c8706118a5c4c9f1adf045024d2abf06ba632b5674b23421019ee4d3edcae ! error: Import failed (InternalError): Internal error occurred: docker.io/centos/ruby-22-centos7@sha256:a18c8706118a5c4c9f1adf045024d2abf06ba632b5674b23421019ee4d3edcae: Get https://registry-1.docker.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers) 1 second ago 3.Create imagecontentsourcepolicy $ oc get imagecontentsourcepolicy image-policy-centos -o yaml apiVersion: operator.openshift.io/v1alpha1 kind: ImageContentSourcePolicy metadata: creationTimestamp: "2019-11-18T03:37:35Z" generation: 2 name: image-policy-centos resourceVersion: "90520" selfLink: /apis/operator.openshift.io/v1alpha1/imagecontentsourcepolicies/image-policy-centos uid: ce4f362f-6d15-4e41-b874-b36f943c4767 spec: repositoryDigestMirrors: - mirrors: - jialiu-663.mirror-registry.qe.gcp.devcluster.openshift.com:5000/centos/ruby-22-centos7 source: docker.io/centos/ruby-22-centos7 4.Import image with digest again $ oc describe is rubycentos -n openshift Name: rubycentos Namespace: openshift Created: 3 hours ago Labels: <none> Annotations: openshift.io/image.dockerRepositoryCheck=2019-11-18T05:38:22Z Image Repository: default-route-openshift-image-registry.apps.jialiu-663.qe.gcp.devcluster.openshift.com/openshift/rubycentos Image Lookup: local=false Unique Images: 1 Tags: 1 v1 tagged from docker.io/centos/ruby-22-centos7@sha256:a18c8706118a5c4c9f1adf045024d2abf06ba632b5674b23421019ee4d3edcae * docker.io/centos/ruby-22-centos7@sha256:a18c8706118a5c4c9f1adf045024d2abf06ba632b5674b23421019ee4d3edcae 22 minutes ago Verified with 4.3.0-0.nightly-2019-11-15-213610, and PASS
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2020:0062