Created attachment 1606437 [details] jenkins imagestream info Description of problem: Although the images in payload have been mirrored to the mirror registry, the jenkins related images still can't import succeed with error: " ! error: Import failed (InternalError): Internal error occurred: Get https://quay.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Version-Release number of selected component (if applicable): 4.2.0-0.nightly-2019-08-20-213632 How reproducible: always Steps to Reproduce: 1.Mirror payload to the mirror registry 2.Launch disconnect env, then check all node if set the mirror registries config 3.Check jenkins imagestream under openshift project Actual results: All node has set the mirror registries config $ cat /etc/containers/registries.conf unqualified-search-registries = ["registry.access.redhat.com", "docker.io"] [[registry]] location = "quay.io/openshift-release-dev/ocp-v4.0-art-dev" insecure = false blocked = false mirror-by-digest-only = true prefix = "" [[registry.mirror]] location = "internal-registry.qe.devcluster.openshift.com:5000/ocp/release" insecure = false [[registry]] location = "registry.svc.ci.openshift.org/ocp/release" insecure = false blocked = false mirror-by-digest-only = true prefix = "" [[registry.mirror]] location = "internal-registry.qe.devcluster.openshift.com:5000/ocp/release" insecure = false $ oc describe is jenkins -n openshift Name: jenkins Namespace: openshift Created: 26 hours ago Labels: samples.operator.openshift.io/managed=true Annotations: openshift.io/display-name=Jenkins openshift.io/image.dockerRepositoryCheck=2019-08-21T08:14:48Z samples.operator.openshift.io/version=4.2.0-0.nightly-2019-08-20-213632 Image Repository: default-route-openshift-image-registry.apps.qe-sharedenv-8201.qe.devcluster.openshift.com/openshift/jenkins Image Lookup: local=false Unique Images: 0 Tags: 2 2 (latest) tagged from quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c056fad59c13eab71e7e3ea128da64c501ac7356a4f7e7fdcab015e7e7be5816 prefer registry pullthrough when referencing this tag Provides a Jenkins 2.X server on RHEL 7. For more information about using this container image, including OpenShift considerations, see https://github.com/openshift/jenkins/blob/master/README.md. Tags: jenkins ~ importing latest image ... ! error: Import failed (InternalError): Internal error occurred: Get https://quay.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers) 26 hours ago Expected results: Should import the quay.io images which are included in payload succeed. Additional info:
Moving to Samples (more likely related to the samples operator configuration), assigning to Gabe. Shouldn't our nightly builds reference registry.redhat.io for all imagestreams?
@Adam - the jenkins images are in the payload, so those imagestreams do *NOT* reference registry.redhat.io That said, my prior fix to include the api.ci registry that Ben and I previously crafted is insufficient, since even after substituting the the mirror registry ... see quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c056fad59c13eab71e7e3ea128da64c501ac7356a4f7e7fdcab015e7e7be5816 the repository and image name do not match with what is up at quay.io for the jenkins image, which is quay.io/openshift/origin-jenkins For 4.2, this will have to be a documented workaround. I'll update https://docs.google.com/document/d/1BL3z5T-BL2EZWKpK-9Z461lUsAwmAaY8iOTpBgypHEk/edit that is the input for https://bugzilla.redhat.com/show_bug.cgi?id=1738476 We do have a devex 4.3 consideration, simplify mirroring of samples, at https://docs.google.com/document/d/15gr5VzAB2C_VTBzghgv26GbnZqVi5YeknTu0zID1li4/edit#heading=h.9khfep37qdjo that would allow us to manipulate the image pull spec for specific imagestreams. That would allow a mapping the default jenkins image stream image reference from the payload location to things like quay.io/openshift/origin-jenkins. For now, per XiuJuan's original report, marking jenkins skipped and manually changing the image ref with oc tag are required.
OK wait, I missed [[registry]] location = "quay.io/openshift-release-dev/ocp-v4.0-art-dev" insecure = false blocked = false mirror-by-digest-only = true prefix = "" in the description Retract #Comment 2, except for my explanation about the jenkins imagestream coming from the payload.
Went back and looked at https://bugzilla.redhat.com/show_bug.cgi?id=1741391 and realized I need to clarify something @XiuJuan - I don't see you mentioning setting 'samplesRegsitry' to anything in your description above. Is that correct? If so, then https://bugzilla.redhat.com/show_bug.cgi?id=1741391#c6 and https://bugzilla.redhat.com/show_bug.cgi?id=1741391#c8 are not relevant, and we simply need Oleg's fix so that the imagestream import image pull for quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c056fad59c13eab71e7e3ea128da64c501ac7356a4f7e7fdcab015e7e7be5816 works. Please provide more that just `oc describe ...` I want 1) oc get is jenkins -n openshift -o yaml 2) oc get configs.samples.operator.openshift.io cluster -o yaml
@Gabe, I have added attachment that your need info in comment #0. When I reported bug https://bugzilla.redhat.com/show_bug.cgi?id=1738476, I do set the jenkins in skipped list, and push it to mirror registry then 'oc tag' it. Last time I didn't realize the jenkins with quay.io importing error is a issue untill I saw the bug https://bugzilla.redhat.com/show_bug.cgi?id=1741391.
Gabe, it looks like the samples operator is deliberately not touching the jenkins pullspecs during substitution: https://github.com/openshift/cluster-samples-operator/blob/111fabeeeaaca0b377b54d1738ad8d0df0dffc6c/pkg/stub/imagestreams.go#L242
OK both the data XiuJuan provide and Ben's find are helpful. For completeness: - One thing that had me confused was seeing "quay.io" for the jenkins registry vs. "registry.redhat.io" or "registry.svc.ci.openshift.org" in the *SPEC* ... I thought that "quay.io" had been set as the samplesRegistry override. I know realize that the installer must be updating the image ref via https://github.com/openshift/cluster-samples-operator/blob/master/manifests/image-references#L25-L28 to quay.io and that since this is a mirrored install, etc. that is why I am not seeing it ultimately substituted for "registry.svc.ci.openshift.org" - As Ben discovered, the link he found means that our prior attempt at https://github.com/openshift/cluster-samples-operator/blob/master/pkg/stub/imagestreams.go#L186 to allow for jenkins overrides was incomplete ... I had forgotten / missed the additional check. I'll use this bug to address that. - Also not though that for the imagestreams we did substitute, those imports are currently failing. For example: "internal-registry.qe.devcluster.openshift.com:5000/redhat-sso-7/sso71-openshift:1.0" not found<imagestream/redhat-sso71-openshift><imagestream/jboss-eap72-openshift>dockerimage.image.openshift.io that is where I believe Oleg's bug https://bugzilla.redhat.com/show_bug.cgi?id=1741391 comes in. Please correct me if I'm wrong there Ben.
> - Also not though that for the imagestreams we did substitute, those imports are currently failing. For example: > "internal-registry.qe.devcluster.openshift.com:5000/redhat-sso-7/sso71-openshift:1.0" > not found<imagestream/redhat-sso71-openshift><imagestream/jboss-eap72-openshift>dockerimage.image.openshift.io > that is where I believe Oleg's bug https://bugzilla.redhat.com/show_bug.cgi?id=1741391 comes in. > Please correct me if I'm wrong there Ben. Since those imports are not "import by SHA", 1741391 will have no impact on it. Plus they look correct (the substitution happened, the import is being attempted against the disconnected/mirror registry, as we would expect/want it to). I assume the "not found" is occurring because QE didn't mirror those particular images into "internal-registry.qe.devcluster.openshift.com", but you'd have to check with them.
yeah I was worried about that (sha vs. tag) @XiuJuan - can you confirm you did *NOT* mirror those images?
@Gabe @Ben Yeah, I didn't mirror jboss related images due to the limit quota of the mirror registry cluster. Only rhscl and some images using frequestly are mirrored.
Can't install disconnect env successfully today, will try to do tomorrow. @Gabe, This is just a workaround before https://bugzilla.redhat.com/show_bug.cgi?id=1741391 fix?
They are really mutually exclusive as much as alternatives, XiuJuan. And I must admit that it has taken hearing explanations from Ben on a couple of occasions to get a fully grasp. I'll try to clarify: 1) The first thing to understand is that with imagestreams, the image reference for the IST can either be a tag reference or a sha reference. 2) When it is a tag reference, setting up of the cri-o mirrors like you noted #comment 0 is sufficient. That is why the image registry as is today would have been able to pull the "internal-registry.qe.devcluster.openshift.com:5000/redhat-sso-7/sso71-openshift:1.0" as Ben noted in #comment 8 if you had mirrored the content 3) On that mirror set up though, do be aware that the samples can come from different places. In addition to quay.io for some things, many imagestreams in fact pull content from registry.redhat.io. The rhscl images are among those (registry.redhat.io/rhscl/ruby-23-rhel7:latest). So I'm assuming you defined a mirror for registry.redhat.io as well. 4) Now, for sha references, we need Oleg's https://bugzilla.redhat.com/show_bug.cgi?id=1741391 in order to pull the image. Images that are from the install payload fall in the SHA category. So that includes the jenkins related imagestreams, as well as those imagestreams not managed by samples operator that are in the openshift namespacae, like cli, must-gather, etc. 5) Finally, yes, even independent of disconnected, you can configured samples operator to change the pull spec. Customer's might have "non-disconnected" reasons for doing this. In this case, we were not allowing override of the jenkins imagestreams, but there was no need to prevent that. Hence this fix. But you should not need to employ both the samples operator registry config override and cri-o mirroring once Oleg's bug arrives.
> When it is a tag reference, setting up of the cri-o mirrors like you noted #comment 0 is sufficient. setting up the crio mirrors isn't sufficient. the imagestream import will still fail because the imagestream you're trying to import going to still point to a registry you can't reach (registry.redhat.io). you need to: 1) mirror the content into your disconnected registry 2) use the samples operator config to point the samples to the disconnected registry that will get import to succeed if you actually want to *pull* the imported image, you also need to setup the crio mirror configuration. This is the case for all the imagestreams the samples operator manages except the Jenkins imagestream. (Note: I am not considering the "test/cli/must-gather" as imagestreams the samples operator manages.....those are managed by the CVO)
OK XiuJuan Ben and I talked at lunch and figured out how we diverged in our understandings. So yeah, take Ben's #comment 14, and back to your original question 1) the change here is part of what is needed to fix pulling the jenkins imagestream in disconnected ... it at least gets the registry in the pull spec correct for the subsequent pull attempt 2) in addition to 1), you'll still need https://bugzilla.redhat.com/show_bug.cgi?id=1741391 to get the actual image pull for the jenkins image stream to work once the pull spec has been updated to use the disconnected registry, as jenkins image will still be a sha ref, even with the updated registry ref
Evidently I still don't have it right :-) Ben just corrected me on slack. The gist: This fix allows you to import the jenkins image such that you do NOT need https://bugzilla.redhat.com/show_bug.cgi?id=1741391 as long as you use the samples operator config to cause the imagestreams to be modified to point to the disconnected registry instead of the provided registry for the imagestream Having the registry used for the jenkins import pointing directly to the disconnected registry will allow an image import to work, even if it is a sha. If you have the fix to https://bugzilla.redhat.com/show_bug.cgi?id=1741391, you would not need to update the image registry for the jenkins image import to work, since it is by sha, but since you need to update the registry for the other sample imagestreams (which are not in the payload and do not have a sha reference), you have to make the configuration change anyway.
@Gabe @Ben Thanks your so so detailed explaining, I could understand well about the feature now. I could prove the jenkins related images could be overrided after setting samplesregistry in normal cluster with 4.2.0-0.nightly-2019-08-26-235330 payload. But I still want confirm if anyother risks exist in disconnect env(Can't install a fresh version disconnect env yet).
The only possible "risk" wrt the imagestreams that the samples operator manages @XiuJuan is that if you don't import all the samples into the mirror, after 2 hours, for any failed imports, the samples operator will set Degraded to true. Of course, if you add the imagestreams you do not add to the mirror to the skipped list, that can be avoided. Otherwise, if you have mirrors set up, and you override the registry setting in the samples operator config, the imagestream imports should work.
After mirror the jenkins images to mirror registry and override the samplesregistry, the jenkins image could import succeed Also other mirrored images the samples operator managed import succeed. $oc image mirror quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c0ae54496dd2eff47b34b3e4f4bc74aebef8c542b1f84cacb2b1e5a24def57fe mirror-registry.qe.devcluster.openshift.com:5000/openshift-release-dev/ocp-v4.0-art-dev oc get config.samples -o yaml spec: architectures: - x86_64 managementState: Managed samplesRegistry: mirror-registry.qe.devcluster.openshift.com:5000 status: $ oc describe is jenkins -n openshift Name: jenkins Namespace: openshift Created: 6 minutes ago Labels: samples.operator.openshift.io/managed=true Annotations: openshift.io/display-name=Jenkins openshift.io/image.dockerRepositoryCheck=2019-08-28T07:41:01Z samples.operator.openshift.io/version=4.2.0-0.nightly-2019-08-27-105356 Image Repository: default-route-openshift-image-registry.apps.jialiu-share0828.qe.devcluster.openshift.com/openshift/jenkins Image Lookup: local=false Unique Images: 1 Tags: 2 2 (latest) tagged from mirror-registry.qe.devcluster.openshift.com:5000/openshift-release-dev/ocp-v4.0-art-dev@sha256:c0ae54496dd2eff47b34b3e4f4bc74aebef8c542b1f84cacb2b1e5a24def57fe prefer registry pullthrough when referencing this tag Provides a Jenkins 2.X server on RHEL 7. For more information about using this container image, including OpenShift considerations, see https://github.com/openshift/jenkins/blob/master/README.md. Tags: jenkins * mirror-registry.qe.devcluster.openshift.com:5000/openshift-release-dev/ocp-v4.0-art-dev@sha256:c0ae54496dd2eff47b34b3e4f4bc74aebef8c542b1f84cacb2b1e5a24def57fe 6 minutes ago
Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2019:2922