Bug 1744071 - [Disconnect]Can't import jenkins and other quay.io images which are included in payload.
Summary: [Disconnect]Can't import jenkins and other quay.io images which are included ...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Samples
Version: 4.2.0
Hardware: Unspecified
OS: Unspecified
high
high
Target Milestone: ---
: 4.2.0
Assignee: Gabe Montero
QA Contact: XiuJuan Wang
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2019-08-21 09:25 UTC by XiuJuan Wang
Modified: 2019-10-16 06:37 UTC (History)
3 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2019-10-16 06:36:58 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)
jenkins imagestream info (18.77 KB, text/plain)
2019-08-21 09:25 UTC, XiuJuan Wang
no flags Details


Links
System ID Private Priority Status Summary Last Updated
Github openshift cluster-samples-operator pull 177 0 'None' closed Bug 1744071: allow override of jenkins* imagestream registries for disconnected/mi… 2020-04-08 20:20:50 UTC
Red Hat Product Errata RHBA-2019:2922 0 None None None 2019-10-16 06:37:07 UTC

Description XiuJuan Wang 2019-08-21 09:25:01 UTC
Created attachment 1606437 [details]
jenkins imagestream info

Description of problem:
Although the images in payload have been mirrored to the mirror registry, the jenkins related images still can't import succeed with error:
" ! error: Import failed (InternalError): Internal error occurred: Get https://quay.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"

Version-Release number of selected component (if applicable):
4.2.0-0.nightly-2019-08-20-213632

How reproducible:
always

Steps to Reproduce:
1.Mirror payload to the mirror registry
2.Launch disconnect env, then check all node if set the mirror registries config
3.Check jenkins imagestream under openshift project

Actual results:
All node has set the mirror registries config
$ cat /etc/containers/registries.conf
unqualified-search-registries = ["registry.access.redhat.com", "docker.io"]

[[registry]]
  location = "quay.io/openshift-release-dev/ocp-v4.0-art-dev"
  insecure = false
  blocked = false
  mirror-by-digest-only = true
  prefix = ""

  [[registry.mirror]]
    location = "internal-registry.qe.devcluster.openshift.com:5000/ocp/release"
    insecure = false

[[registry]]
  location = "registry.svc.ci.openshift.org/ocp/release"
  insecure = false
  blocked = false
  mirror-by-digest-only = true
  prefix = ""

  [[registry.mirror]]
    location = "internal-registry.qe.devcluster.openshift.com:5000/ocp/release"
    insecure = false

$ oc describe is jenkins  -n openshift
Name:			jenkins
Namespace:		openshift
Created:		26 hours ago
Labels:			samples.operator.openshift.io/managed=true
Annotations:		openshift.io/display-name=Jenkins
			openshift.io/image.dockerRepositoryCheck=2019-08-21T08:14:48Z
			samples.operator.openshift.io/version=4.2.0-0.nightly-2019-08-20-213632
Image Repository:	default-route-openshift-image-registry.apps.qe-sharedenv-8201.qe.devcluster.openshift.com/openshift/jenkins
Image Lookup:		local=false
Unique Images:		0
Tags:			2

2 (latest)
  tagged from quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c056fad59c13eab71e7e3ea128da64c501ac7356a4f7e7fdcab015e7e7be5816
    prefer registry pullthrough when referencing this tag

  Provides a Jenkins 2.X server on RHEL 7. For more information about using this container image, including OpenShift considerations, see https://github.com/openshift/jenkins/blob/master/README.md.
  Tags: jenkins

  ~ importing latest image ...
  ! error: Import failed (InternalError): Internal error occurred: Get https://quay.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
      26 hours ago

Expected results:

Should import the quay.io images which are included in payload succeed.
Additional info:

Comment 1 Adam Kaplan 2019-08-22 14:21:35 UTC
Moving to Samples (more likely related to the samples operator configuration), assigning to Gabe.

Shouldn't our nightly builds reference registry.redhat.io for all imagestreams?

Comment 2 Gabe Montero 2019-08-22 14:58:20 UTC
@Adam - the jenkins images are in the payload, so those imagestreams do *NOT* reference registry.redhat.io

That said, my prior fix to include the api.ci registry that Ben and I previously crafted is insufficient, since even after
substituting the the mirror registry ... see 

quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c056fad59c13eab71e7e3ea128da64c501ac7356a4f7e7fdcab015e7e7be5816

the repository and image name do not match with what is up at quay.io for the jenkins image, which is quay.io/openshift/origin-jenkins

For 4.2, this will have to be a documented workaround.  I'll update https://docs.google.com/document/d/1BL3z5T-BL2EZWKpK-9Z461lUsAwmAaY8iOTpBgypHEk/edit that is the input for https://bugzilla.redhat.com/show_bug.cgi?id=1738476

We do have a devex 4.3 consideration, simplify mirroring of samples, at https://docs.google.com/document/d/15gr5VzAB2C_VTBzghgv26GbnZqVi5YeknTu0zID1li4/edit#heading=h.9khfep37qdjo that 
would allow us to manipulate the image pull spec for specific imagestreams.  That would allow a mapping the default jenkins image stream image reference from the payload location to 
things like quay.io/openshift/origin-jenkins.

For now, per XiuJuan's original report, marking jenkins skipped and manually changing the image ref with oc tag are required.

Comment 3 Gabe Montero 2019-08-22 15:09:47 UTC
OK wait, I missed

[[registry]]
  location = "quay.io/openshift-release-dev/ocp-v4.0-art-dev"
  insecure = false
  blocked = false
  mirror-by-digest-only = true
  prefix = ""


in the description

Retract #Comment 2, except for my explanation about the jenkins imagestream coming from the payload.

Comment 4 Gabe Montero 2019-08-22 15:16:49 UTC
Went back and looked at https://bugzilla.redhat.com/show_bug.cgi?id=1741391 and realized I need to clarify something

@XiuJuan - I don't see you mentioning setting 'samplesRegsitry' to anything in your description above.  Is that correct?

If so, then https://bugzilla.redhat.com/show_bug.cgi?id=1741391#c6 and https://bugzilla.redhat.com/show_bug.cgi?id=1741391#c8
are not relevant, and we simply need Oleg's fix so that the imagestream import image pull for 

quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c056fad59c13eab71e7e3ea128da64c501ac7356a4f7e7fdcab015e7e7be5816

works.

Please provide more that just `oc describe ...` 


I want
1) oc get is jenkins -n openshift -o yaml
2) oc get configs.samples.operator.openshift.io cluster -o yaml

Comment 5 XiuJuan Wang 2019-08-23 02:09:30 UTC
@Gabe, I have added attachment that your need info in comment #0.

When I reported bug https://bugzilla.redhat.com/show_bug.cgi?id=1738476, I do set the jenkins in skipped list, and push it to mirror registry then 'oc tag' it. 
Last time I didn't realize the jenkins with quay.io importing error is a issue untill I saw the bug https://bugzilla.redhat.com/show_bug.cgi?id=1741391.

Comment 6 Ben Parees 2019-08-23 04:08:11 UTC
Gabe, it looks like the samples operator is deliberately not touching the jenkins pullspecs during substitution:

https://github.com/openshift/cluster-samples-operator/blob/111fabeeeaaca0b377b54d1738ad8d0df0dffc6c/pkg/stub/imagestreams.go#L242

Comment 7 Gabe Montero 2019-08-23 13:27:48 UTC
OK both the data XiuJuan provide and Ben's find are helpful.

For completeness:
- One thing that had me confused was seeing "quay.io" for the jenkins registry vs. "registry.redhat.io" or "registry.svc.ci.openshift.org" in the *SPEC* ... I thought that "quay.io" had been set as the samplesRegistry override.  I know realize that the installer must be updating the image ref via https://github.com/openshift/cluster-samples-operator/blob/master/manifests/image-references#L25-L28 to quay.io and that since this is a mirrored install, etc. that is why I am not seeing it ultimately substituted for "registry.svc.ci.openshift.org"

- As Ben discovered, the link he found means that our prior attempt at https://github.com/openshift/cluster-samples-operator/blob/master/pkg/stub/imagestreams.go#L186 to allow for jenkins overrides was incomplete ... I had forgotten / missed the additional check.  I'll use this bug to address that.

- Also not though that for the imagestreams we did substitute, those imports are currently failing. For example:

        "internal-registry.qe.devcluster.openshift.com:5000/redhat-sso-7/sso71-openshift:1.0"
        not found<imagestream/redhat-sso71-openshift><imagestream/jboss-eap72-openshift>dockerimage.image.openshift.io

that is where I believe Oleg's bug https://bugzilla.redhat.com/show_bug.cgi?id=1741391 comes in.

Please correct me if I'm wrong there Ben.

Comment 8 Ben Parees 2019-08-23 14:11:20 UTC
> - Also not though that for the imagestreams we did substitute, those imports are currently failing. For example:
>        "internal-registry.qe.devcluster.openshift.com:5000/redhat-sso-7/sso71-openshift:1.0"
>        not found<imagestream/redhat-sso71-openshift><imagestream/jboss-eap72-openshift>dockerimage.image.openshift.io
> that is where I believe Oleg's bug https://bugzilla.redhat.com/show_bug.cgi?id=1741391 comes in.
> Please correct me if I'm wrong there Ben.

Since those imports are not "import by SHA", 1741391 will have no impact on it.  Plus they look correct (the substitution happened, the import is being attempted against the disconnected/mirror registry, as we would expect/want it to).

I assume the "not found" is occurring because QE didn't mirror those particular images into "internal-registry.qe.devcluster.openshift.com", but you'd have to check with them.

Comment 9 Gabe Montero 2019-08-23 14:29:19 UTC
yeah I was worried about that (sha vs. tag)


@XiuJuan - can you confirm you did *NOT* mirror those images?

Comment 11 XiuJuan Wang 2019-08-26 01:30:39 UTC
@Gabe @Ben
Yeah, I didn't mirror jboss related images due to the limit quota of the mirror registry cluster. Only rhscl and some images using frequestly are mirrored.

Comment 12 XiuJuan Wang 2019-08-26 09:15:10 UTC
Can't install disconnect env successfully today, will try to do tomorrow.

@Gabe, This is just a workaround before https://bugzilla.redhat.com/show_bug.cgi?id=1741391 fix?

Comment 13 Gabe Montero 2019-08-26 14:19:27 UTC
They are really mutually exclusive as much as alternatives, XiuJuan.  And I must admit that it has taken hearing explanations from Ben on a couple of occasions to get a fully grasp.  I'll try to clarify:

1) The first thing to understand is that with imagestreams, the image reference for the IST can either be a tag reference or a sha reference.

2) When it is a tag reference, setting up of the cri-o mirrors like you noted #comment 0 is sufficient.  That is why the image registry as is today would have been able to pull the "internal-registry.qe.devcluster.openshift.com:5000/redhat-sso-7/sso71-openshift:1.0" as Ben noted in #comment 8 if you had mirrored the content

3) On that mirror set up though, do be aware that the samples can come from different places.  In addition to quay.io for some things, many imagestreams in fact pull content from registry.redhat.io.  The rhscl images are among those (registry.redhat.io/rhscl/ruby-23-rhel7:latest).  So I'm assuming you defined a mirror for registry.redhat.io as well.

4) Now, for sha references, we need Oleg's https://bugzilla.redhat.com/show_bug.cgi?id=1741391 in order to pull the image.  Images that are from the install payload fall in the SHA category.  So that includes the jenkins related imagestreams, as well as those imagestreams not managed by samples operator that are in the openshift namespacae, like cli, must-gather, etc.

5) Finally, yes, even independent of disconnected, you can configured samples operator to change the pull spec.  Customer's might have "non-disconnected" reasons for doing this.  In this case, we were not allowing override of the jenkins imagestreams, but there was no need to prevent that.  Hence this fix.

But you should not need to employ both the samples operator registry config override and cri-o mirroring once Oleg's bug arrives.

Comment 14 Ben Parees 2019-08-26 14:35:11 UTC
> When it is a tag reference, setting up of the cri-o mirrors like you noted #comment 0 is sufficient. 

setting up the crio mirrors isn't sufficient.  the imagestream import will still fail because the imagestream you're trying to import going to still point to a registry you can't reach (registry.redhat.io).  

you need to:
1) mirror the content into your disconnected registry
2) use the samples operator config to point the samples to the disconnected registry

that will get import to succeed

if you actually want to *pull* the imported image, you also need to setup the crio mirror configuration.


This is the case for all the imagestreams the samples operator manages except the Jenkins imagestream.

(Note: I am not considering the "test/cli/must-gather" as imagestreams the samples operator manages.....those are managed by the CVO)

Comment 15 Gabe Montero 2019-08-26 16:27:01 UTC
OK XiuJuan Ben and I talked at lunch and figured out how we diverged in our understandings.

So yeah, take Ben's #comment 14, and back to your original question

1) the change here is part of what is needed to fix pulling the jenkins imagestream in disconnected ... it at least gets the registry in the pull spec correct for the subsequent pull attempt
2) in addition to 1), you'll still need https://bugzilla.redhat.com/show_bug.cgi?id=1741391 to get the actual image pull for the jenkins image stream to work once the pull spec has been updated to use the disconnected registry, as jenkins image will still be a sha ref, even with the updated registry ref

Comment 16 Gabe Montero 2019-08-26 17:07:09 UTC
Evidently I still don't have it right :-)

Ben just corrected me on slack.  The gist:

This fix allows you to import the jenkins image such that you do NOT need https://bugzilla.redhat.com/show_bug.cgi?id=1741391 as long as you use the samples operator config to cause the imagestreams to be modified to point to the disconnected registry instead of the provided registry for the imagestream

Having the registry used for the jenkins import  pointing directly to the disconnected registry will allow an image import to work, even if it is a sha.

If you have the fix to https://bugzilla.redhat.com/show_bug.cgi?id=1741391, you would not need to update the image registry for the jenkins image import to work, since it is by sha,
but since you need to update the registry for the other sample imagestreams (which are not in the payload and do not have a sha reference), you have to make the configuration change
anyway.

Comment 17 XiuJuan Wang 2019-08-27 09:44:45 UTC
@Gabe @Ben
Thanks your so so detailed explaining, I could understand well about the feature now.
I could prove the jenkins related images could be overrided after setting samplesregistry in normal cluster with 4.2.0-0.nightly-2019-08-26-235330 payload.
But I still want confirm if anyother risks exist in disconnect env(Can't install a fresh version disconnect env yet).

Comment 18 Gabe Montero 2019-08-27 13:18:32 UTC
The only possible "risk" wrt the imagestreams that the samples operator manages @XiuJuan is that if you don't import all the samples 
into the mirror, after 2 hours, for any failed imports, the samples operator will set Degraded to true.

Of course, if you add the imagestreams you do not add to the mirror to the skipped list, that can be avoided.

Otherwise, if you have mirrors set up, and you override the registry setting in the samples operator config, the imagestream imports should work.

Comment 19 XiuJuan Wang 2019-08-28 07:49:36 UTC
After mirror the jenkins images to mirror registry and override the samplesregistry, the jenkins image could import succeed
Also other mirrored images the samples operator managed import succeed.

$oc image mirror quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:c0ae54496dd2eff47b34b3e4f4bc74aebef8c542b1f84cacb2b1e5a24def57fe mirror-registry.qe.devcluster.openshift.com:5000/openshift-release-dev/ocp-v4.0-art-dev

oc get config.samples -o yaml 
  spec:
    architectures:
    - x86_64
    managementState: Managed
    samplesRegistry: mirror-registry.qe.devcluster.openshift.com:5000
  status:

$ oc describe is jenkins  -n openshift 
Name:			jenkins
Namespace:		openshift
Created:		6 minutes ago
Labels:			samples.operator.openshift.io/managed=true
Annotations:		openshift.io/display-name=Jenkins
			openshift.io/image.dockerRepositoryCheck=2019-08-28T07:41:01Z
			samples.operator.openshift.io/version=4.2.0-0.nightly-2019-08-27-105356
Image Repository:	default-route-openshift-image-registry.apps.jialiu-share0828.qe.devcluster.openshift.com/openshift/jenkins
Image Lookup:		local=false
Unique Images:		1
Tags:			2

2 (latest)
  tagged from mirror-registry.qe.devcluster.openshift.com:5000/openshift-release-dev/ocp-v4.0-art-dev@sha256:c0ae54496dd2eff47b34b3e4f4bc74aebef8c542b1f84cacb2b1e5a24def57fe
    prefer registry pullthrough when referencing this tag

  Provides a Jenkins 2.X server on RHEL 7. For more information about using this container image, including OpenShift considerations, see https://github.com/openshift/jenkins/blob/master/README.md.
  Tags: jenkins

  * mirror-registry.qe.devcluster.openshift.com:5000/openshift-release-dev/ocp-v4.0-art-dev@sha256:c0ae54496dd2eff47b34b3e4f4bc74aebef8c542b1f84cacb2b1e5a24def57fe
      6 minutes ago

Comment 20 errata-xmlrpc 2019-10-16 06:36:58 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2019:2922


Note You need to log in before you can comment on or make changes to this bug.