Bug 1786246 - jenkins imagestream not downloading from redhat.registry.io
Summary: jenkins imagestream not downloading from redhat.registry.io
Keywords:
Status: CLOSED CURRENTRELEASE
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Documentation
Version: 4.2.0
Hardware: Unspecified
OS: Unspecified
low
low
Target Milestone: ---
: 4.2.z
Assignee: Brandi Munilla
QA Contact: XiuJuan Wang
Vikram Goyal
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2019-12-24 05:36 UTC by mmariyan
Modified: 2023-03-24 16:34 UTC (History)
4 users (show)

Fixed In Version:
Doc Type: If docs needed, set a value
Doc Text:
Clone Of:
Environment:
Last Closed: 2020-05-07 20:51:15 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)

Description mmariyan 2019-12-24 05:36:33 UTC
Description of problem:


When the samples operator includes the samplesRegistry configuration below, the Jenkins image cannot be pulled and shows an error.

    spec:
      architectures:
      - x86_64
      managementState: Managed
      samplesRegistry: registry.redhat.io

After removing the last line, the Jenkins image is available for use.

Error:

~~~
 - 
    message: '<imagestream/jenkins-agent-nodejs>Internal error occurred: unknown:
      Not Found<imagestream/jenkins-agent-nodejs><imagestream/jenkins>Internal error
      occurred: unknown: Not Found<imagestream/jenkins><imagestream/jenkins-agent-maven>Internal
      error occurred: unknown: Not Found<imagestream/jenkins-agent-maven>'
    reason: 'jenkins-agent-nodejs jenkins jenkins-agent-maven '
    status: "True"
    type: ImportImageErrorsExist
~~~

Version-Release number of selected component (if applicable):


How reproducible:


Steps to Reproduce:
1. edit sample operator config
2. Add sampleregistry and check the status in the opearator config
3.

Actual results:


Expected results:
 getting download failure error!

Additional info:

Comment 3 Gabe Montero 2020-01-02 16:37:14 UTC
Sure thing.  There are definitely some nuances here.

The jenkins imagestreams (jenkins core, as well as the maven/nodejs agents) are special because since the image is produced by the openshift dev organization, it is part of the install payload, and tied to the openshift release lifecycle.

This is discussed in https://docs.openshift.com/container-platform/4.2/openshift_images/configuring-samples-operator.html#samples-operator-overview_configuring-samples-operator

As a result, it is not going to have an images spec that lines up with what is provided at registry.redhat.io.

The registry.redhat.io looks like 

registry.redhat.io/openshift4/ose-jenkins-agent-maven:v4.2.11-201912100122

And depending on where you install from, the image payload spec will look something like

<registry you installed from>/ocp/4.2.....:@sha56:....

The replacement of the registry alone via the samplesOperator samplesRegistry setting is not sufficient to construct a valid pull spec.

In fact, prior to 4.2, we did not allow the samplesRegistry setting to affect the jenkins related imagestreams, but with the introduction of disconnected/restricted network install support, this was lifted.

The customer can see some of the details there at https://docs.openshift.com/container-platform/4.2/installing/installing_restricted_networks/installing-restricted-networks-preparations.html#installation-restricted-network-samples_installing-restricted-networks-preparations

So moving forward:
a) increasing the scope of the samplesRegistry override to optionally override not just the registry, but repository / entire image pull spec, is already a registered requirement
b) you can still use `oc tag --source=docker <docker pull spec> <imagestream ref>` to override any samples imagestream post install / upgrade
c) fyi setting registry.redhat.io as the samples registry override in general is a useless exercise since aside from jenkins, all the samples are already from registry.redhat.io
d) while it is under the restricted network docs, it really is more generic ... if you want the samples to come from non default registries, you have to inject the sample imagestreams into that registry, where the repository/imageref matches what is in the imagestream, and only the registry needs replacing (where you use b) if needed)
e) and if you choose to not inject some of the samples, you need to add them to the skip list

Comment 4 Chet Hosey 2020-01-06 03:56:01 UTC
4.1 includes the original configuration. After upgrading to 4.2, the same configuration prevents the Jenkins image from being pulled. Nowhere does this seem to be documented, and the upgrade process doesn’t update the configuration to prevent breakage.

Is this a fair summary?

Comment 5 Chet Hosey 2020-01-06 07:02:22 UTC
Sorry, rereading the initial description, it wasn't made clear that this configuration was left over from a 4.1 installation, which was the context in which the ticket was filed with support.

FYI -- a 4.1 cluster includes the setting. To get Jenkins to pull on a 4.2 cluster, I had to remove it. I'd filed the support ticket to alert them to the issue and workaround.

But overall it seems problematic that clusters upgraded from 4.1 -> 4.2 would no longer be able to pull the Jenkins operator without manual intervention, and even less optimal that it doesn't seem to be documented anywhere.

Comment 6 Gabe Montero 2020-01-06 16:12:13 UTC
We would call setting the samplesRegistry to registry.redhat.io unnecessary and a misconfiguration.

I'm fine with reopening this as a docs bug to clarify that in general, along with a 4.1 upgrade note.

Comment 7 Gabe Montero 2020-01-06 16:15:04 UTC
Brandi - Some more samples docs clarification appear to be needed around setting samplesRegistry to registry.redhat.io in correlation with the jenkins imagestreams in general across 4.x.

Also, a 4.1 -> 4.2 upgrade release note telling folks to turn that off if that have done so seems warranted.

Comment 8 Chet Hosey 2020-01-06 17:20:08 UTC
It might be unnecessary and a misconfiguration but I believe it's what 4.1 did out of the box. (I did try to confirm this using `openshift-install create manifests` with a 4.1.9 installer, but the samples-operator config wasn't included in the output.)

We did toggle managementState, but it would be surprising if we'd ever had a reason to touch the samplesRegistry setting.

Comment 9 Gabe Montero 2020-01-10 14:36:00 UTC
4.1 did not set samplesRegistry to registry.redhat.io out of the box.

Here is the code that creates the default entry out of the box:
https://github.com/openshift/cluster-samples-operator/blob/release-4.1/pkg/stub/handler.go#L285-L303

Comment 10 Brandi Munilla 2020-02-28 21:53:22 UTC
@Gabe I opened the following PR to start calling out the information on Jenkins images in the Samples Operator documentation. It's a first pass, and I still need to address the registry setting and release notes. 

https://github.com/openshift/openshift-docs/pull/20124

Comment 11 Brandi Munilla 2020-04-06 21:45:52 UTC
We've added context and a warning, but are avoiding adding the `oc tag --source=docker ...` procedure to avoid potential complications. 

The PR is ready for QE review. 

https://github.com/openshift/openshift-docs/pull/20124

Comment 12 XiuJuan Wang 2020-04-07 02:10:48 UTC
@Brandi A typo need fix, add comment in PR.
Other lgtm

Comment 13 Brandi Munilla 2020-04-08 20:34:15 UTC
@XiuJuan thank you! I've updated the PR with your and Michael's requests, and am going ahead with the merge + cherry pick.


Note You need to log in before you can comment on or make changes to this bug.