Bug 1558997 - Issue when deploying Jenkins instances which have routes on various sharded routers
Summary: Issue when deploying Jenkins instances which have routes on various sharded r...
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Image
Version: 3.7.0
Hardware: Unspecified
OS: Unspecified
Target Milestone: ---
: 3.9.z
Assignee: Gabe Montero
QA Contact: Dongbo Yan
Depends On:
TreeView+ depends on / blocked
Reported: 2018-03-21 13:34 UTC by Pili Guerra
Modified: 2018-06-18 18:14 UTC (History)
5 users (show)

Fixed In Version: openshift3/jenkins-2-rhel7:v3.9 (38596f5d9a7b)
Doc Type: Bug Fix
Doc Text:
Cause: if the jenkins root URL could not be retrieved from the route from the jenkins template, and unusable url could be used in constructing the various annotations for pipeline builds Consequence: the associated annotation links would not render when referenced from the openshift console Fix: to help account for those edge cases, the sync plugin will now look for explicitly configured root urls in jenkins as well Result:the links associated with the pipeline build annotations will render if the root url is properly configured
Clone Of:
Last Closed: 2018-06-18 18:14:05 UTC
Target Upstream Version:

Attachments (Terms of Use)

System ID Priority Status Summary Last Updated
Red Hat Product Errata RHSA-2018:2013 normal SHIPPED_LIVE Important: OpenShift Container Platform 3.9 security, bug fix, and enhancement update 2018-06-27 22:01:43 UTC

Description Pili Guerra 2018-03-21 13:34:34 UTC
Description of problem:

When deploying Jenkins instances which have routes on various sharded routers ("network zones" subdomains, e.g. "zone1" (default), "zone2") and setting up build pipelines, the incorrect jenkins build job urls are generated. 

For example, a Jenkins instance is deployed to the "zone2 network zone", corresponding to the shared router "zone2".
The route URL will be: jenkins.zone2.apps.ocp.example.com

However, in the pipeline UI, when looking at the logs and other associated urls, they points to jenkins.zone1.apps.ocp.example.com, which doesn't exist.

"zone1" is the default router/shard.

The jenkins pipeline build object annotations show the following incorrect values:

- openshift.io/jenkins-build-uri: https://jenkins.zone1.apps.ocp.example.com/job/test-pipeline/2/
- openshift.io/jenkins-console-log-url: https://jenkins.zone1.apps.ocp.example.com/job/test-pipeline/2/console
- openshift.io/jenkins-log-url: https://jenkins.zone1.apps.ocp.example.com/job/test-pipeline/2/consoleText

Version-Release number of selected component (if applicable):

jenkins-2-rhel7:v3. and openshift-sync:0.1.32

How reproducible:

Every time jenkins instances are created with routes targetting specific sharded routers/"network zones".

Steps to Reproduce:

On an Openshift instance with sharded routers

1. Deploy a new Jenkins instance into a project
2. Run an OpenShift build config of type JenkinsPipeline on Jenkins instance
3. Start the pipeline, the created build instance in OpenShift will create a build, where the progress of the pipeline can be watched and a link to the log of the current jenkins build is provided. 

This link points to the wrong OpenShift router.The URL for this link is taken from the annotations if the OpenShift Build object.

Actual results:

Incorrect annotations and therefore urls are generated and shown in the Openshift UI for the Pipeline Build e.g /console/project/PROJECT_NAME/browse/pipelines/PIPELINE/PIPELINE_BUILD?tab=details

Expected results:

The correct route is used to generate the pipeline build annotations and urls.

Additional info:

These annotations are set by the Jenkins openshift-sync-plugin. The original implementation fetches the host address from the Jenkins route spec.
However, since the project uses a different router, this host address is not correct.

It seems that the correct address can be found within status/ingress[0]/host.

As a workaround, the customer has patched this plugin[1] and is now seeing the correct URLs in his build pipeline. 

[1] https://github.com/bakito/openshift-sync-plugin/commit/105268d90fe4764c9634284a3e2d7fa50e8401bc

Comment 1 Gabe Montero 2018-03-21 14:03:29 UTC
Hello Pili

Yep, as the customer found, that issue has been fixed at the commit level you noted.

That fix will be included in the jenkins-2-rhel7:v3.9 image that is about to get release to registry.access.redhat.com as part of the v3.9 GA, and they could continue to install the version of the plugin with the fix in the meantime.

There are currently no plans to backport this fix to 3.6.

Can the customer upgrade their image to 3.9 when it comes out?  And do they have plans on upgrading their cluster anytime soon?

Comment 3 Gabe Montero 2018-03-21 15:37:47 UTC
Sure thing Pili

First, wrt versions, their master may be at 3.7, but to copy/paste what you had in the description, for their image, they are using:

jenkins-2-rhel7:v3. and openshift-sync:0.1.32

Best practice, they should minimally get their image in sync with their master version.

That said, they key thing to consider is client/server api compatability.

openshift master is suppose to support N-1

since there was no 3.8 released (only an internal item), 3.9 is suppose to maintain backward compatability with 3.7.  So they should be good using the 3.9 image with a 3.7 master.

I've seen regression testing for that scenario pass.

As to the fix, yeah, I should have clarified how we are fixing it.

In particular, we allow for using the jenkins configuration of the root URL to be used for the annotations the customer is worried about.

The commit in question is https://github.com/openshift/jenkins-sync-plugin/commit/4fafda8290bb1022739318194479b6108ca635b5

The lines of code I'm referring to is https://github.com/openshift/jenkins-sync-plugin/blob/master/src/main/java/io/fabric8/jenkins/openshiftsync/OpenShiftUtils.java#L441-L455

The customer can explicitly configure the jenkins instance to the URL he wants vs. the sync plugin trying to mine the URL from the route.

The mining of information from the ingress, etc. is not a complexity we would take on as a bug.  Among other things, the details he notes in his commit around simply taking the first host, etc. and details that are not trivial to get correct in a generic sense.

If explicitly configuring the jenkins instance is not sufficient for the customer as an alternative, then they should open an RFE to ask for configurable support of ingress routers with jenkins routes when creating these annotations.

Comment 4 Gabe Montero 2018-04-03 14:20:49 UTC
The 3.9 GA image has been released and should have the change for explicitly configuring the root URL.

The redhat CA Codrin has also opened an RFE to expand the capabilities of the sync plugin to dynamically handle to noted router configuration.

Ben FYI ^^

I'm moving this to ON QA, so they can validate the function we are delivering with  https://github.com/openshift/jenkins-sync-plugin/commit/4fafda8290bb1022739318194479b6108ca635b5, which is those build annotations using a jenkins root URL that is explicitly configuired in the jenkins instance.

This is the manual configuration "workaround" to this situation.

Dangbo Yan:  here are a couple of links explaining how you can set this root URL from the varioud Jenkins admin constructs:

- https://stackoverflow.com/questions/9089566/jenkins-website-root-path
- https://stackoverflow.com/questions/30355079/jenkins-setting-root-url-via-groovy-api?rq=1

pick which every one you are most comfortable with

then create a pipeline build, and verify the various url annotations use that root URL

Comment 5 Dongbo Yan 2018-04-09 12:00:37 UTC
Verified with
brew-pulp.../openshift3/jenkins-2-rhel7:v3.9 (38596f5d9a7b)

# oc get route
NAME                     HOST/PORT                                                   PATH      SERVICES                 PORT      TERMINATION     WILDCARD
jenkins                  jenkins-dyan2.test.com                                                jenkins                  <all>     edge/Redirect   None

# oc describe builds/sample-pipeline-1

Annotations:	openshift.io/build-config.name=sample-pipeline

Comment 6 Dongbo Yan 2018-04-10 02:23:53 UTC
openshift-sync	1.0.7

Reproduce steps:
1. create a route without default subdomain suffix as the host
2. create a pipeline build, check the build annotations

sync plugin could sync the route host url

Note You need to log in before you can comment on or make changes to this bug.