Bug 1533938 - Jenkins API authentication (with a serviceaccount) fails until the first web access (then it works)
Summary: Jenkins API authentication (with a serviceaccount) fails until the first web ...
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Build
Version: 3.5.1
Hardware: Unspecified
OS: Unspecified
unspecified
medium
Target Milestone: ---
: 3.6.z
Assignee: Gabe Montero
QA Contact: wewang
URL:
Whiteboard:
Depends On:
Blocks:
TreeView+ depends on / blocked
 
Reported: 2018-01-12 15:17 UTC by François Cami
Modified: 2021-09-09 13:02 UTC (History)
6 users (show)

Fixed In Version:
Doc Type: Bug Fix
Doc Text:
Cause: The login plugin servlet filter was not getting re-registered in jenkins after a restart until a login via the console occurred. Consequence: http access to an openshift jenkins pod with oauth via a bearer token was not possible until the servlet filter was re-registered Fix: force the re-register of the login plugin servlet filter after pod restart Result: http access via bearer token to a openshift jenkins pod with oauth is possible without having to login via the jenkins console first
Clone Of:
Environment:
Last Closed: 2018-04-12 06:01:18 UTC
Target Upstream Version:
Embargoed:


Attachments (Terms of Use)


Links
System ID Private Priority Status Summary Last Updated
Red Hat Product Errata RHBA-2018:1106 0 None None None 2018-04-12 06:01:46 UTC

Description François Cami 2018-01-12 15:17:15 UTC
Description of problem:

1 - Deploy the out of box jenkins image with persistent volume
2 - Non-browser access to jenkins API with a Bearer => OK
3 - Delete the jenkins Pod. A new pod is scheduled by Openshift
4 - Non-browser access to jenkins API with a Bearer => KO
5 - Browser access to jenkins, the page with both Jenkins and OpenShift is displayed
6 - Non-browser access to jenkins API with a Bearer => OK


Version-Release number of selected component (if applicable):
v2.32-15

How reproducible:
Always, after scaling the jenkins pod to 0 then to 1.

Comment 1 Ben Parees 2018-01-12 15:19:37 UTC
I think Gabe just found this and has a fix for it, not sure if it's compatible back to 3.5 though (if it is you might be able to just update your openshift login plugin in your image to the latest version).

Comment 3 Gabe Montero 2018-01-12 16:08:45 UTC
Yeah this is the bug that was seen internally recently as well that I just fixed.

And yes, the latest login plugin, v1.0.3, should be compatible with 3.5. 

Can the customer download that version of the login plugin from the jenkins update center, or do they need a new rhel openshift jenkins image built?

Comment 4 François Cami 2018-01-12 16:15:40 UTC
They'd really prefer a new rhel openshift jenkins image but if you can point me to the exact place they could download the plugin I'll send it to them.

Comment 5 Gabe Montero 2018-01-12 18:22:12 UTC
OK, PR https://github.com/openshift/jenkins/pull/479 is the first step in getting a new rhel openshift jenkins image built as part of the openshift errata process (openshift has errata go out on a 3 week cycle I'm told).  I think the next errata is Jan 18.  Not sure if this is getting in too late for that.

We'll have to coordinate with our release / CICD team to see what happens there.

The download url for the plugin with the fix at the jenkins update center is https://updates.jenkins-ci.org/download/plugins/openshift-login/1.0.3/openshift-login.hpi

Per process (as I understand it) moving to modified

Comment 6 Gabe Montero 2018-01-15 14:51:17 UTC
We have build issues with the noted PR that needs to be sorted out.  Ben was able to sort out the cause and provided some possible solutions.  I'll be spending time today seeing which option has the best combination of feasibility and desirability and update this bug once a decision is made.

Moving back to assigned.

Comment 7 Gabe Montero 2018-01-16 19:13:31 UTC
Set target to 3.6

Reminder to folks ... we don't build version specific images prior to 3.6.  In general, the 3.6 images are the latest available images to be applied to 3.5 clusters.

I manually merged the 3.6 openshift/jenkins PR.

The next step as I understand it is for the CI/CI team to reconcile the 3.6 dist git, resulting failed builds from our change, etc. in order to get a new 3.6 based image in the errata pipeline (cc:ed Justin and Adam to keep me honest there).

Moving back to modified.

Comment 10 wewang 2018-01-29 09:07:54 UTC
@Gabe Montero when test jenkins:latest in 3.6 env
openshift v3.6.173.0.97
kubernetes v1.6.1+5115d708d7
etcd 3.2.1
found another issue,pod cannot be running: https://bugzilla.redhat.com/show_bug.cgi?id=1539574

@François Cami
could you help to check if the command is right:
curl -H "Authorization: Bearer $(oc whoami -t)" ${JENKINS_URL}
for  "Non-browser access to jenkins API with a Bearer" step, thanks

Comment 11 wewang 2018-01-29 09:53:51 UTC
or command:curl -H "Authorization: Bearer $(oc whoami -t)" ${JENKINS_URL} --insecure

Comment 12 Gabe Montero 2018-01-29 18:06:25 UTC
OK, I just confirmed with Justin, that a 3.6 build with this fix was not available internally yet.

The build is currently cooking at https://buildvm.openshift.eng.bos.redhat.com:8443/job/aos-cd-builds/job/build%252Fose/166/

that job should include the changes from https://github.com/openshift/jenkins/pull/493 which got the 3.6 jenkins image working again after the changes from https://github.com/openshift/jenkins/pull/482, after pulling in the fix for this actual bug, https://github.com/openshift/jenkins/pull/479

Sending back to modified, and asking QE to get a 3.6 build, and in particular the openshift jenkins rhel image associated with that build, that stems from https://buildvm.openshift.eng.bos.redhat.com:8443/job/aos-cd-builds/job/build%252Fose/166/
 or later.

On the curl access, I know the question was addressed to François, but I can chime in that what was provided in https://bugzilla.redhat.com/show_bug.cgi?id=1533938#c10 looks correct, though you have to be cognizant of the http/https considerations around the jenkins URL and parameters needed with curl.

Comment 13 Gabe Montero 2018-01-29 22:05:12 UTC
Per update from Justin, looks like release candidate  3.6.173.0.101-1 should include the openshift jenkins rhel image update we want to verify here.

Comment 14 Gabe Montero 2018-02-21 03:12:28 UTC
Turns out, this change has not made a 3.6.z image as of yet (specifically, the pulling in v1.0.3 of the openshift-login plugin).

Per Justin's response in our email thread, this fix will be going out in the next non-CVE errata.

Moving back to modified.

Comment 15 Gabe Montero 2018-02-21 03:23:39 UTC
Sam - please make sure this defect is updated once we get a 3.6 image with the RPM update that QE can go after.  Thanks.

Comment 17 wewang 2018-03-06 06:21:46 UTC
verfied  latest jenkins(2.89.2)in 3.6 env:
openshift version: v3.6.173.0.104

steps:
1. Create jenkins app
  $oc new-app jenkins-persistent
  $oc get pods
NAME              READY     STATUS    RESTARTS   AGE
jenkins-1-ft41j   1/1       Running   0          8m

2. Non-browser access to jenkins API with a Bearer    => OK
 $curl  -H "Authorization: Bearer YAhj3xxxxxxxxeE" https://jenkins-wewang1.apps.0306-ajh.qe.rhcloud.com/login --insecure
  <!DOCTYPE html><html><head resURL="/static/c4687114" data-rooturl="" data-resurl="/static/c4687114">
    <title>Jenkins</title><link rel="stylesheet" href="/static/c4687114/css/layout-common.css" type="text/css" /><link rel="stylesheet" href="/static/c4687114/css/style.css" type="text/css" /><link rel="stylesheet" href="/static/c4687114/css/color.css" type="text/css" /><link rel="stylesheet" href="/static/c4687114/css/responsive-grid.css" type="text/css" /><link rel="shortcut icon" href="/static/c4687114/favicon.ico" type="image/vnd.microsoft.icon" /><link color="black" rel="mask-icon" href="/images/mask-icon.svg" /><script>var isRunAsTest=false; var rootURL=""; var resURL="/static/c4687114";</script>

3. Delete the jenkins Pod. A new pod is scheduled by Openshift
$ oc delete pod jenkins-1-ft41j
pod "jenkins-1-ft41j" deleted
$# oc get pods 
NAME              READY     STATUS    RESTARTS   AGE
jenkins-1-wpj7n   1/1       Running   0          15m

4.Non-browser access to jenkins API with a Bearer => OK
 $curl  -H "Authorization: Bearer YAhj3xxxxxxxxeE" https://jenkins-wewang1.apps.0306-ajh.qe.rhcloud.com/login --insecure
  <!DOCTYPE html><html><head resURL="/static/c4687114" data-rooturl="" data-resurl="/static/c4687114">
    <title>Jenkins</title><link rel="stylesheet" href="/static/c4687114/css/layout-common.css" type="text/css" /><link rel="stylesheet" href="/static/c4687114/css/style.css" type="text/css" /><link rel="stylesheet" href="/static/c4687114/css/color.css" type="text/css" /><link rel="stylesheet" href="/static/c4687114/css/responsive-grid.css" type="text/css" /><link rel="shortcut icon" href="/static/c4687114/favicon.ico" type="image/vnd.microsoft.icon" /><link color="black" rel="mask-icon" href="/images/mask-icon.svg" /><script>var isRunAsTest=false; var rootURL=""; var resURL="/static/c4687114";</script>

5.Browser access to jenkins, the page with both Jenkins and OpenShift is displayed  =>  OK
6 - Non-browser access to jenkins API with a Bearer => OK

Comment 18 wewang 2018-03-06 07:01:02 UTC
Add a case for the senario.

Comment 21 errata-xmlrpc 2018-04-12 06:01:18 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2018:1106


Note You need to log in before you can comment on or make changes to this bug.