Bug 1671633 - Cannot log into jenkins console with Google IDP Authorization/htpasswd
Summary: Cannot log into jenkins console with Google IDP Authorization/htpasswd
Keywords:
Status: CLOSED ERRATA
Alias: None
Product: OpenShift Container Platform
Classification: Red Hat
Component: Image
Version: 4.1.0
Hardware: Unspecified
OS: Unspecified
high
medium
Target Milestone: ---
: 4.1.0
Assignee: Gabe Montero
QA Contact: XiuJuan Wang
URL:
Whiteboard:
Depends On:
Blocks: 1698720
TreeView+ depends on / blocked
 
Reported: 2019-02-01 06:43 UTC by Wenjing Zheng
Modified: 2019-06-04 10:43 UTC (History)
8 users (show)

Fixed In Version:
Doc Type: No Doc Update
Doc Text:
undefined
Clone Of:
: 1698720 (view as bug list)
Environment:
Last Closed: 2019-06-04 10:42:30 UTC
Target Upstream Version:


Attachments (Terms of Use)
Jenkins log (23.32 KB, text/plain)
2019-02-01 07:41 UTC, Wenjing Zheng
no flags Details


Links
System ID Priority Status Summary Last Updated
Red Hat Product Errata RHBA-2019:0758 None None None 2019-06-04 10:43:52 UTC

Description Wenjing Zheng 2019-02-01 06:43:35 UTC
Description of problem:
When try to login jenkins console with Google IDP Authorization, below error will appear:
com.google.api.client.auth.oauth2.TokenResponseException: 403 Forbidden
{
  "kind" : "Status",
  "apiVersion" : "v1",
  "metadata" : { },
  "status" : "Failure",
  "message" : "forbidden: User \"system:anonymous\" cannot post path \"/oauth/token\"",
  "reason" : "Forbidden",
  "details" : { },
  "code" : 403
}
	at com.google.api.client.auth.oauth2.TokenResponseException.from(TokenResponseException.java:105)
	at com.google.api.client.auth.oauth2.TokenRequest.executeUnparsed(TokenRequest.java:287)
	at com.google.api.client.auth.openidconnect.IdTokenResponse.execute(IdTokenResponse.java:120)
	at org.openshift.jenkins.plugins.openshiftlogin.OpenShiftOAuth2SecurityRealm$7.onSuccess(OpenShiftOAuth2SecurityRealm.java:824)
	at org.openshift.jenkins.plugins.openshiftlogin.OAuthSession.doFinishLogin(OAuthSession.java:129)
	at org.openshift.jenkins.plugins.openshiftlogin.OpenShiftOAuth2SecurityRealm.doFinishLogin(OpenShiftOAuth2SecurityRealm.java:1075)
	at java.lang.invoke.MethodHandle.invokeWithArguments(MethodHandle.java:627)
	at org.kohsuke.stapler.Function$MethodFunction.invoke(Function.java:396)
	at org.kohsuke.stapler.Function$InstanceFunction.invoke(Function.java:408)
	at org.kohsuke.stapler.Function.bindAndInvoke(Function.java:212)
	at org.kohsuke.stapler.Function.bindAndInvokeAndServeResponse(Function.java:145)
	at org.kohsuke.stapler.MetaClass$11.doDispatch(MetaClass.java:537)
	at org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:58)
	at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:739)
	at org.kohsuke.stapler.Stapler.invoke(Stapler.java:870)
	at org.kohsuke.stapler.MetaClass$2.doDispatch(MetaClass.java:221)
	at org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:58)
	at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:739)
	at org.kohsuke.stapler.Stapler.invoke(Stapler.java:870)
	at org.kohsuke.stapler.Stapler.invoke(Stapler.java:668)
	at org.kohsuke.stapler.Stapler.service(Stapler.java:238)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:790)
	at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:865)
	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1655)
	at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:154)
	at org.openshift.jenkins.plugins.openshiftlogin.OpenShiftPermissionFilter.doFilter(OpenShiftPermissionFilter.java:242)
	at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:151)
	at org.jenkinsci.plugins.ssegateway.Endpoint$SSEListenChannelFilter.doFilter(Endpoint.java:243)
	at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:151)
	at io.jenkins.blueocean.ResourceCacheControl.doFilter(ResourceCacheControl.java:134)
	at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:151)
	at io.jenkins.blueocean.auth.jwt.impl.JwtAuthenticationFilter.doFilter(JwtAuthenticationFilter.java:61)
	at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:151)
	at hudson.util.PluginServletFilter.doFilter(PluginServletFilter.java:157)
	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1642)
	at hudson.security.csrf.CrumbFilter.doFilter(CrumbFilter.java:64)
	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1642)
	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:84)
	at hudson.security.UnwrapSecurityExceptionFilter.doFilter(UnwrapSecurityExceptionFilter.java:51)
	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
	at jenkins.security.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:117)
	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
	at org.acegisecurity.providers.anonymous.AnonymousProcessingFilter.doFilter(AnonymousProcessingFilter.java:125)
	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
	at org.acegisecurity.ui.rememberme.RememberMeProcessingFilter.doFilter(RememberMeProcessingFilter.java:135)
	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
	at org.acegisecurity.ui.AbstractProcessingFilter.doFilter(AbstractProcessingFilter.java:271)
	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
	at jenkins.security.BasicHeaderProcessor.doFilter(BasicHeaderProcessor.java:93)
	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
	at org.acegisecurity.context.HttpSessionContextIntegrationFilter.doFilter(HttpSessionContextIntegrationFilter.java:249)
	at hudson.security.HttpSessionContextIntegrationFilter2.doFilter(HttpSessionContextIntegrationFilter2.java:67)
	at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87)
	at hudson.security.ChainedServletFilter.doFilter(ChainedServletFilter.java:90)
	at hudson.security.HudsonFilter.doFilter(HudsonFilter.java:171)
	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1642)
	at org.kohsuke.stapler.compression.CompressionFilter.doFilter(CompressionFilter.java:49)
	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1642)
	at hudson.util.CharacterEncodingFilter.doFilter(CharacterEncodingFilter.java:82)
	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1642)
	at org.kohsuke.stapler.DiagnosticThreadNameFilter.doFilter(DiagnosticThreadNameFilter.java:30)
	at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1642)
	at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:533)
	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:146)
	at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:524)
	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
	at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:257)
	at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1595)
	at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:255)
	at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1317)
	at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:203)
	at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:473)
	at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1564)
	at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:201)
	at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1219)
	at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:144)
	at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132)
	at org.eclipse.jetty.server.Server.handle(Server.java:531)
	at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:352)
	at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:260)
	at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:281)
	at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:102)
	at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:118)
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:333)
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:310)
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168)
	at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:126)
	at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:366)
	at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:762)
	at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:680)
	at java.lang.Thread.run(Thread.java:748)


Version-Release number of selected component (if applicable):
$ oc get clusterversion
NAME      VERSION                             AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.0.0-0.nightly-2019-01-30-174704   True        False         2h      Cluster version is 4.0.0-0.nightly-2019-01-30-174704


How reproducible:
always

Steps to Reproduce:
1.Config 4.0 cluster with Google IDP Authorization
2.Create a jenkins server with jenkins template
3.Access jenkins route when pod is running

Actual results:
com.google.api.client.auth.oauth2.TokenResponseException: 403 Forbidden

Expected results:
Should access

Additional info:

Comment 1 Wenjing Zheng 2019-02-01 07:41:51 UTC
Created attachment 1525747 [details]
Jenkins log

Comment 3 Mo 2019-02-03 23:29:55 UTC
Gabe, the forbidden exception is what I would expect to see if Jenkins was still trying to call <master_url>/oauth/token instead of <oauth_server_route>/oauth/token but I do not actually see that in the attached logs - PTAL.

Comment 4 Mo 2019-02-03 23:31:03 UTC
Wenjing, please provide kubeconfig and kubeadmin password.

Also, confirm that the issue is still present and not transient during master rolling restart.

Comment 5 Gabe Montero 2019-02-04 15:18:34 UTC
Yeah I have had a branch to now try oauth server first before master url (so as to support both 4.0 and 3.x with the same plugin version)

Looks like things have changed enough that I'm forced to see it through :-)


Thanks for the triage/analysis Mo

Comment 6 Gabe Montero 2019-02-04 15:21:18 UTC
We also had a recent bump to 2.150.2 of jenkins .... let's keep our fingers crossed that does not have any bearing as well

Comment 7 Gabe Montero 2019-02-04 16:18:04 UTC
OK, just realized that "Google IDP Authorization" may not be the default oauth setting.

I brought up the latest install (i.e. refreshed openshift/origin this AM) and can still log in by default

Mo - can you point me to or provide instructions on how to set up "Google IDP Authorization"

Comment 8 Gabe Montero 2019-02-04 16:19:11 UTC
Also, Ben - the latest install still has a jenkins image at 2.138.4

Who should we start bugging about that ... Clayton?  ART team ?

Comment 9 Mo 2019-02-04 16:53:36 UTC
Currently you need to run https://gist.github.com/enj/4725980d063133d9bb3508b8ef83bdcb

The top part of that script gets around router cert issues.  The bottom part configures HTPasswd, which you could easily change to configure Google.

Comment 10 Mo 2019-02-04 16:55:53 UTC
Wenjing please provide the Google IDP configuration to make it easier for Gabe to test Jenkins.

Comment 11 Gabe Montero 2019-02-05 18:01:03 UTC
I just went ahead and finalized my old https://github.com/openshift/jenkins-openshift-login-plugin/pull/49

That change, on every login attempt, first tries the endpoints retrieved from the provider, and if they are inaccessible,
leverage our old defaults.

It will be in v1.0.16 of the login plugin

If after it is available in a jenkins image Wenjian can try, we can debug from there.

The key messages to look for in the Jenkins pod logs:

1) "Using OAuth Provider specified endpoints for this login flow"
2) "Using the OpenShift Jenkins Login Plugin default for the OAuth endpoints"

I assume what the mean are self explanatory.

Comment 12 Gabe Montero 2019-02-06 14:37:46 UTC
v1.0.16 of the login plugin has been initiated with the jenkins update center

Comment 13 Gabe Montero 2019-02-07 03:14:20 UTC
PR https://github.com/openshift/jenkins/pull/789 is updating the openshift/jenkins image with v1.0.16 of the plugin

Comment 14 Gabe Montero 2019-02-07 05:19:14 UTC
jenkins PR has merged

rpm update job https://buildvm.openshift.eng.bos.redhat.com:8443/job/devex/job/devex%252Fjenkins-plugins/92/

Comment 16 Wenjing Zheng 2019-02-13 08:18:39 UTC
Still met 403 error when try to use githubIDP to log in with below version:
$oc get clusterversion
NAME      VERSION                             AVAILABLE   PROGRESSING   SINCE 
  STATUS
version   4.0.0-0.nightly-2019-02-12-150919   True        False         43m   
  Cluster version is 4.0.0-0.nightly-2019-02-12-150919

quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:946d88f4c19ce9952f3fc44fcab7fdd15015dc91a57f7788fdfb0546046db90c

Comment 18 Gabe Montero 2019-02-13 16:25:55 UTC
@Wenjing - please provide the jenkins pod logs from you attempt

Aside from any exceptions, I want to see which of the logs I mentioned in https://bugzilla.redhat.com/show_bug.cgi?id=1671633#c11
appeared.

Comment 19 Gabe Montero 2019-02-13 16:29:05 UTC
@Mo - I got as far as https://github.com/openshift/api/blob/master/config/v1/types_oauth.go#L474-L488
in trying to convert your script from htpasswd to google id provider and got stuck in that I did 
not know what the contents of the client secret should be.

Can you elaborate?  
Or is that something @Wenjing could provide?
Or could you give me access to a cluster that is set up for this so I can bring up Jenkins myself?

Comment 20 Gabe Montero 2019-02-13 20:35:31 UTC
I just brought up jenkins in a 4.0 cluster and I see the login plugin using the oauth provider endpoints:

Feb 13, 2019 8:30:03 PM org.openshift.jenkins.plugins.openshiftlogin.OpenShiftOAuth2SecurityRealm newOAuthSession
INFO: Using OAuth Provider specified endpoints for this login flow
Feb 13, 2019 8:30:49 PM org.openshift.jenkins.plugins.openshiftlogin.OpenShiftOAuth2SecurityRealm populateDefaults
INFO: OpenShift OAuth: provider: OpenShiftProviderInfo: issuer: https://gmontero-api.devcluster.openshift.com:6443 auth ep: https://gmontero-api.devcluster.openshift.com:6443/oauth/authorize token ep: https://gmontero-api.devcluster.openshift.com:6443/oauth/token


if something else is needed for oauth to github/google IDP I'll need that info.

Or something is up on the oauth / idp side and we send this bug to Mo.

Comment 21 Wenjing Zheng 2019-02-14 08:54:17 UTC
I tried with today's latest beta2 payload and can test successfully with below version, so will verify this bug.
quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:946d88f4c19ce9952f3fc44fcab7fdd15015dc91a57f7788fdfb0546046db90c

$ oc get clusterversion
NAME      VERSION                             AVAILABLE   PROGRESSING   SINCE     STATUS
version   4.0.0-0.nightly-2019-02-13-204401   True        False         55m       Cluster version is 4.0.0-0.nightly-2019-02-13-204401

Comment 24 errata-xmlrpc 2019-06-04 10:42:30 UTC
Since the problem described in this bug report should be
resolved in a recent advisory, it has been closed with a
resolution of ERRATA.

For information on the advisory, and where to find the updated
files, follow the link below.

If the solution does not work for you, open a new bug report.

https://access.redhat.com/errata/RHBA-2019:0758


Note You need to log in before you can comment on or make changes to this bug.