Bug 1671633
| Summary: | Cannot log into jenkins console with Google IDP Authorization/htpasswd | ||||||
|---|---|---|---|---|---|---|---|
| Product: | OpenShift Container Platform | Reporter: | Wenjing Zheng <wzheng> | ||||
| Component: | ImageStreams | Assignee: | Gabe Montero <gmontero> | ||||
| Status: | CLOSED ERRATA | QA Contact: | XiuJuan Wang <xiuwang> | ||||
| Severity: | medium | Docs Contact: | |||||
| Priority: | high | ||||||
| Version: | 4.1.0 | CC: | aos-bugs, bparees, gmontero, jokerman, mkhan, mmccomas, sponnaga, wzheng | ||||
| Target Milestone: | --- | ||||||
| Target Release: | 4.1.0 | ||||||
| Hardware: | Unspecified | ||||||
| OS: | Unspecified | ||||||
| Whiteboard: | |||||||
| Fixed In Version: | Doc Type: | No Doc Update | |||||
| Doc Text: |
undefined
|
Story Points: | --- | ||||
| Clone Of: | |||||||
| : | 1698720 (view as bug list) | Environment: | |||||
| Last Closed: | 2019-06-04 10:42:30 UTC | Type: | Bug | ||||
| Regression: | --- | Mount Type: | --- | ||||
| Documentation: | --- | CRM: | |||||
| Verified Versions: | Category: | --- | |||||
| oVirt Team: | --- | RHEL 7.3 requirements from Atomic Host: | |||||
| Cloudforms Team: | --- | Target Upstream Version: | |||||
| Embargoed: | |||||||
| Bug Depends On: | |||||||
| Bug Blocks: | 1698720 | ||||||
| Attachments: |
|
||||||
Created attachment 1525747 [details]
Jenkins log
Gabe, the forbidden exception is what I would expect to see if Jenkins was still trying to call <master_url>/oauth/token instead of <oauth_server_route>/oauth/token but I do not actually see that in the attached logs - PTAL. Wenjing, please provide kubeconfig and kubeadmin password. Also, confirm that the issue is still present and not transient during master rolling restart. Yeah I have had a branch to now try oauth server first before master url (so as to support both 4.0 and 3.x with the same plugin version) Looks like things have changed enough that I'm forced to see it through :-) Thanks for the triage/analysis Mo We also had a recent bump to 2.150.2 of jenkins .... let's keep our fingers crossed that does not have any bearing as well OK, just realized that "Google IDP Authorization" may not be the default oauth setting. I brought up the latest install (i.e. refreshed openshift/origin this AM) and can still log in by default Mo - can you point me to or provide instructions on how to set up "Google IDP Authorization" Also, Ben - the latest install still has a jenkins image at 2.138.4 Who should we start bugging about that ... Clayton? ART team ? Currently you need to run https://gist.github.com/enj/4725980d063133d9bb3508b8ef83bdcb The top part of that script gets around router cert issues. The bottom part configures HTPasswd, which you could easily change to configure Google. Wenjing please provide the Google IDP configuration to make it easier for Gabe to test Jenkins. I just went ahead and finalized my old https://github.com/openshift/jenkins-openshift-login-plugin/pull/49 That change, on every login attempt, first tries the endpoints retrieved from the provider, and if they are inaccessible, leverage our old defaults. It will be in v1.0.16 of the login plugin If after it is available in a jenkins image Wenjian can try, we can debug from there. The key messages to look for in the Jenkins pod logs: 1) "Using OAuth Provider specified endpoints for this login flow" 2) "Using the OpenShift Jenkins Login Plugin default for the OAuth endpoints" I assume what the mean are self explanatory. v1.0.16 of the login plugin has been initiated with the jenkins update center PR https://github.com/openshift/jenkins/pull/789 is updating the openshift/jenkins image with v1.0.16 of the plugin jenkins PR has merged rpm update job https://buildvm.openshift.eng.bos.redhat.com:8443/job/devex/job/devex%252Fjenkins-plugins/92/ dist git for rhel plugins rpm dist git http://pkgs.devel.redhat.com/cgit/rpms/jenkins-2-plugins/commit/?h=rhaos-4.0-rhel-7&id=b787f89ac0e4fb094cef954e274846e11bf8e603 Still met 403 error when try to use githubIDP to log in with below version: $oc get clusterversion NAME VERSION AVAILABLE PROGRESSING SINCE STATUS version 4.0.0-0.nightly-2019-02-12-150919 True False 43m Cluster version is 4.0.0-0.nightly-2019-02-12-150919 quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:946d88f4c19ce9952f3fc44fcab7fdd15015dc91a57f7788fdfb0546046db90c @Wenjing - please provide the jenkins pod logs from you attempt Aside from any exceptions, I want to see which of the logs I mentioned in https://bugzilla.redhat.com/show_bug.cgi?id=1671633#c11 appeared. @Mo - I got as far as https://github.com/openshift/api/blob/master/config/v1/types_oauth.go#L474-L488 in trying to convert your script from htpasswd to google id provider and got stuck in that I did not know what the contents of the client secret should be. Can you elaborate? Or is that something @Wenjing could provide? Or could you give me access to a cluster that is set up for this so I can bring up Jenkins myself? I just brought up jenkins in a 4.0 cluster and I see the login plugin using the oauth provider endpoints: Feb 13, 2019 8:30:03 PM org.openshift.jenkins.plugins.openshiftlogin.OpenShiftOAuth2SecurityRealm newOAuthSession INFO: Using OAuth Provider specified endpoints for this login flow Feb 13, 2019 8:30:49 PM org.openshift.jenkins.plugins.openshiftlogin.OpenShiftOAuth2SecurityRealm populateDefaults INFO: OpenShift OAuth: provider: OpenShiftProviderInfo: issuer: https://gmontero-api.devcluster.openshift.com:6443 auth ep: https://gmontero-api.devcluster.openshift.com:6443/oauth/authorize token ep: https://gmontero-api.devcluster.openshift.com:6443/oauth/token if something else is needed for oauth to github/google IDP I'll need that info. Or something is up on the oauth / idp side and we send this bug to Mo. I tried with today's latest beta2 payload and can test successfully with below version, so will verify this bug. quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:946d88f4c19ce9952f3fc44fcab7fdd15015dc91a57f7788fdfb0546046db90c $ oc get clusterversion NAME VERSION AVAILABLE PROGRESSING SINCE STATUS version 4.0.0-0.nightly-2019-02-13-204401 True False 55m Cluster version is 4.0.0-0.nightly-2019-02-13-204401 Since the problem described in this bug report should be resolved in a recent advisory, it has been closed with a resolution of ERRATA. For information on the advisory, and where to find the updated files, follow the link below. If the solution does not work for you, open a new bug report. https://access.redhat.com/errata/RHBA-2019:0758 |
Description of problem: When try to login jenkins console with Google IDP Authorization, below error will appear: com.google.api.client.auth.oauth2.TokenResponseException: 403 Forbidden { "kind" : "Status", "apiVersion" : "v1", "metadata" : { }, "status" : "Failure", "message" : "forbidden: User \"system:anonymous\" cannot post path \"/oauth/token\"", "reason" : "Forbidden", "details" : { }, "code" : 403 } at com.google.api.client.auth.oauth2.TokenResponseException.from(TokenResponseException.java:105) at com.google.api.client.auth.oauth2.TokenRequest.executeUnparsed(TokenRequest.java:287) at com.google.api.client.auth.openidconnect.IdTokenResponse.execute(IdTokenResponse.java:120) at org.openshift.jenkins.plugins.openshiftlogin.OpenShiftOAuth2SecurityRealm$7.onSuccess(OpenShiftOAuth2SecurityRealm.java:824) at org.openshift.jenkins.plugins.openshiftlogin.OAuthSession.doFinishLogin(OAuthSession.java:129) at org.openshift.jenkins.plugins.openshiftlogin.OpenShiftOAuth2SecurityRealm.doFinishLogin(OpenShiftOAuth2SecurityRealm.java:1075) at java.lang.invoke.MethodHandle.invokeWithArguments(MethodHandle.java:627) at org.kohsuke.stapler.Function$MethodFunction.invoke(Function.java:396) at org.kohsuke.stapler.Function$InstanceFunction.invoke(Function.java:408) at org.kohsuke.stapler.Function.bindAndInvoke(Function.java:212) at org.kohsuke.stapler.Function.bindAndInvokeAndServeResponse(Function.java:145) at org.kohsuke.stapler.MetaClass$11.doDispatch(MetaClass.java:537) at org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:58) at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:739) at org.kohsuke.stapler.Stapler.invoke(Stapler.java:870) at org.kohsuke.stapler.MetaClass$2.doDispatch(MetaClass.java:221) at org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:58) at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:739) at org.kohsuke.stapler.Stapler.invoke(Stapler.java:870) at org.kohsuke.stapler.Stapler.invoke(Stapler.java:668) at org.kohsuke.stapler.Stapler.service(Stapler.java:238) at javax.servlet.http.HttpServlet.service(HttpServlet.java:790) at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:865) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1655) at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:154) at org.openshift.jenkins.plugins.openshiftlogin.OpenShiftPermissionFilter.doFilter(OpenShiftPermissionFilter.java:242) at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:151) at org.jenkinsci.plugins.ssegateway.Endpoint$SSEListenChannelFilter.doFilter(Endpoint.java:243) at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:151) at io.jenkins.blueocean.ResourceCacheControl.doFilter(ResourceCacheControl.java:134) at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:151) at io.jenkins.blueocean.auth.jwt.impl.JwtAuthenticationFilter.doFilter(JwtAuthenticationFilter.java:61) at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:151) at hudson.util.PluginServletFilter.doFilter(PluginServletFilter.java:157) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1642) at hudson.security.csrf.CrumbFilter.doFilter(CrumbFilter.java:64) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1642) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:84) at hudson.security.UnwrapSecurityExceptionFilter.doFilter(UnwrapSecurityExceptionFilter.java:51) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at jenkins.security.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:117) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at org.acegisecurity.providers.anonymous.AnonymousProcessingFilter.doFilter(AnonymousProcessingFilter.java:125) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at org.acegisecurity.ui.rememberme.RememberMeProcessingFilter.doFilter(RememberMeProcessingFilter.java:135) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at org.acegisecurity.ui.AbstractProcessingFilter.doFilter(AbstractProcessingFilter.java:271) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at jenkins.security.BasicHeaderProcessor.doFilter(BasicHeaderProcessor.java:93) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at org.acegisecurity.context.HttpSessionContextIntegrationFilter.doFilter(HttpSessionContextIntegrationFilter.java:249) at hudson.security.HttpSessionContextIntegrationFilter2.doFilter(HttpSessionContextIntegrationFilter2.java:67) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at hudson.security.ChainedServletFilter.doFilter(ChainedServletFilter.java:90) at hudson.security.HudsonFilter.doFilter(HudsonFilter.java:171) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1642) at org.kohsuke.stapler.compression.CompressionFilter.doFilter(CompressionFilter.java:49) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1642) at hudson.util.CharacterEncodingFilter.doFilter(CharacterEncodingFilter.java:82) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1642) at org.kohsuke.stapler.DiagnosticThreadNameFilter.doFilter(DiagnosticThreadNameFilter.java:30) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1642) at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:533) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:146) at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:524) at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132) at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:257) at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1595) at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:255) at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1317) at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:203) at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:473) at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1564) at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:201) at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1219) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:144) at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:132) at org.eclipse.jetty.server.Server.handle(Server.java:531) at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:352) at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:260) at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:281) at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:102) at org.eclipse.jetty.io.ChannelEndPoint$2.run(ChannelEndPoint.java:118) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:333) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:310) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:168) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:126) at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:366) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:762) at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:680) at java.lang.Thread.run(Thread.java:748) Version-Release number of selected component (if applicable): $ oc get clusterversion NAME VERSION AVAILABLE PROGRESSING SINCE STATUS version 4.0.0-0.nightly-2019-01-30-174704 True False 2h Cluster version is 4.0.0-0.nightly-2019-01-30-174704 How reproducible: always Steps to Reproduce: 1.Config 4.0 cluster with Google IDP Authorization 2.Create a jenkins server with jenkins template 3.Access jenkins route when pod is running Actual results: com.google.api.client.auth.oauth2.TokenResponseException: 403 Forbidden Expected results: Should access Additional info: